From tziporet at dev.mellanox.co.il Sun Jun 1 01:52:48 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 01 Jun 2008 11:52:48 +0300 Subject: [ofa-general] ***SPAM*** How to test OFED install In-Reply-To: References: Message-ID: <48426360.1000204@mellanox.co.il> Sangamesh B wrote: > > > Can some one send the link/document which can explain the OFA tests:To > check drivers installed properly? > Which tests are you refer to? OFED install process and its outcome is explained in the OFED_Installation_Guide.txt that come with OFED release under docs Tziporet From vlad at lists.openfabrics.org Sun Jun 1 03:11:17 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 1 Jun 2008 03:11:17 -0700 (PDT) Subject: [ofa-general] ofa_1_3_kernel 20080601-0200 daily build status Message-ID: <20080601101117.C4460E60B66@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_3/linux-2.6.git git_branch: ofed_kernel Common build parameters: --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-user_mad-mod --with-user_access-mod --with-mthca-mod --with-mlx4-mod --with-core-mod --with-addr_trans-mod --with-rds-mod --with-cxgb3-mod --with-nes-mod Passed: Passed on i686 with 2.6.15-23-server Passed on i686 with linux-2.6.13 Passed on i686 with linux-2.6.12 Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.14 Passed on i686 with linux-2.6.15 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on x86_64 with linux-2.6.12 Passed on x86_64 with linux-2.6.13 Passed on x86_64 with linux-2.6.15 Passed on x86_64 with linux-2.6.14 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.12 Passed on ia64 with linux-2.6.13 Passed on ia64 with linux-2.6.14 Passed on ia64 with linux-2.6.15 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on powerpc with linux-2.6.13 Passed on powerpc with linux-2.6.12 Passed on powerpc with linux-2.6.14 Passed on powerpc with linux-2.6.15 Passed on ppc64 with linux-2.6.12 Passed on ppc64 with linux-2.6.13 Passed on ppc64 with linux-2.6.15 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.14 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.24 Passed on ppc64 with linux-2.6.19 Failed: From BlogBlaster at lists.openfabrics.org Sun Jun 1 03:38:55 2008 From: BlogBlaster at lists.openfabrics.org (BlogBlaster at lists.openfabrics.org) Date: 01 Jun 2008 03:38:55 -0700 Subject: [ofa-general] "How would you like to have your ad on 2 Million Websites ?" Message-ID: <20080601033855.047FB7064CA6BD7C@from.header.has.no.domain> How would you like 2 Million Sites linking to your ad ? Weblog or blog population is exploding around the world, resembling the growth of e-mail users in the 1990s. Post your ads where people read them! - What if you could place your ad on all these sites ? Right, that would mean you would have millions of sites linking to your ad. For Full details please read the attached .html file Unsubscribe please read the attached .html file, click on contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Sun Jun 1 08:05:01 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 01 Jun 2008 18:05:01 +0300 Subject: [ofa-general] [PATCH 1/2] IB/core: Add comletion flag for send with invalidate Message-ID: <1212332701.4433.15.camel@mtls03> >From abebd3a1809c9c5ba422e36d9d75a3cbf5fde1fe Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Sun, 11 May 2008 14:59:08 +0300 Subject: [PATCH] IB/core: Add comletion flag for send with invalidate Add IB_WC_WITH_INVALIDATE to enum ib_wc_flags to mark completions of "send with invalidate" operations. Signed-off-by: Eli Cohen --- include/rdma/ib_verbs.h | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 911a661..57a11f8 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -424,7 +424,8 @@ enum ib_wc_opcode { enum ib_wc_flags { IB_WC_GRH = 1, - IB_WC_WITH_IMM = (1<<1) + IB_WC_WITH_IMM = (1<<1), + IB_WC_WITH_INVALIDATE = (1<<2), }; struct ib_wc { -- 1.5.5.1 From eli at dev.mellanox.co.il Sun Jun 1 08:05:08 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 01 Jun 2008 18:05:08 +0300 Subject: [ofa-general] [PATCH 2/2 v3] IB/mlx4: Add send with invalidate support Message-ID: <1212332708.4433.16.camel@mtls03> >From f463e695702406987a0365ebbaae0a6cf344fd23 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Sun, 11 May 2008 15:02:04 +0300 Subject: Add send with invalidate support to mlx4. Signed-off-by: Eli Cohen --- Changes since v2: use query dev cap to check device capabilities regarding send with invalidate. drivers/infiniband/hw/mlx4/cq.c | 8 ++++++++ drivers/infiniband/hw/mlx4/main.c | 2 ++ drivers/infiniband/hw/mlx4/qp.c | 22 +++++++++++++++++----- drivers/net/mlx4/fw.c | 9 +-------- drivers/net/mlx4/fw.h | 2 +- drivers/net/mlx4/mr.c | 6 ++++-- include/linux/mlx4/device.h | 7 ++++++- 7 files changed, 39 insertions(+), 17 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c index 4521319..291e856 100644 --- a/drivers/infiniband/hw/mlx4/cq.c +++ b/drivers/infiniband/hw/mlx4/cq.c @@ -637,6 +637,7 @@ repoll: case MLX4_OPCODE_SEND_IMM: wc->wc_flags |= IB_WC_WITH_IMM; case MLX4_OPCODE_SEND: + case MLX4_OPCODE_SEND_INVAL: wc->opcode = IB_WC_SEND; break; case MLX4_OPCODE_RDMA_READ: @@ -676,6 +677,13 @@ repoll: wc->wc_flags = IB_WC_WITH_IMM; wc->imm_data = cqe->immed_rss_invalid; break; + case MLX4_RECV_OPCODE_SEND_INVAL: + wc->opcode = IB_WC_RECV; + wc->wc_flags = IB_WC_WITH_INVALIDATE; + /* + * TBD: maybe we should just call this ieth_val + */ + wc->imm_data = be32_to_cpu(cqe->immed_rss_invalid); } wc->slid = be16_to_cpu(cqe->rlid); diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 4d61e32..a574050 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -103,6 +103,8 @@ static int mlx4_ib_query_device(struct ib_device *ibdev, props->device_cap_flags |= IB_DEVICE_UD_IP_CSUM; if (dev->dev->caps.max_gso_sz) props->device_cap_flags |= IB_DEVICE_UD_TSO; + if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_RI) + props->device_cap_flags |= IB_DEVICE_SEND_W_INV; props->vendor_id = be32_to_cpup((__be32 *) (out_mad->data + 36)) & 0xffffff; diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index a80df22..293b57e 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -78,6 +78,7 @@ static const __be32 mlx4_ib_opcode[] = { [IB_WR_RDMA_READ] = __constant_cpu_to_be32(MLX4_OPCODE_RDMA_READ), [IB_WR_ATOMIC_CMP_AND_SWP] = __constant_cpu_to_be32(MLX4_OPCODE_ATOMIC_CS), [IB_WR_ATOMIC_FETCH_AND_ADD] = __constant_cpu_to_be32(MLX4_OPCODE_ATOMIC_FA), + [IB_WR_SEND_WITH_INV] = __constant_cpu_to_be32(MLX4_OPCODE_SEND_INVAL), }; static struct mlx4_ib_sqp *to_msqp(struct mlx4_ib_qp *mqp) @@ -1447,6 +1448,21 @@ static int build_lso_seg(struct mlx4_lso_seg *wqe, struct ib_send_wr *wr, return 0; } +static __be32 get_ieth(struct ib_send_wr *wr) +{ + switch (wr->opcode) { + case IB_WR_SEND_WITH_IMM: + case IB_WR_RDMA_WRITE_WITH_IMM: + return wr->ex.imm_data; + + case IB_WR_SEND_WITH_INV: + return cpu_to_be32(wr->ex.invalidate_rkey); + + default: + return 0; + } +} + int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, struct ib_send_wr **bad_wr) { @@ -1493,11 +1509,7 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, MLX4_WQE_CTRL_TCP_UDP_CSUM) : 0) | qp->sq_signal_bits; - if (wr->opcode == IB_WR_SEND_WITH_IMM || - wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM) - ctrl->imm = wr->ex.imm_data; - else - ctrl->imm = 0; + ctrl->imm = get_ieth(wr); wqe += sizeof *ctrl; size = sizeof *ctrl / 16; diff --git a/drivers/net/mlx4/fw.c b/drivers/net/mlx4/fw.c index d82f275..02bff1d 100644 --- a/drivers/net/mlx4/fw.c +++ b/drivers/net/mlx4/fw.c @@ -170,7 +170,7 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) #define QUERY_DEV_CAP_C_MPT_ENTRY_SZ_OFFSET 0x8e #define QUERY_DEV_CAP_MTT_ENTRY_SZ_OFFSET 0x90 #define QUERY_DEV_CAP_D_MPT_ENTRY_SZ_OFFSET 0x92 -#define QUERY_DEV_CAP_BMME_FLAGS_OFFSET 0x97 +#define QUERY_DEV_CAP_BMME_FLAGS_OFFSET 0x94 #define QUERY_DEV_CAP_RSVD_LKEY_OFFSET 0x98 #define QUERY_DEV_CAP_MAX_ICM_SZ_OFFSET 0xa0 @@ -345,13 +345,6 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) } } - if (dev_cap->bmme_flags & 1) - mlx4_dbg(dev, "Base MM extensions: yes " - "(flags %d, rsvd L_Key %08x)\n", - dev_cap->bmme_flags, dev_cap->reserved_lkey); - else - mlx4_dbg(dev, "Base MM extensions: no\n"); - /* * Each UAR has 4 EQ doorbells; so if a UAR is reserved, then * we can't use any EQs whose doorbell falls on that page, diff --git a/drivers/net/mlx4/fw.h b/drivers/net/mlx4/fw.h index 306cb9b..11544d5 100644 --- a/drivers/net/mlx4/fw.h +++ b/drivers/net/mlx4/fw.h @@ -93,7 +93,7 @@ struct mlx4_dev_cap { int cmpt_entry_sz; int mtt_entry_sz; int resize_srq; - u8 bmme_flags; + u32 bmme_flags; u32 reserved_lkey; u64 max_icm_sz; int max_gso_sz; diff --git a/drivers/net/mlx4/mr.c b/drivers/net/mlx4/mr.c index 03a9abc..e78f53d 100644 --- a/drivers/net/mlx4/mr.c +++ b/drivers/net/mlx4/mr.c @@ -47,7 +47,7 @@ struct mlx4_mpt_entry { __be32 flags; __be32 qpn; __be32 key; - __be32 pd; + __be32 pd_flags; __be64 start; __be64 length; __be32 lkey; @@ -71,6 +71,8 @@ struct mlx4_mpt_entry { #define MLX4_MPT_STATUS_SW 0xF0 #define MLX4_MPT_STATUS_HW 0x00 +#define MLX4_MPT_FLAG_EN_INV 0x3000000 + static u32 mlx4_buddy_alloc(struct mlx4_buddy *buddy, int order) { int o; @@ -320,7 +322,7 @@ int mlx4_mr_enable(struct mlx4_dev *dev, struct mlx4_mr *mr) mr->access); mpt_entry->key = cpu_to_be32(key_to_hw_index(mr->key)); - mpt_entry->pd = cpu_to_be32(mr->pd); + mpt_entry->pd_flags = cpu_to_be32(mr->pd | MLX4_MPT_FLAG_EN_INV); mpt_entry->start = cpu_to_be64(mr->iova); mpt_entry->length = cpu_to_be64(mr->size); mpt_entry->entity_size = cpu_to_be32(mr->mtt.page_shift); diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index a744383..b7eccca 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -65,7 +65,12 @@ enum { MLX4_DEV_CAP_FLAG_ATOMIC = 1 << 18, MLX4_DEV_CAP_FLAG_RAW_MCAST = 1 << 19, MLX4_DEV_CAP_FLAG_UD_AV_PORT = 1 << 20, - MLX4_DEV_CAP_FLAG_UD_MCAST = 1 << 21 + MLX4_DEV_CAP_FLAG_UD_MCAST = 1 << 21, + MLX4_DEV_CAP_FLAG_LI = 1 << 6, + MLX4_DEV_CAP_FLAG_RI = 1 << 7, + MLX4_DEV_CAP_FLAG_TYPE2WIN = 1 << 9, + MLX4_DEV_CAP_FLAG_RLKEY = 1 << 10, + MLX4_DEV_CAP_FLAG_FRWR = 1 << 11 }; enum mlx4_event { -- 1.5.5.1 From ronli at voltaire.com Sun Jun 1 09:26:46 2008 From: ronli at voltaire.com (Ron Livne) Date: Sun, 1 Jun 2008 19:26:46 +0300 Subject: [ofa-general] [PATCH] fix for ia64 unalignment umad. Message-ID: This is a fix for ia64 unalignment umad. Ron -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: alignment_mad_ia64.patch Type: application/octet-stream Size: 898 bytes Desc: alignment_mad_ia64.patch URL: From sashak at voltaire.com Sun Jun 1 09:33:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 1 Jun 2008 19:33:29 +0300 Subject: [ofa-general] Re: [PATCH] fix for ia64 unalignment umad. In-Reply-To: References: Message-ID: <20080601163329.GZ4616@sashak.voltaire.com> On 19:26 Sun 01 Jun , Ron Livne wrote: > This is a fix for ia64 unalignment umad. Nice fix! Applied. Thanks. Sasha From marcel.heinz at informatik.tu-chemnitz.de Sun Jun 1 10:18:44 2008 From: marcel.heinz at informatik.tu-chemnitz.de (Marcel Heinz) Date: Sun, 01 Jun 2008 19:18:44 +0200 Subject: [ofa-general] Multicast Performance In-Reply-To: <20080531150922.GL22418@sashak.voltaire.com> References: <4836E231.4000601@informatik.tu-chemnitz.de> <48371B2D.3040908@gmail.com> <483A7E40.5040407@informatik.tu-chemnitz.de> <483BBBDB.6000605@informatik.tu-chemnitz.de> <483DA512.2070403@gmail.com> <483E7520.1000302@informatik.tu-chemnitz.de> <1212065181.27600.96.camel@hrosenstock-ws.xsigo.com> <483EB11A.5000000@informatik.tu-chemnitz.de> <1212068243.17997.48.camel@hrosenstock-ws.xsigo.com> <483EBE95.60901@informatik.tu-chemnitz.de> <20080531150922.GL22418@sashak.voltaire.com> Message-ID: <4842D9F4.3000009@informatik.tu-chemnitz.de> Hi, Sasha Khapyorsky wrote: > On 16:32 Thu 29 May , Marcel Heinz wrote: >> >> Now, all 3 instances measure 950MB/s throughput. >> >> The returned MCMember Records are absolutely identical except >> for the PortGid and the membership state. > > So the difference is only membership. If you have just 2 full member > instances could you see performance degradation? I've already tried with the client as FullMember, and it didn't change anything. Regards, Marcel From olaf.kirch at ORACLE.COM Mon Jun 2 01:25:44 2008 From: olaf.kirch at ORACLE.COM (Olaf Kirch) Date: Mon, 2 Jun 2008 10:25:44 +0200 Subject: [ofa-general] [PATCH] rds: use separate ports for TCP and IB In-Reply-To: References: <20080528225549.GC6288@opengridcomputing.com> Message-ID: <200806021025.46789.olaf.kirch@oracle.com> On Friday 30 May 2008 18:29:01 Talpey, Thomas wrote: > Yes, at the moment TCP to the iWARP NIC is like talking to a different > host. But, RDMA-aware versions of a given protocol still need a second > port, unless there is explicit upper layer support for initiating the MPA > exchange. We have the same issue with NFSv3/RDMA, and we have > applied for a second port (the application is still pending within IANA). This may also become a problem for RDS, but in a different sense. If you bind an RDS socket to a specific IP address, this also selects a transport. If you bind to an IP address owned by ib0, you select the IB transport. If you bind to an IP address owned by say eth0, you select the TCP transport. Now if the one and only NIC in the system is an iWARP NIC, which transport should we select? Both iWARP and TCP can be legitimate choices, depending on the hosts we're talking to (and how do we find out whether the remote node supports iWARP or not?) I think for now it's okay to default to iWARP and punt if the remote doesn't support it. But in the long run this is something that needs to be addressed - either in RDS, or in the way ofed handles iWARP NICs. In fact it may not be such a bad idea to treat an iWARP NIC as two devices, and register one ethX device (owned exclusively by the normal stack) and one ibX device (owned mostly by the ofed stack, maybe with the exception of ARP and such). Would that work with protocols like NFS/RDMA neogtiation - ie can you negotiate specify a secondary port _and_ address? Olaf -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play okir at lst.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax From vlad at lists.openfabrics.org Mon Jun 2 03:18:05 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 2 Jun 2008 03:18:05 -0700 (PDT) Subject: [ofa-general] ofa_1_3_kernel 20080602-0200 daily build status Message-ID: <20080602101805.4492DE60D00@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_3/linux-2.6.git git_branch: ofed_kernel Common build parameters: --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-user_mad-mod --with-user_access-mod --with-mthca-mod --with-mlx4-mod --with-core-mod --with-addr_trans-mod --with-rds-mod --with-cxgb3-mod --with-nes-mod Passed: Passed on i686 with 2.6.15-23-server Passed on i686 with linux-2.6.13 Passed on i686 with linux-2.6.12 Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.15 Passed on i686 with linux-2.6.14 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on x86_64 with linux-2.6.12 Passed on x86_64 with linux-2.6.14 Passed on x86_64 with linux-2.6.15 Passed on x86_64 with linux-2.6.13 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.12 Passed on ia64 with linux-2.6.13 Passed on ia64 with linux-2.6.14 Passed on ia64 with linux-2.6.15 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.23 Passed on powerpc with linux-2.6.12 Passed on powerpc with linux-2.6.13 Passed on powerpc with linux-2.6.14 Passed on powerpc with linux-2.6.15 Passed on ppc64 with linux-2.6.13 Passed on ppc64 with linux-2.6.12 Passed on ppc64 with linux-2.6.15 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.14 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.24 Passed on ppc64 with linux-2.6.19 Failed: From eli at dev.mellanox.co.il Mon Jun 2 03:37:38 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 02 Jun 2008 13:37:38 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Initialize DS field for stamping Message-ID: <1212403058.17351.7.camel@mtls03> >From fe01c4c153fb3126a8c21a5297da581d1fb711e6 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Mon, 2 Jun 2008 11:09:30 +0300 Subject: [PATCH] IB/mlx4: Initialize DS field for stamping When modifying a QP to from reset to init, make sure the DS field is initialized to the max descriptor size so that the subsequent stamping will be made on the entire descriptor area. Signed-off-by: Eli Cohen --- drivers/infiniband/hw/mlx4/qp.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index cb2f456..c61c3e9 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -1065,6 +1065,8 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp, for (i = 0; i < qp->sq.wqe_cnt; ++i) { ctrl = get_send_wqe(qp, i); ctrl->owner_opcode = cpu_to_be32(1 << 31); + if (qp->sq_max_wqes_per_wr < 2) + ctrl->fence_size = 1 << (qp->sq.wqe_shift - 4); stamp_send_wqe(qp, i, 1 << qp->sq.wqe_shift); } -- 1.5.5.1 From at165303 at ohio.edu Mon Jun 2 04:26:22 2008 From: at165303 at ohio.edu (British National Lottery Board) Date: Mon, 02 Jun 2008 07:26:22 -0400 Subject: [ofa-general] Congratulations***Your Email Address Has Won*** Message-ID: <1212405982.4843d8de9b188@webmail.ohio.edu> Contact our fiduciary agent for claims with: ============================================== Agent Name: Mr. Ken Smith Tel::+44-703-197-1576 +44-703-197-2053 Email:( mr.kensmith11 at yahoo.ie ) ============================================== This is to inform you that you have been selected for a cash prize of �1,000,000 (British Pounds) held on the 21st of may 2008 in London Uk.The selection process was carried out through random selection in our computerized email selection system(ess) from a database of over 250,000 email addresses drawn from which you were selected. Fill the below: 1. Name: 2. Address 3. Marital Status: 4. Occupation: 5. Age: 6. Sex: 7. Nationality: 8. Country of Residence: 9. Telephone Number: Yours Truly, Mrs. Stella Ellis. Co-ordinator(Online Promo Programme From Feed at lists.openfabrics.org Mon Jun 2 05:19:48 2008 From: Feed at lists.openfabrics.org (Feed at lists.openfabrics.org) Date: 02 Jun 2008 05:19:48 -0700 Subject: [ofa-general] Feed Blaster puts your ad right to the screens of millions in 15 Minutes ! Message-ID: <20080602051947.E73ED8B43EFCFF6A@from.header.has.no.domain> More and more people are subscribing to feeds every day and there are millions who are already subscribed. Thus, your ad will reach a very broad range of potential customers with each use of Feed Blaster! Feed Blaster is the first & only submitter that can submit your ads to thousands of feeds within a few minutes! Post your ads where people read them! - What if you could place your ad into all these feeds ? Right, that would mean you would have millions of sites linking to your ad - and millions of users reading your message within minutes - and my idea actually works For Full details please read the attached .html file Unsubscribe: On Full details page click on contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Mon Jun 2 07:00:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 07:00:29 -0700 Subject: [ofa-general] [PATCH] OpenSM release notes: Update to 3.1.11 Message-ID: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> OpenSM release notes: Update to 3.1.11 Please apply to ofed_1_3 and master Signed-off-by: Hal Rosenstock diff --git a/opensm/doc/opensm_release_notes-3.1.10.txt b/opensm/doc/opensm_release_notes-3.1.10.txt deleted file mode 100644 index 2b6253d..0000000 --- a/opensm/doc/opensm_release_notes-3.1.10.txt +++ /dev/null @@ -1,492 +0,0 @@ - OpenSM Release Notes 3.1.10 - ============================= - -Version: OpenFabrics Enterprise Distribution (OFED) 1.3 -Repo: git://git.openfabrics.org/~ofed_1_3/management.git (release) - git://git.openfabrics.org/~sashak/management.git (development) -Date: February 2008 - -1 Overview ----------- -This document describes the contents of the OpenSM OFED 1.3 release. -OpenSM is an InfiniBand compliant Subnet Manager and Administration, -and runs on top of OpenIB. The OpenSM version for this release -is openib-3.1.10 - -This document includes the following sections: -1 This Overview section (describing new features and software - dependencies) -2 Known Issues And Limitations -3 Unsupported IB compliance statements -4 Major Bug Fixes -5 Main Verification Flows -6 Qualified software stacks and devices - -1.1 Major New Features - -* QoS manager (experimental) - This QoS manager implementation is in accordance with IBA QoS Annex. - Highly configurable QoS Policy is parsed from OpenSM QoS policy file. - Valid QoS parameters will be reported in SA PathRecord and - MultiPathRecord. In addition simple QoS levels per ULPs configuration - is supported too. - -* Performance Manager - When enabled it collects a fabric port counters and able to log it or - to pass to external program via event plugin interface. It handles - counters overflow, supports LID/QP redirection and is able to work - when OpenSM is in master, standby, and inactive states. - -* Dimension Order routing (DOR) algorithm - DOR Unicast routing algorithm - based on the Min Hop algorithm, but - avoids port equalization except for redundant links between the - same two switches. This provides deadlock free routes for hypercubes - when the fabric is cabled as a hypercube and for meshes when cabled - as a mesh (see details in OpenSM man page). - -* Routing improvements - Speedup the current routing algorithms default MinHops, Up/Down and - LASH and lid matrix generation. Fat Tree routing engine is able to work - with not pure fat free topology. - -* Multiple IB routers support - OpenSM now able to keep configurable subnet prefix to router table. - SA will report path to this routers when SA PathRecord was issued with - non-local DGID. - -* Node map - This is possible to name nodes in this config file. Those names will be - used for logging and by QoS configuration. - -* PKey index support - Proper support for PKey index in GSI queries. - -* Incremental LFTs, PKey, SL2VL, and VLarbitration table updates - OpenSM will only fetch those tables in first heavy sweep and then - will maintain this internally. - -* Fast port and switch detector - When port and/or switch was externally reset and it was fast so sweep - doesn't find this device as disconnected OpenSM will detect this by - changed port states and handle accordingly. - -* Duplicated GUIDs/port moving detector - OpenSM will be able to detect port moving during a fabric discovery - and will not report duplicated GUIDs in this case. - -* Multicast rerouting speedup - Now OpenSM will calculate and setup multicast forwarding tables for - all altered multicast groups and not for each one. - -* Event plugin API - OpenSM allows to load dynamically various plugin modules. - -* Many generic improvements - -1.2 Minor New Features: - -* Daemon mode can be activated with -B option. - -* Support multiple scopes for IPoIB multicast groups in partition config. - -* Loopback connection handling - Loopback connection is not interpreted as duplicated GUID anymore. - -* Connect root nodes option for Up/Down routing engine. - When this option is specified Up/Down will create routing paths between - its root nodes. - -* Dump and log filenames changed from osm* to opensm*. - -* Support loopback console - Socket console with only local access. - -* Configurable config directory (the default value is /etc/opensm) and - configurable default values of OpenSM config filenames. - -* Add option for force SDR link speed - Add option to opensm.opts to force link speed. Currently, only forcing - to SDR link speed is supported. This option is not supported as a - command line option. - -* Better packaging - Building and RPM packaging were improved and simplified. - -* Handle "babbling" ports - When a babbling port (port which causes a frequent trap generation) is - detected, OpenSM will disable the port which should terminate the trap - storm. - -1.3 Library API Changes - - None - -1.4 Software Dependencies - -OpenSM depends on the installation of either OFED 1.3, OFED 1.2, OFED 1.1, -OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD -distribution), or Mellanox VAPI stacks. The qualified driver versions -are provided in Table 2, "Qualified IB Stacks". - -Also building of QoS manager policy file parser requires flex, and either -bison or byacc installed. - -1.5 Supported Devices Firmware - -The main task of OpenSM is to initialize InfiniBand devices. The -qualified devices and their corresponding firmware versions -are listed in Table 3. - -2 Known Issues And Limitations ------------------------------- - -* No Service / Key associations: - There is no way to manage Service access by Keys. - -* No SM to SM SMDB synchronization: - Puts the burden of re-registering services, multicast groups, and - inform-info on the client application (or IB access layer core). - -3 Unsupported IB Compliance Statements --------------------------------------- -The following section lists all the IB compliance statements which -OpenSM does not support. Please refer to the IB specification for detailed -information regarding each compliance statement. - -* C14-22 (Authentication): - M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one - SubnSet method. As a work-around, an OpenSM option is provided for - defining the protect bits. - -* C14-67 (Authentication): - On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then - the SM shall generate a SubnGetResp if the M_Key matches, or - silently drop the packet if M_Key does not match. - -* C15-0.1.23.4 (Authentication): - InformInfoRecords shall always be provided with the QPN set to 0, - except for the case of a trusted request, in which case the actual - subscriber QPN shall be returned. - -* o13-17.1.2 (Event-FWD): - If no permission to forward, the subscription should be removed and - no further forwarding should occur. - -* C14-24.1.1.5 and C14-62.1.1.22 (Initialization): - GUIDInfo - SM should enable assigning Port GUIDInfo. - -* C14-44 (Initialization): - If the SM discovers that it is missing an M_Key to update CA/RT/SW, - it should notify the higher level. - -* C14-62.1.1.12 (Initialization): - PortInfo:M_Key - Set the M_Key to a node based random value. - -* C14-62.1.1.13 (Initialization): - PortInfo:P_KeyProtectBits - set according to an optional policy. - -* C14-62.1.1.24 (Initialization): - SwitchInfo:DefaultPort - should be configured for random FDB. - -* C14-62.1.1.32 (Initialization): - RandomForwardingTable should be configured. - -* o15-0.1.12 (Multicast): - If the JoinState is SendOnlyNonMember = 1 (only), then the endport - should join as sender only. - -* o15-0.1.8 (Multicast): - If a request for creating an MCG with fields that cannot be met, - return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass). - -* C15-0.1.8.6 (SA-Query): - Respond to SubnAdmGetTraceTable - this is an optional attribute. - -* C15-0.1.13 Services: - Reject ServiceRecord create, modify or delete if the given - ServiceP_Key does not match the one included in the ServiceGID port - and the port that sent the request. - -* C15-0.1.14 (Services): - Provide means to associate service name and ServiceKeys. - -4 Major Bug Fixes ------------------ - -The following is a list of bugs that were fixed. Note that other less critical -or visible bugs were also fixed. - -* osm_ucast_ftree.c: do load-leveling of non-CN routes - -* osm_ucast_ftree.c: ignore port 0 and loopbacks on switches - -* lash: fix possible segfault in osm_get_lash_sl() - -* osm_ucast_ftree.c: fixing coredump in fat-tree routing - -* osm_sa_slvl_record: fix overflow crash - -* Break multicast rerouting requests processing when heavy sweep is - scheduled. - -* updn: report fallback properly - -* Fix incorrect identification of routing engine used - -* Don't zero base LID when invalid value is received - -* lash: fix wrong allocation size - -* Fixing broken logic in 'process world' part of LinkRecord processing - -* Fix lmc_mask bit order in osm_sa_link_record.c - -* Adding missing comparison by to_lid/from_lid in LinkRecord processing - -* Broken logic when scanning subnet for PIR request - -* No interactive games in daemon mode - -* Fixing memory leak in node description - -* Fix PortInfo update issues for switch port 0 - -* Changed method_mask type in user_mad interface in accordance with - kernel ABI - -* Use umad_get_issm_path() in osm_vendor_set_sm() - -* Report message fix - -* Uninitialized variables usage fix - -* osm_ucast_ftree.c: Possible NULL ptr seg fault - -* osm_mcast_mgr.c: Possible NULL ptr seg fault - -* TrapRepress was failing for mkey != 0 - -* IB_PR_COMPMASK was used in MPR - -* Set hop limit when creating ipoib multicast groups - -* Fix outstanding mad counters tracking on the error paths. - -* Report new ports before handover mastership - -* Fix opvls and neighbormtu when remote port invalid. - -* Bug in coding trying to set vl_arb_high_limit when PortInfo.base_lid - was still zero. - -* Protect SMInfo response against port moving issue. - -5 Main Verification Flows -------------------------- - -OpenSM verification is run using the following activities: -* osmtest - a stand-alone program -* ibmgtsim (IB management simulator) based - a set of flows that - simulate clusters, inject errors and verify OpenSM capability to - respond and bring up the network correctly. -* small cluster regression testing - where the SM is used on back to - back or single switch configurations. The regression includes - multiple OpenSM dedicated tests. -* cluster testing - when we run OpenSM to setup a large cluster, perform - hand-off, reboots and reconnects, verify routing correctness and SA - responsiveness at the ULP level (IPoIB and SDP). - -5.1 osmtest - -osmtest is an automated verification tool used for OpenSM -testing. Its verification flows are described by list below. - -* Inventory File: Obtain and verify all port info, node info, link and path - records parameters. - -* Service Record: - - Register new service - - Register another service (with a lease period) - - Register another service (with service p_key set to zero) - - Get all services by name - - Delete the first service - - Delete the third service - - Added bad flows of get/delete non valid service - - Add / Get same service with different data - - Add / Get / Delete by different component mask values (services - by Name & Key / Name & Data / Name & Id / Id only ) - -* Multicast Member Record: - - Query of existing Groups (IPoIB) - - BAD Join with insufficient comp mask (o15.0.1.3) - - Create given MGID=0 (o15.0.1.4) - - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4) - - Create BAD MGID=0xFA. (o15.0.1.6) - - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6) - - New MGID with invalid join state (o15.0.1.9) - - Retry of existing MGID - See JoinState update (o15.0.1.11) - - BAD RATE when connecting to existing MGID (o15.0.1.13) - - Partial JoinState delete request - removing FullMember (o15.0.1.14) - - Full Delete of a group (o15.0.1.14) - - Verify Delete by trying to Join deleted group (o15.0.1.14) - - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15) - -* GUIDInfo Record: - - All GUIDInfoRecords in subnet are obtained - -* MultiPathRecord: - - Perform some compliant and noncompliant MultiPathRecord requests - - Validation is via status in responses and IB analyzer - -* PKeyTableRecord: - - Perform some compliant and noncompliant PKeyTableRecord queries - - Validation is via status in responses and IB analyzer - -* LinearForwardingTableRecord: - - Perform some compliant and noncompliant LinearForwardingTableRecord queries - - Validation is via status in responses and IB analyzer - -* Event Forwarding: Register for trap forwarding using reports - - Send a trap and wait for report - - Unregister non-existing - -* Trap 64/65 Flow: Register to Trap 64-65, create traps (by - disconnecting/connecting ports) and wait for report, then unregister. - -* Stress Test: send PortInfoRecord queries, both single and RMPP and - check for the rate of responses as well as their validity. - - -5.2 IB Management Simulator OpenSM Test Flows: - -The simulator provides ability to simulate the SM handling of virtual -topologies that are not limited to actual lab equipment availability. -OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily -regressions use smaller (16 and 128 nodes clusters). - -The following test flows are run on the IB management simulator: - -* Stability: - Up to 12 links from the fabric are randomly selected to drop packets - at drop rates up to 90%. The SM is required to succeed in bringing the - fabric up. The resulting routing is verified to be correct as well. - -* LID Manager: - Using LMC = 2 the fabric is initialized with LIDs. Faults such as - zero LID, Duplicated LID, non-aligned (to LMC) LIDs are - randomly assigned to various nodes and other errors are randomly - output to the guid2lid cache file. The SM sweep is run 5 times and - after each iteration a complete verification is made to ensure that all - LIDs that could possibly be maintained are kept, as well as that all nodes - were assigned a legal LID range. - -* Multicast Routing: - Nodes randomly join the 0xc000 group and eventually the - resulting routing is verified for completeness and adherence to - Up/Down routing rules. - -* osmtest: - The complete osmtest flow as described in the previous table is run on - the simulated fabrics. - -* Stress Test: - This flow merges fabric, LID and stability issues with continuous - PathRecord, ServiceRecord and Multicast Join/Leave activity to - stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get - were added to the test such both existing and non existing nodes - perform them in random order. - -5.3 OpenSM Regression - -Using a back-to-back or single switch connection, the following set of -tests is run nightly on the stacks described in table 2. The included -tests are: - -* Stress Testing: Flood the SA with queries from multiple channel - adapters to check the robustness of the entire stack up to the SA. - -* Dynamic Changes: Dynamic Topology changes, through randomly - dropping SMP packets, used to test OpenSM adaptation to an unstable - network & verify DB correctness. - -* Trap Injection: This flow injects traps to the SM and verifies that it - handles them gracefully. - -* SA Query Test: This test exhaustively checks the SA responses to all - possible single component mask. To do that the test examines the - entire set of records the SA can provide, classifies them by their - field values and then selects every field (using component mask and a - value) and verifies that the response matches the expected set of records. - A random selection using multiple component mask bits is also performed. - -5.4 Cluster testing: - -Cluster testing is usually run before a distribution release. It -involves real hardware setups of 16 to 32 nodes (or more if a beta site -is available). Each test is validated by running all-to-all ping through the IB -interface. The test procedure includes: - -* Cluster bringup - -* Hand-off between 2 or 3 SM's while performing: - - Node reboots - - Switch power cycles (disconnecting the SM's) - -* Unresponsive port detection and recovery - -* osmtest from multiple nodes - -* Trap injection and recovery - - -6 Qualification ----------------- - -Table 2 - Qualified IB Stacks -============================= - -Stack | Version ------------------------------------------|-------------------------- -OFED | 1.3 -OFED | 1.2 -OFED | 1.1 -OFED | 1.0 -OpenIB Gen2 (IBG2 distribution) | 1.0 -OpenIB Gen1 (IBGD distribution) | 1.8.0 -VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later - -Table 3 - Qualified Devices and Corresponding Firmware -====================================================== - -Mellanox -Device | FW versions -------------------------------------|------------------------------- -InfiniScale | fw-43132 5.2.000 (and later) -InfiniScale III | fw-47396 0.5.000 (and later) -InfiniHost | fw-23108 3.5.000 (and later) -InfiniHost III Lx | fw-25204 1.2.000 (and later) -InfiniHost III Ex (InfiniHost Mode) | fw-25208 4.8.200 (and later) -InfiniHost III Ex (MemFree Mode) | fw-25218 5.3.000 (and later) -ConnectX IB | fw-25408 2.3.000 (and later) - -QLogic/PathScale -Device | Note ---------|----------------------------------------------------------- -iPath | QHT6040 (PathScale InfiniPath HT-460) -iPath | QHT6140 (PathScale InfiniPath HT-465) -iPath | QLE6140 (PathScale InfiniPath PE-880) -iPath | QLE7240 -iPath | QLE7280 - -Note 1: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose -QP0 and QP1. However, it does support it as a device on the subnet. - -Note 2: QoS firmware and Mellanox devices - -HCAs: QoS supported by ConnectX. The current FW release -doesn't support QoS. QoS-enabled FW release (2_5_000) is -planned for May. If someone wishes to get QoS-enabled FW -before the official release, they should contact Mellanox FAE. - -Switches: QoS supported by InfiniScale III -Any InfiniScale III FW that is supported by OpenSM supports QoS. diff --git a/opensm/doc/opensm_release_notes-3.1.11.txt b/opensm/doc/opensm_release_notes-3.1.11.txt new file mode 100644 index 0000000..c7bbb41 --- /dev/null +++ b/opensm/doc/opensm_release_notes-3.1.11.txt @@ -0,0 +1,492 @@ + OpenSM Release Notes 3.1.11 + ============================= + +Version: OpenFabrics Enterprise Distribution (OFED) 1.3 +Repo: git://git.openfabrics.org/~ofed_1_3/management.git (release) + git://git.openfabrics.org/~sashak/management.git (development) +Date: May 2008 + +1 Overview +---------- +This document describes the contents of the OpenSM OFED 1.3 release. +OpenSM is an InfiniBand compliant Subnet Manager and Administration, +and runs on top of OpenIB. The OpenSM version for this release +is openib-3.1.11 + +This document includes the following sections: +1 This Overview section (describing new features and software + dependencies) +2 Known Issues And Limitations +3 Unsupported IB compliance statements +4 Major Bug Fixes +5 Main Verification Flows +6 Qualified software stacks and devices + +1.1 Major New Features + +* QoS manager (experimental) + This QoS manager implementation is in accordance with IBA QoS Annex. + Highly configurable QoS Policy is parsed from OpenSM QoS policy file. + Valid QoS parameters will be reported in SA PathRecord and + MultiPathRecord. In addition simple QoS levels per ULPs configuration + is supported too. + +* Performance Manager + When enabled it collects a fabric port counters and able to log it or + to pass to external program via event plugin interface. It handles + counters overflow, supports LID/QP redirection and is able to work + when OpenSM is in master, standby, and inactive states. + +* Dimension Order routing (DOR) algorithm + DOR Unicast routing algorithm - based on the Min Hop algorithm, but + avoids port equalization except for redundant links between the + same two switches. This provides deadlock free routes for hypercubes + when the fabric is cabled as a hypercube and for meshes when cabled + as a mesh (see details in OpenSM man page). + +* Routing improvements + Speedup the current routing algorithms default MinHops, Up/Down and + LASH and lid matrix generation. Fat Tree routing engine is able to work + with not pure fat free topology. + +* Multiple IB routers support + OpenSM now able to keep configurable subnet prefix to router table. + SA will report path to this routers when SA PathRecord was issued with + non-local DGID. + +* Node map + This is possible to name nodes in this config file. Those names will be + used for logging and by QoS configuration. + +* PKey index support + Proper support for PKey index in GSI queries. + +* Incremental LFTs, PKey, SL2VL, and VLarbitration table updates + OpenSM will only fetch those tables in first heavy sweep and then + will maintain this internally. + +* Fast port and switch detector + When port and/or switch was externally reset and it was fast so sweep + doesn't find this device as disconnected OpenSM will detect this by + changed port states and handle accordingly. + +* Duplicated GUIDs/port moving detector + OpenSM will be able to detect port moving during a fabric discovery + and will not report duplicated GUIDs in this case. + +* Multicast rerouting speedup + Now OpenSM will calculate and setup multicast forwarding tables for + all altered multicast groups and not for each one. + +* Event plugin API + OpenSM allows to load dynamically various plugin modules. + +* Many generic improvements + +1.2 Minor New Features: + +* Daemon mode can be activated with -B option. + +* Support multiple scopes for IPoIB multicast groups in partition config. + +* Loopback connection handling + Loopback connection is not interpreted as duplicated GUID anymore. + +* Connect root nodes option for Up/Down routing engine. + When this option is specified Up/Down will create routing paths between + its root nodes. + +* Dump and log filenames changed from osm* to opensm*. + +* Support loopback console + Socket console with only local access. + +* Configurable config directory (the default value is /etc/opensm) and + configurable default values of OpenSM config filenames. + +* Add option for force SDR link speed + Add option to opensm.opts to force link speed. Currently, only forcing + to SDR link speed is supported. This option is not supported as a + command line option. + +* Better packaging + Building and RPM packaging were improved and simplified. + +* Handle "babbling" ports + When a babbling port (port which causes a frequent trap generation) is + detected, OpenSM will disable the port which should terminate the trap + storm. + +1.3 Library API Changes + + None + +1.4 Software Dependencies + +OpenSM depends on the installation of either OFED 1.3, OFED 1.2, OFED 1.1, +OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD +distribution), or Mellanox VAPI stacks. The qualified driver versions +are provided in Table 2, "Qualified IB Stacks". + +Also building of QoS manager policy file parser requires flex and bison +installed. + +1.5 Supported Devices Firmware + +The main task of OpenSM is to initialize InfiniBand devices. The +qualified devices and their corresponding firmware versions +are listed in Table 3. + +2 Known Issues And Limitations +------------------------------ + +* No Service / Key associations: + There is no way to manage Service access by Keys. + +* No SM to SM SMDB synchronization: + Puts the burden of re-registering services, multicast groups, and + inform-info on the client application (or IB access layer core). + +3 Unsupported IB Compliance Statements +-------------------------------------- +The following section lists all the IB compliance statements which +OpenSM does not support. Please refer to the IB specification for detailed +information regarding each compliance statement. + +* C14-22 (Authentication): + M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one + SubnSet method. As a work-around, an OpenSM option is provided for + defining the protect bits. + +* C14-67 (Authentication): + On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then + the SM shall generate a SubnGetResp if the M_Key matches, or + silently drop the packet if M_Key does not match. + +* C15-0.1.23.4 (Authentication): + InformInfoRecords shall always be provided with the QPN set to 0, + except for the case of a trusted request, in which case the actual + subscriber QPN shall be returned. + +* o13-17.1.2 (Event-FWD): + If no permission to forward, the subscription should be removed and + no further forwarding should occur. + +* C14-24.1.1.5 and C14-62.1.1.22 (Initialization): + GUIDInfo - SM should enable assigning Port GUIDInfo. + +* C14-44 (Initialization): + If the SM discovers that it is missing an M_Key to update CA/RT/SW, + it should notify the higher level. + +* C14-62.1.1.12 (Initialization): + PortInfo:M_Key - Set the M_Key to a node based random value. + +* C14-62.1.1.13 (Initialization): + PortInfo:P_KeyProtectBits - set according to an optional policy. + +* C14-62.1.1.24 (Initialization): + SwitchInfo:DefaultPort - should be configured for random FDB. + +* C14-62.1.1.32 (Initialization): + RandomForwardingTable should be configured. + +* o15-0.1.12 (Multicast): + If the JoinState is SendOnlyNonMember = 1 (only), then the endport + should join as sender only. + +* o15-0.1.8 (Multicast): + If a request for creating an MCG with fields that cannot be met, + return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass). + +* C15-0.1.8.6 (SA-Query): + Respond to SubnAdmGetTraceTable - this is an optional attribute. + +* C15-0.1.13 Services: + Reject ServiceRecord create, modify or delete if the given + ServiceP_Key does not match the one included in the ServiceGID port + and the port that sent the request. + +* C15-0.1.14 (Services): + Provide means to associate service name and ServiceKeys. + +4 Major Bug Fixes +----------------- + +The following is a list of bugs that were fixed. Note that other less critical +or visible bugs were also fixed. + +* osm_ucast_ftree.c: do load-leveling of non-CN routes + +* osm_ucast_ftree.c: ignore port 0 and loopbacks on switches + +* lash: fix possible segfault in osm_get_lash_sl() + +* osm_ucast_ftree.c: fixing coredump in fat-tree routing + +* osm_sa_slvl_record: fix overflow crash + +* Break multicast rerouting requests processing when heavy sweep is + scheduled. + +* updn: report fallback properly + +* Fix incorrect identification of routing engine used + +* Don't zero base LID when invalid value is received + +* lash: fix wrong allocation size + +* Fixing broken logic in 'process world' part of LinkRecord processing + +* Fix lmc_mask bit order in osm_sa_link_record.c + +* Adding missing comparison by to_lid/from_lid in LinkRecord processing + +* Broken logic when scanning subnet for PIR request + +* No interactive games in daemon mode + +* Fixing memory leak in node description + +* Fix PortInfo update issues for switch port 0 + +* Changed method_mask type in user_mad interface in accordance with + kernel ABI + +* Use umad_get_issm_path() in osm_vendor_set_sm() + +* Report message fix + +* Uninitialized variables usage fix + +* osm_ucast_ftree.c: Possible NULL ptr seg fault + +* osm_mcast_mgr.c: Possible NULL ptr seg fault + +* TrapRepress was failing for mkey != 0 + +* IB_PR_COMPMASK was used in MPR + +* Set hop limit when creating ipoib multicast groups + +* Fix outstanding mad counters tracking on the error paths. + +* Report new ports before handover mastership + +* Fix opvls and neighbormtu when remote port invalid. + +* Bug in coding trying to set vl_arb_high_limit when PortInfo.base_lid + was still zero. + +* Protect SMInfo response against port moving issue. + +5 Main Verification Flows +------------------------- + +OpenSM verification is run using the following activities: +* osmtest - a stand-alone program +* ibmgtsim (IB management simulator) based - a set of flows that + simulate clusters, inject errors and verify OpenSM capability to + respond and bring up the network correctly. +* small cluster regression testing - where the SM is used on back to + back or single switch configurations. The regression includes + multiple OpenSM dedicated tests. +* cluster testing - when we run OpenSM to setup a large cluster, perform + hand-off, reboots and reconnects, verify routing correctness and SA + responsiveness at the ULP level (IPoIB and SDP). + +5.1 osmtest + +osmtest is an automated verification tool used for OpenSM +testing. Its verification flows are described by list below. + +* Inventory File: Obtain and verify all port info, node info, link and path + records parameters. + +* Service Record: + - Register new service + - Register another service (with a lease period) + - Register another service (with service p_key set to zero) + - Get all services by name + - Delete the first service + - Delete the third service + - Added bad flows of get/delete non valid service + - Add / Get same service with different data + - Add / Get / Delete by different component mask values (services + by Name & Key / Name & Data / Name & Id / Id only ) + +* Multicast Member Record: + - Query of existing Groups (IPoIB) + - BAD Join with insufficient comp mask (o15.0.1.3) + - Create given MGID=0 (o15.0.1.4) + - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4) + - Create BAD MGID=0xFA. (o15.0.1.6) + - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6) + - New MGID with invalid join state (o15.0.1.9) + - Retry of existing MGID - See JoinState update (o15.0.1.11) + - BAD RATE when connecting to existing MGID (o15.0.1.13) + - Partial JoinState delete request - removing FullMember (o15.0.1.14) + - Full Delete of a group (o15.0.1.14) + - Verify Delete by trying to Join deleted group (o15.0.1.14) + - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15) + +* GUIDInfo Record: + - All GUIDInfoRecords in subnet are obtained + +* MultiPathRecord: + - Perform some compliant and noncompliant MultiPathRecord requests + - Validation is via status in responses and IB analyzer + +* PKeyTableRecord: + - Perform some compliant and noncompliant PKeyTableRecord queries + - Validation is via status in responses and IB analyzer + +* LinearForwardingTableRecord: + - Perform some compliant and noncompliant LinearForwardingTableRecord queries + - Validation is via status in responses and IB analyzer + +* Event Forwarding: Register for trap forwarding using reports + - Send a trap and wait for report + - Unregister non-existing + +* Trap 64/65 Flow: Register to Trap 64-65, create traps (by + disconnecting/connecting ports) and wait for report, then unregister. + +* Stress Test: send PortInfoRecord queries, both single and RMPP and + check for the rate of responses as well as their validity. + + +5.2 IB Management Simulator OpenSM Test Flows: + +The simulator provides ability to simulate the SM handling of virtual +topologies that are not limited to actual lab equipment availability. +OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily +regressions use smaller (16 and 128 nodes clusters). + +The following test flows are run on the IB management simulator: + +* Stability: + Up to 12 links from the fabric are randomly selected to drop packets + at drop rates up to 90%. The SM is required to succeed in bringing the + fabric up. The resulting routing is verified to be correct as well. + +* LID Manager: + Using LMC = 2 the fabric is initialized with LIDs. Faults such as + zero LID, Duplicated LID, non-aligned (to LMC) LIDs are + randomly assigned to various nodes and other errors are randomly + output to the guid2lid cache file. The SM sweep is run 5 times and + after each iteration a complete verification is made to ensure that all + LIDs that could possibly be maintained are kept, as well as that all nodes + were assigned a legal LID range. + +* Multicast Routing: + Nodes randomly join the 0xc000 group and eventually the + resulting routing is verified for completeness and adherence to + Up/Down routing rules. + +* osmtest: + The complete osmtest flow as described in the previous table is run on + the simulated fabrics. + +* Stress Test: + This flow merges fabric, LID and stability issues with continuous + PathRecord, ServiceRecord and Multicast Join/Leave activity to + stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get + were added to the test such both existing and non existing nodes + perform them in random order. + +5.3 OpenSM Regression + +Using a back-to-back or single switch connection, the following set of +tests is run nightly on the stacks described in table 2. The included +tests are: + +* Stress Testing: Flood the SA with queries from multiple channel + adapters to check the robustness of the entire stack up to the SA. + +* Dynamic Changes: Dynamic Topology changes, through randomly + dropping SMP packets, used to test OpenSM adaptation to an unstable + network & verify DB correctness. + +* Trap Injection: This flow injects traps to the SM and verifies that it + handles them gracefully. + +* SA Query Test: This test exhaustively checks the SA responses to all + possible single component mask. To do that the test examines the + entire set of records the SA can provide, classifies them by their + field values and then selects every field (using component mask and a + value) and verifies that the response matches the expected set of records. + A random selection using multiple component mask bits is also performed. + +5.4 Cluster testing: + +Cluster testing is usually run before a distribution release. It +involves real hardware setups of 16 to 32 nodes (or more if a beta site +is available). Each test is validated by running all-to-all ping through the IB +interface. The test procedure includes: + +* Cluster bringup + +* Hand-off between 2 or 3 SM's while performing: + - Node reboots + - Switch power cycles (disconnecting the SM's) + +* Unresponsive port detection and recovery + +* osmtest from multiple nodes + +* Trap injection and recovery + + +6 Qualification +---------------- + +Table 2 - Qualified IB Stacks +============================= + +Stack | Version +-----------------------------------------|-------------------------- +OFED | 1.3 +OFED | 1.2 +OFED | 1.1 +OFED | 1.0 +OpenIB Gen2 (IBG2 distribution) | 1.0 +OpenIB Gen1 (IBGD distribution) | 1.8.0 +VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later + +Table 3 - Qualified Devices and Corresponding Firmware +====================================================== + +Mellanox +Device | FW versions +------------------------------------|------------------------------- +InfiniScale | fw-43132 5.2.000 (and later) +InfiniScale III | fw-47396 0.5.000 (and later) +InfiniHost | fw-23108 3.5.000 (and later) +InfiniHost III Lx | fw-25204 1.2.000 (and later) +InfiniHost III Ex (InfiniHost Mode) | fw-25208 4.8.200 (and later) +InfiniHost III Ex (MemFree Mode) | fw-25218 5.3.000 (and later) +ConnectX IB | fw-25408 2.3.000 (and later) + +QLogic/PathScale +Device | Note +--------|----------------------------------------------------------- +iPath | QHT6040 (PathScale InfiniPath HT-460) +iPath | QHT6140 (PathScale InfiniPath HT-465) +iPath | QLE6140 (PathScale InfiniPath PE-880) +iPath | QLE7240 +iPath | QLE7280 + +Note 1: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose +QP0 and QP1. However, it does support it as a device on the subnet. + +Note 2: QoS firmware and Mellanox devices + +HCAs: QoS supported by ConnectX. The current FW release +doesn't support QoS. QoS-enabled FW release (2_5_000) is +planned for May. If someone wishes to get QoS-enabled FW +before the official release, they should contact Mellanox FAE. + +Switches: QoS supported by InfiniScale III +Any InfiniScale III FW that is supported by OpenSM supports QoS. From hrosenstock at xsigo.com Mon Jun 2 07:00:39 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 07:00:39 -0700 Subject: [ofa-general] [PATCH] OpenSM/osm_sa_mcmember_record.c: Validate some more MGID bits for IPv6 SNM Message-ID: <1212415240.32691.32.camel@hrosenstock-ws.xsigo.com> OpenSM/osm_sa_mcmember_record.c: Include high order 24 bits of low 64 MGID bits in validation as these bits are supposed to be 0 Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index c14632d..73b3fc4 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1088,14 +1088,15 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) * YYYYYY is the last 24 bits of the port guid */ #define PREFIX_MASK (0xff10ffff0000ffffULL) #define PREFIX_SIGNATURE (0xff10601b00000000ULL) -#define INT_ID_MASK (0x00000001ff000000ULL) +#define INT_ID_MASK (0xffffff1ff000000ULL) +#define INT_ID_SIGNATURE (0x00000001ff000000ULL) uint64_t g_prefix = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix); uint64_t g_interface_id = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id); uint64_t rcv_prefix = cl_ntoh64(p_recvd_mgid->unicast.prefix); uint64_t rcv_interface_id = cl_ntoh64(p_recvd_mgid->unicast.interface_id); if ((rcv_prefix & PREFIX_MASK) == PREFIX_SIGNATURE && - (rcv_interface_id & INT_ID_MASK) == INT_ID_MASK && + (rcv_interface_id & INT_ID_MASK) == INT_ID_SIGNATURE && (g_prefix & PREFIX_MASK) == (rcv_prefix && PREFIX_MASK) && (g_interface_id & INT_ID_MASK) == From hrosenstock at xsigo.com Mon Jun 2 07:01:16 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 07:01:16 -0700 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <20080531172815.GO22418@sashak.voltaire.com> References: <20080522145607.GE32128@sashak.voltaire.com> <1211469029.18236.188.camel@hrosenstock-ws.xsigo.com> <20080523100634.GD4164@sashak.voltaire.com> <1211541313.13185.80.camel@hrosenstock-ws.xsigo.com> <20080523123414.GB4640@sashak.voltaire.com> <1211547161.13185.103.camel@hrosenstock-ws.xsigo.com> <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> Message-ID: <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> On Sat, 2008-05-31 at 20:28 +0300, Sasha Khapyorsky wrote: > This adds possibility to specify SM_Key value with saquery. It should > work with queries where OSM_DEFAULT_SM_KEY was used. > > If non-numeric string (like 'x') is provided with --smkey option then > saquery will prompt to get SM_Key value. > > Signed-off-by: Sasha Khapyorsky > --- > > SM_key value prompting was added as addition to v1 of the patch. > > infiniband-diags/src/saquery.c | 20 +++++++++++++++++--- man page change is needed for this too. > 1 files changed, 17 insertions(+), 3 deletions(-) > > diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c > index 3d4ab24..d3875fc 100644 > --- a/infiniband-diags/src/saquery.c > +++ b/infiniband-diags/src/saquery.c > @@ -37,6 +37,7 @@ > * > */ > > +#include > #include > #include > #include > @@ -69,6 +70,7 @@ char *argv0 = "saquery"; > > static char *node_name_map_file = NULL; > static nn_map_t *node_name_map = NULL; > +static ib_net64_t smkey = OSM_DEFAULT_SA_KEY; > > /** > * Declare some globals because I don't want this to be too complex. > @@ -730,7 +732,7 @@ get_all_records(osm_bind_handle_t bind_handle, > int trusted) > { > return get_any_records(bind_handle, query_id, 0, 0, NULL, attr_offset, > - trusted ? OSM_DEFAULT_SA_KEY : 0); > + trusted ? smkey : 0); > } > > /** > @@ -1254,8 +1256,7 @@ print_pkey_tbl_records(const struct query_cmd *q, osm_bind_handle_t bind_handle, > > status = get_any_records(bind_handle, IB_MAD_ATTR_PKEY_TBL_RECORD, 0, > comp_mask, &pktr, > - ib_get_attr_offset(sizeof(pktr)), > - OSM_DEFAULT_SA_KEY); > + ib_get_attr_offset(sizeof(pktr)), smkey); > if (status != IB_SUCCESS) > return status; > > @@ -1411,6 +1412,10 @@ usage(void) > "IPv6 format\n"); > fprintf(stderr, " -C specify the SA query HCA\n"); > fprintf(stderr, " -P specify the SA query port\n"); > + fprintf(stderr, " --smkey specify SM_Key value for the query." > + " If non-numeric value \n" > + " (like 'x') is specified then " > + "saquery will prompt for a value\n"); > fprintf(stderr, " -t | --timeout specify the SA query " > "response timeout (default %u msec)\n", > DEFAULT_SA_TIMEOUT_MS); > @@ -1466,6 +1471,7 @@ main(int argc, char **argv) > {"sgid-to-dgid", 1, 0, 2}, > {"timeout", 1, 0, 't'}, > {"node-name-map", 1, 0, 3}, > + {"smkey", 1, 0, 4}, > { } > }; > > @@ -1512,6 +1518,14 @@ main(int argc, char **argv) > case 3: > node_name_map_file = strdup(optarg); > break; > + case 4: > + if (!isxdigit(*optarg) && > + !(optarg = getpass("SM_Key: "))) { Man page for getpass indicates: "This function is obsolete. Do not use it." -- Hal > + fprintf(stderr, "cannot get SM_Key\n"); > + usage(); > + } > + smkey = cl_hton64(strtoull(optarg, NULL, 0)); > + break; > case 'p': > query_type = IB_MAD_ATTR_PATH_RECORD; > break; From hrosenstock at xsigo.com Mon Jun 2 07:06:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 07:06:38 -0700 Subject: [ofa-general] Re: OSM_DEFAULT_SM_KEY byte order In-Reply-To: <20080531214919.GS22418@sashak.voltaire.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> Message-ID: <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> On Sun, 2008-06-01 at 00:49 +0300, Sasha Khapyorsky wrote: > On 07:52 Thu 22 May , Hal Rosenstock wrote: > > > +#define OSM_DEFAULT_SM_KEY CL_HTON64(1) > > > /********/ > > > /****s* OpenSM: Base/OSM_DEFAULT_LMC > > > * NAME > > > > > > > > > , but sort of backward compatibility (currently I know that > > > OSM_DEFAULT_SM_KEY is used with 'osmtest' and 'saquery') could be lost. > > > Is this so important? Ideas? > > > > IMO yes, I think this breaks both backward compatibility and what was > > actually observed from some other SMs during interop testing. > > > > I agree it needs fixing but I think the proper thing is probably more > > like: > > > > #define OSM_DEFAULT_SM_KEY CL_HTON64(0x0100000000000000); > > Using value like this we will break on big endian machines where > originally the value is correct. I think that '1' in network byte order > is better (especially in long term) - it is more "native" non-zero > value. Also I found at least one vendor SM which uses 1 as default SM > key in network byte order (and this is expected, I doubt somebody uses > 0x0100000000000000). This came from informal interop testing a while ago. It wasn't invented out of thin air. > Our own backward compatibility could be solved by configuring sm key > (this will work with OpenSM and saquery). > > Another opinions? I think that third party SMs are a side issue as this is not sanctioned by IBTA and there is other evidence of a vendor SM using SM key. To me, key is back interop with older OpenSMs (at least for x86 as that is the larger part of the installed base) and this is the aspect which is sanctioned by IBTA. -- Hal > Sasha From tziporet at mellanox.co.il Mon Jun 2 07:21:50 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 2 Jun 2008 17:21:50 +0300 Subject: [ofa-general] Agenda for the OFED meeting today (June 2) Message-ID: <6C2C79E72C305246B504CBA17B5500C9042828D3@mtlexch01.mtl.com> Hi, This is the agenda for the OFED meeting today: 1. OFED 1.3.1 status: We inserted one more patch after RC3: http://lists.openfabrics.org/pipermail/general/2008-May/050551.html Regression is in good shape. If everything works fine we will do final GA release tomorrow (June 3) Anyone that had not yet sent the release notes has last opportunity to send them today 2. OFED 1.4: - Kernel rebase status: Backports are available for all kernels between 2.6.26-rc2 and kernel 2.6.16. Work is ongoing and expected to be completed this week. URL: git://git.openfabrics.org/ofed_1_4/linux-2.6.git ofed_kernel - Update from the participants (mainly on new components/features) 3. Open discussion - Elect a chair to the EWG: Decide on a process for electing a chair and a co-chair to the EWG group. - Other topics ... Tziporet From hrosenstock at xsigo.com Mon Jun 2 07:25:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 07:25:29 -0700 Subject: [ofa-general] [PATCH] opensm/osm_lid_mgr.c: Eliminate some potential NULL pointer dereferences Message-ID: <1212416729.32691.43.camel@hrosenstock-ws.xsigo.com> opensm/osm_lid_mgr.c: Eliminate some potential NULL pointer dereferences Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index 7f25750..b5e2b04 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -370,7 +370,8 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) "Skipping all lids as we are reassigning them\n"); p_range = (osm_lid_mgr_range_t *) malloc(sizeof(osm_lid_mgr_range_t)); - p_range->min_lid = 1; + if (p_range) + p_range->min_lid = 1; goto AfterScanningLids; } @@ -557,8 +558,10 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) else { p_range = (osm_lid_mgr_range_t *) malloc(sizeof(osm_lid_mgr_range_t)); - p_range->min_lid = lid; - p_range->max_lid = lid; + if (p_range) { + p_range->min_lid = lid; + p_range->max_lid = lid; + } } } else { /* this lid is used so we need to finalize the previous free range */ @@ -586,13 +589,16 @@ AfterScanningLids: no free range exists and we want to define it after the last mapped lid. */ - p_range->min_lid = lid; + if (p_range) + p_range->min_lid = lid; + } + if (p_range) { + p_range->max_lid = p_mgr->p_subn->max_unicast_lid_ho - 1; + cl_qlist_insert_tail(&p_mgr->free_ranges, &p_range->item); + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, + "final free lid range [0x%x:0x%x]\n", + p_range->min_lid, p_range->max_lid); } - p_range->max_lid = p_mgr->p_subn->max_unicast_lid_ho - 1; - cl_qlist_insert_tail(&p_mgr->free_ranges, &p_range->item); - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "final free lid range [0x%x:0x%x]\n", - p_range->min_lid, p_range->max_lid); OSM_LOG_EXIT(p_mgr->p_log); return status; From ronli at voltaire.com Mon Jun 2 10:30:52 2008 From: ronli at voltaire.com (Ron Livne) Date: Mon, 2 Jun 2008 17:30:52 +0000 (UTC) Subject: [ofa-general] [PATCH 1/3] mcast loopback block Message-ID: mcast block loopback patch [1/3] ib_verbs: add mcast block loopback capability to the ib device. This patch adds a capability flag for blocking mcast loopback packets in an ib device. Signed-off-by: Ron Livne Index: kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-05-19 13:38:13.000000000 +0300 +++ kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-05-19 13:38:16.000000000 +0300 @@ -106,6 +106,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_SEND_W_INV = (1<<21), + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), }; enum ib_atomic_cap { @@ -500,6 +501,7 @@ enum ib_qp_type { enum ib_qp_create_flags { IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + IB_QP_BLOCK_LOOPBACK = 1 << 1, }; struct ib_qp_init_attr { From yevgenyp at mellanox.co.il Mon Jun 2 07:30:44 2008 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Mon, 02 Jun 2008 17:30:44 +0300 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: <483C2966.1080606@mellanox.co.il> References: <483C2966.1080606@mellanox.co.il> Message-ID: <48440414.4080909@mellanox.co.il> Hello Roland, Is there any update on this patch? Thanks, Yevgeny From ronli at voltaire.com Mon Jun 2 10:32:35 2008 From: ronli at voltaire.com (Ron Livne) Date: Mon, 2 Jun 2008 17:32:35 +0000 (UTC) Subject: [ofa-general] [PATCH 2/3] mcast loopback block Message-ID: mcast block loopback patch [2/3] mlx4: enable discarding/passing multicast loopback packets by FW/HW. When attaching a QP to a multicast group, checks if mcast loopback packets should be blocked (receives it as a parameter). If so, it sets the mcast loopback block bit to 1. When querying a QP (which is mlx4_ib_qp) using ib_qp_query, the field create_flags of the struct ib_qp_init_attr that is given as a parameter, will now be set with the flags that were used when this QP was created. Signed-off-by: Ron Livne Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c 2008-05-19 14:04:16.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c 2008-05-19 14:16:40.000000000 +0300 @@ -90,7 +90,8 @@ static int mlx4_ib_query_device(struct i props->device_cap_flags = IB_DEVICE_CHANGE_PHY_PORT | IB_DEVICE_PORT_ACTIVE_EVENT | IB_DEVICE_SYS_IMAGE_GUID | - IB_DEVICE_RC_RNR_NAK_GEN; + IB_DEVICE_RC_RNR_NAK_GEN | + IB_DEVICE_BLOCK_LOOPBACK; if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_BAD_PKEY_CNTR) props->device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR; if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_BAD_QKEY_CNTR) @@ -436,8 +437,12 @@ static int mlx4_ib_dealloc_pd(struct ib_ static int mlx4_ib_mcg_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) { - return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, - &to_mqp(ibqp)->mqp, gid->raw); + if (to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK) + return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, + &to_mqp(ibqp)->mqp, gid->raw, 1); + else + return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, + &to_mqp(ibqp)->mqp, gid->raw, 0); } static int mlx4_ib_mcg_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-05-19 14:04:16.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-05-19 14:16:40.000000000 +0300 @@ -101,7 +101,8 @@ struct mlx4_ib_wq { }; enum mlx4_ib_qp_flags { - MLX4_IB_QP_LSO = 1 << 0 + MLX4_IB_QP_LSO = 1 << 0, + MLX4_IB_QP_BLOCK_LOOPBACK = 1 << 1, }; struct mlx4_ib_qp { Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c 2008-05-19 14:04:16.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c 2008-05-19 14:16:40.000000000 +0300 @@ -506,6 +506,9 @@ static int create_qp_common(struct mlx4_ } else { qp->sq_no_prefetch = 0; + if (init_attr->create_flags & IB_QP_BLOCK_LOOPBACK) + qp->flags |= MLX4_IB_QP_BLOCK_LOOPBACK; + if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO) qp->flags |= MLX4_IB_QP_LSO; @@ -680,7 +683,7 @@ struct ib_qp *mlx4_ib_create_qp(struct i int err; /* We only support LSO, and only for kernel UD QPs. */ - if (init_attr->create_flags & ~IB_QP_CREATE_IPOIB_UD_LSO) + if (init_attr->create_flags & ~(IB_QP_CREATE_IPOIB_UD_LSO | IB_QP_BLOCK_LOOPBACK)) return ERR_PTR(-EINVAL); if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO && (pd->uobject || init_attr->qp_type != IB_QPT_UD)) @@ -1862,6 +1865,13 @@ done: qp_init_attr->cap = qp_attr->cap; + qp_init_attr->create_flags = 0; + if (qp->flags & MLX4_IB_QP_BLOCK_LOOPBACK) + qp_init_attr->create_flags |= IB_QP_BLOCK_LOOPBACK; + + if (qp->flags & MLX4_IB_QP_LSO) + qp_init_attr->create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + out: mutex_unlock(&qp->mutex); return err; Index: kernels/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c 2008-05-19 14:04:16.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c 2008-05-19 14:16:40.000000000 +0300 @@ -38,6 +38,9 @@ #include "mlx4.h" +#define MGM_QPN_MASK 0x00FFFFFF +#define MGM_BLCK_LB_BIT 30 + struct mlx4_mgm { __be32 next_gid_index; __be32 members_count; @@ -153,7 +156,7 @@ static int find_mgm(struct mlx4_dev *dev return err; } -int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]) +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_cmd_mailbox *mailbox; @@ -202,13 +205,17 @@ int mlx4_multicast_attach(struct mlx4_de } for (i = 0; i < members_count; ++i) - if (mgm->qp[i] == cpu_to_be32(qp->qpn)) { + if ((be32_to_cpu(mgm->qp[i]) & MGM_QPN_MASK) == qp->qpn) { mlx4_dbg(dev, "QP %06x already a member of MGM\n", qp->qpn); err = 0; goto out; } - mgm->qp[members_count++] = cpu_to_be32(qp->qpn); + if (block_mcast_lb) + mgm->qp[members_count++] = cpu_to_be32((qp->qpn & MGM_QPN_MASK) | (1 << MGM_BLCK_LB_BIT)); + else + mgm->qp[members_count++] = cpu_to_be32(qp->qpn & MGM_QPN_MASK); + mgm->members_count = cpu_to_be32(members_count); err = mlx4_WRITE_MCG(dev, index, mailbox); @@ -283,7 +290,7 @@ int mlx4_multicast_detach(struct mlx4_de members_count = be32_to_cpu(mgm->members_count); for (loc = -1, i = 0; i < members_count; ++i) - if (mgm->qp[i] == cpu_to_be32(qp->qpn)) + if ((be32_to_cpu(mgm->qp[i]) & MGM_QPN_MASK) == qp->qpn) loc = i; if (loc == -1) { Index: kernels/linux-2.6.26-rc2/include/linux/mlx4/device.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/include/linux/mlx4/device.h 2008-05-19 14:04:16.000000000 +0300 +++ kernels/linux-2.6.26-rc2/include/linux/mlx4/device.h 2008-05-19 14:16:40.000000000 +0300 @@ -398,7 +398,7 @@ int mlx4_srq_query(struct mlx4_dev *dev, int mlx4_INIT_PORT(struct mlx4_dev *dev, int port); int mlx4_CLOSE_PORT(struct mlx4_dev *dev, int port); -int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]); +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb); int mlx4_multicast_detach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]); int mlx4_map_phys_fmr(struct mlx4_dev *dev, struct mlx4_fmr *fmr, u64 *page_list, From ronli at voltaire.com Mon Jun 2 10:33:25 2008 From: ronli at voltaire.com (Ron Livne) Date: Mon, 2 Jun 2008 17:33:25 +0000 (UTC) Subject: [ofa-general] [PATCH 3/3] mcast loopback block Message-ID: mcast block loopback patch [3/3] IPOIB: blocking mcast loopback IPoIB packets This patch makes any mcast packet sent via IPoIB from any host, to be blocked when looped back to the same host it was sent from. The blocking is done by the HCA if possible. This patch creates an improvement of up to 39% in bandwidth when seding mcast packets with IPoIB. There is also an improvment of 12% in cpu usage. Signed-off-by: Ron Livne Index: kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-05-19 14:04:07.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-05-19 14:16:44.000000000 +0300 @@ -153,7 +153,8 @@ int ipoib_transport_dev_init(struct net_ .max_recv_sge = IPOIB_UD_RX_SG }, .sq_sig_type = IB_SIGNAL_ALL_WR, - .qp_type = IB_QPT_UD + .qp_type = IB_QPT_UD, + .create_flags = 0 }; int ret, size; @@ -201,7 +202,10 @@ int ipoib_transport_dev_init(struct net_ init_attr.recv_cq = priv->recv_cq; if (priv->hca_caps & IB_DEVICE_UD_TSO) - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + + if (priv->hca_caps & IB_DEVICE_BLOCK_LOOPBACK) + init_attr.create_flags |= IB_QP_BLOCK_LOOPBACK; if (dev->features & NETIF_F_SG) init_attr.cap.max_send_sge = MAX_SKB_FRAGS + 1; From vlad at dev.mellanox.co.il Mon Jun 2 07:43:50 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Mon, 02 Jun 2008 17:43:50 +0300 Subject: [ofa-general] [PATCH v2] mlx4: implement MOD_STAT_CFG command to use for changing Message-ID: <48440726.1000209@dev.mellanox.co.il> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-Path: vlad at dev.mellanox.co.il X-OriginalArrivalTime: 02 Jun 2008 14:43:14.0110 (UTC) FILETIME=[FC9DF5E0:01C8C4BE] Hello Roland, What is the status of this patch? http://lists.openfabrics.org/pipermail/general/2008-May/050537.html Regards, Vladimir From tziporet at mellanox.co.il Mon Jun 2 07:53:57 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 2 Jun 2008 17:53:57 +0300 Subject: [ofa-general] RE: [ewg] Agenda for the OFED meeting today (June 2) In-Reply-To: <6C2C79E72C305246B504CBA17B5500C9042828D3@mtlexch01.mtl.com> Message-ID: <6C2C79E72C305246B504CBA17B5500C904282934@mtlexch01.mtl.com> With few updates -----Original Message----- From: ewg-bounces at lists.openfabrics.org [mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of Tziporet Koren Sent: Monday, June 02, 2008 5:22 PM To: ewg at lists.openfabrics.org Cc: general at lists.openfabrics.org Subject: [ewg] Agenda for the OFED meeting today (June 2) Hi, This is the agenda for the OFED meeting today: 1. OFED 1.3.1 status: We inserted two more patches after RC3: http://lists.openfabrics.org/pipermail/general/2008-May/050551.html http://lists.openfabrics.org/pipermail/general/2008-May/050794.html (only the update for v2) Regression is in good shape. If everything works fine we will do final GA release tomorrow (June 3) Anyone that had not yet sent the release notes has last opportunity to send them today 2. OFED 1.4: - Kernel rebase status: Backports are available for all kernels between 2.6.26-rc2 and kernel 2.6.16. Work is ongoing and expected to be completed this week. URL: git://git.openfabrics.org/ofed_1_4/linux-2.6.git ofed_kernel - Update from the participants (mainly on new components/features) 3. Open discussion - Elect a chair to the EWG: Decide on a process for electing a chair and a co-chair to the EWG group. - Compatibility of libraries - John Russo - Other topics ... Tziporet _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From swise at opengridcomputing.com Mon Jun 2 08:29:22 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 02 Jun 2008 10:29:22 -0500 Subject: [ofa-general] [PATCH] rds: use separate ports for TCP and IB In-Reply-To: <200806021025.46789.olaf.kirch@oracle.com> References: <20080528225549.GC6288@opengridcomputing.com> <200806021025.46789.olaf.kirch@oracle.com> Message-ID: <484411D2.5010905@opengridcomputing.com> Olaf Kirch wrote: > On Friday 30 May 2008 18:29:01 Talpey, Thomas wrote: > >> Yes, at the moment TCP to the iWARP NIC is like talking to a different >> host. But, RDMA-aware versions of a given protocol still need a second >> port, unless there is explicit upper layer support for initiating the MPA >> exchange. We have the same issue with NFSv3/RDMA, and we have >> applied for a second port (the application is still pending within IANA). >> > > This may also become a problem for RDS, but in a different sense. > If you bind an RDS socket to a specific IP address, this also selects > a transport. If you bind to an IP address owned by ib0, you select > the IB transport. If you bind to an IP address owned by say eth0, you > select the TCP transport. > > Now if the one and only NIC in the system is an iWARP NIC, which transport > should we select? Both iWARP and TCP can be legitimate choices, depending > on the hosts we're talking to (and how do we find out whether the remote > node supports iWARP or not?) > If you use different ports for TCP vs iWARP, then you could try one, and if you don't connect (ie get a "connect refused") then you could try the other one... > I think for now it's okay to default to iWARP and punt if the remote > doesn't support it. But in the long run this is something that needs > to be addressed - either in RDS, or in the way ofed handles iWARP NICs. > > In fact it may not be such a bad idea to treat an iWARP NIC as two > devices, and register one ethX device (owned exclusively by the normal > stack) and one ibX device (owned mostly by the ofed stack, maybe with > the exception of ARP and such). That is basically what happens in linux today. There is a ethX interface that plugs into the stack, and the rnic driver also registers its rdma device with the rdma core. RDS can detect this by virtue of the fact that the MAC address is used as the gid for the device. So if a given ethX interface has an associated gid registered in the gid pool, then you know that interface supports both TCP and iWARP. Does that make sense? The rdma-cm code in core/cma.c does this sort of stuff for address resolution for iwarp devices. IE when resolving a remote address, the rdma-cm looks up the remote addr in the kernel routing table. That returns a local net device, aka ethX. The rdma-cm then looks up the mac address for that device in the gid cache to see if that device is rdma capable... Steve. > Would that work with protocols like > NFS/RDMA neogtiation - ie can you negotiate specify a secondary port > _and_ address? > > Olaf > From jon at opengridcomputing.com Mon Jun 2 08:50:03 2008 From: jon at opengridcomputing.com (Jon Mason) Date: Mon, 2 Jun 2008 10:50:03 -0500 Subject: [ofa-general] [PATCH] rds: use separate ports for TCP and IB In-Reply-To: <484411D2.5010905@opengridcomputing.com> References: <20080528225549.GC6288@opengridcomputing.com> <200806021025.46789.olaf.kirch@oracle.com> <484411D2.5010905@opengridcomputing.com> Message-ID: <20080602155003.GB17063@opengridcomputing.com> On Mon, Jun 02, 2008 at 10:29:22AM -0500, Steve Wise wrote: > Olaf Kirch wrote: >> On Friday 30 May 2008 18:29:01 Talpey, Thomas wrote: >> >>> Yes, at the moment TCP to the iWARP NIC is like talking to a different >>> host. But, RDMA-aware versions of a given protocol still need a second >>> port, unless there is explicit upper layer support for initiating the MPA >>> exchange. We have the same issue with NFSv3/RDMA, and we have >>> applied for a second port (the application is still pending within IANA). >>> >> >> This may also become a problem for RDS, but in a different sense. >> If you bind an RDS socket to a specific IP address, this also selects >> a transport. If you bind to an IP address owned by ib0, you select >> the IB transport. If you bind to an IP address owned by say eth0, you >> select the TCP transport. >> >> Now if the one and only NIC in the system is an iWARP NIC, which transport >> should we select? Both iWARP and TCP can be legitimate choices, depending >> on the hosts we're talking to (and how do we find out whether the remote >> node supports iWARP or not?) >> > > If you use different ports for TCP vs iWARP, then you could try one, and > if you don't connect (ie get a "connect refused") then you could try the > other one... The only way I can see this working is if RDS serially goes through each IP address on the system and tries to connect (and fails over to the next method if unable). Otherwise, it could destroy the current seperation that exists in RDS between the IB and TCP modules. >> I think for now it's okay to default to iWARP and punt if the remote >> doesn't support it. But in the long run this is something that needs >> to be addressed - either in RDS, or in the way ofed handles iWARP NICs. >> >> In fact it may not be such a bad idea to treat an iWARP NIC as two >> devices, and register one ethX device (owned exclusively by the normal >> stack) and one ibX device (owned mostly by the ofed stack, maybe with >> the exception of ARP and such). > > That is basically what happens in linux today. There is a ethX > interface that plugs into the stack, and the rnic driver also registers > its rdma device with the rdma core. RDS can detect this by virtue of > the fact that the MAC address is used as the gid for the device. So if > a given ethX interface has an associated gid registered in the gid pool, > then you know that interface supports both TCP and iWARP. Does that > make sense? > > The rdma-cm code in core/cma.c does this sort of stuff for address > resolution for iwarp devices. IE when resolving a remote address, the > rdma-cm looks up the remote addr in the kernel routing table. That > returns a local net device, aka ethX. The rdma-cm then looks up the mac > address for that device in the gid cache to see if that device is rdma > capable... > > Steve. > > > >> Would that work with protocols like >> NFS/RDMA neogtiation - ie can you negotiate specify a secondary port >> _and_ address? >> >> Olaf >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Mon Jun 2 09:24:54 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 09:24:54 -0700 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: <48440414.4080909@mellanox.co.il> (Yevgeny Petrilin's message of "Mon, 02 Jun 2008 17:30:44 +0300") References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> Message-ID: > Is there any update on this patch? Not that I have seen. From rdreier at cisco.com Mon Jun 2 09:26:01 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 09:26:01 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Initialize DS field for stamping In-Reply-To: <1212403058.17351.7.camel@mtls03> (Eli Cohen's message of "Mon, 02 Jun 2008 13:37:38 +0300") References: <1212403058.17351.7.camel@mtls03> Message-ID: This is a fix for your "optimize stamping" patch, right? I'll roll it into that patch. From rdreier at cisco.com Mon Jun 2 09:26:26 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 09:26:26 -0700 Subject: [ofa-general] Re: [PATCH v2] mlx4: implement MOD_STAT_CFG command to use for changing In-Reply-To: <48440726.1000209@dev.mellanox.co.il> (Vladimir Sokolovsky's message of "Mon, 02 Jun 2008 17:43:50 +0300") References: <48440726.1000209@dev.mellanox.co.il> Message-ID: > What is the status of this patch? Seems fine, I will get it into 2.6.27. From jon at opengridcomputing.com Mon Jun 2 09:31:59 2008 From: jon at opengridcomputing.com (Jon Mason) Date: Mon, 2 Jun 2008 11:31:59 -0500 Subject: [ofa-general] iWARP using IPv6 addresses Message-ID: <20080602163159.GC17063@opengridcomputing.com> I know there are some iWARP adapters that do NOT support rdma_bind_addr with an IPv6 address. Is there a way to determine if a given device can support IPv6? Would checking for max_raw_ipv6_qp being set to 0 be valid, or should there be a bit in device_cap_flags to show its support (or lack their of)? Thanks, Jon From rdreier at cisco.com Mon Jun 2 09:35:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 09:35:21 -0700 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: (Ron Livne's message of "Mon, 2 Jun 2008 17:30:52 +0000 (UTC)") References: Message-ID: > + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), This clashes with MEM_MGT_EXTENSIONS. Do we really need to burn a capability bit for this? Would it cause any problems to make IB_QP_BLOCK_LOOPBACK a hint that the underlying HCA may or may not support? > enum ib_qp_create_flags { > IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, > + IB_QP_BLOCK_LOOPBACK = 1 << 1, > }; The naming convention is that this should be IB_QP_CREATE_BLOCK_LOOPBACK. - R. From rdreier at cisco.com Mon Jun 2 09:36:35 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 09:36:35 -0700 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: (Ron Livne's message of "Mon, 2 Jun 2008 17:32:35 +0000 (UTC)") References: Message-ID: > + IB_DEVICE_BLOCK_LOOPBACK; Does all firmware support this so it can be enabled unconditionally? From or.gerlitz at gmail.com Mon Jun 2 09:53:51 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Mon, 2 Jun 2008 19:53:51 +0300 Subject: [ofa-general] Update on features that are delayed In-Reply-To: <000001c8c00b$50df8be0$ebc8180a@amr.corp.intel.com> References: <6C2C79E72C305246B504CBA17B5500C9041CE703@mtlexch01.mtl.com> <483BCDF9.2080207@voltaire.com> <000001c8c00b$50df8be0$ebc8180a@amr.corp.intel.com> Message-ID: <15ddcffd0806020953r4dac2a8p8e3a4bcfee7482a4@mail.gmail.com> On 5/27/08, Sean Hefty wrote: > > The main item that's missing is code in ib_addr to convert the IPv6 address > to > an IB GID. Once that's available, there may be a couple code paths where > IPv6 > checks are needed, but I can't think of anything that would be that hard. > OK. I assume that by convert from ipv6 address to gid you mean resolve to gid, correct? so the way to go is apply the same methods used by the stack eg when ping6 is used with ipoib? do you know if its the ipv6 Neighbor Discovery (ND) protocol? Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Mon Jun 2 10:05:11 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 10:05:11 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/sim_cmd.c: Cosmetic changes to help Message-ID: <1212426311.32691.60.camel@hrosenstock-ws.xsigo.com> ibsim/sim_cmd.c: Cosmetic changes to help Signed-off-by: Hal Rosenstock diff --git a/ibsim/sim_cmd.c b/ibsim/sim_cmd.c index fe3e9be..a6aab9d 100644 --- a/ibsim/sim_cmd.c +++ b/ibsim/sim_cmd.c @@ -700,10 +700,10 @@ static int dump_help(FILE * f) fprintf(f, "\tDump [nodeid] (def all network)\n"); fprintf(f, "\tRoute \n"); fprintf(f, "\tLink \"nodeid\"[port] \"remoteid\"[port]\n"); - fprintf(f, "\tUnlink \"nodeid\" : removes all link of the node\n"); + fprintf(f, "\tUnlink \"nodeid\" : remove all links of the node\n"); fprintf(f, "\tUnlink \"nodeid\"[port]\n"); fprintf(f, - "\tClear \"nodeid\" : unlink & reset all link of the node\n"); + "\tClear \"nodeid\" : unlink & reset all links of the node\n"); fprintf(f, "\tClear \"nodeid\"[port] : unlink & reset port\n"); fprintf(f, "\tGuid \"nodeid\" : set GUID value for this node\n"); fprintf(f, "\tGuid \"nodeid\"[port] : set GUID value for this port\n"); From hrosenstock at xsigo.com Mon Jun 2 10:05:17 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 10:05:17 -0700 Subject: [ofa-general] [PATCH] opensm/osm_lin_fwd_tbl.c: Eliminate potential NULL pointer dereference Message-ID: <1212426317.32691.62.camel@hrosenstock-ws.xsigo.com> opensm/osm_lin_fwd_tbl.c: Eliminate potential NULL pointer dereference Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_lin_fwd_tbl.c b/opensm/opensm/osm_lin_fwd_tbl.c index dc62f84..d311650 100644 --- a/opensm/opensm/osm_lin_fwd_tbl.c +++ b/opensm/opensm/osm_lin_fwd_tbl.c @@ -77,8 +77,8 @@ osm_lin_fwd_tbl_t *osm_lin_tbl_new(IN uint16_t const size) /* Initialize the table to OSM_NO_PATH, which means "invalid port" */ - memset(p_tbl, OSM_NO_PATH, __osm_lin_tbl_compute_obj_size(size)); if (p_tbl != NULL) { + memset(p_tbl, OSM_NO_PATH, __osm_lin_tbl_compute_obj_size(size)); p_tbl->size = (uint16_t) size; } return (p_tbl); From ralph.campbell at qlogic.com Mon Jun 2 10:09:58 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Mon, 02 Jun 2008 10:09:58 -0700 Subject: [ofa-general] [PATCH 1/3] mcast loopback block In-Reply-To: References: Message-ID: <1212426598.22429.2.camel@chromite.mv.qlogic.com> What do you mean by "block loopback"? Does this mean a posted send WR for a multicast packet that matches the local LID is not sent at all or the packet is sent on the wire but not sent to the local QP? Or is it something else? On Mon, 2008-06-02 at 17:30 +0000, Ron Livne wrote: > mcast block loopback patch [1/3] > > ib_verbs: add mcast block loopback capability to the ib device. > > This patch adds a capability flag for blocking mcast loopback > packets in an ib device. > > > Signed-off-by: Ron Livne > > > Index: kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h > =================================================================== > --- kernels.orig/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-05-19 13:38:13.000000000 +0300 > +++ kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-05-19 13:38:16.000000000 +0300 > @@ -106,6 +106,7 @@ enum ib_device_cap_flags { > IB_DEVICE_UD_IP_CSUM = (1<<18), > IB_DEVICE_UD_TSO = (1<<19), > IB_DEVICE_SEND_W_INV = (1<<21), > + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), > }; > > enum ib_atomic_cap { > @@ -500,6 +501,7 @@ enum ib_qp_type { > > enum ib_qp_create_flags { > IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, > + IB_QP_BLOCK_LOOPBACK = 1 << 1, > }; > > struct ib_qp_init_attr { > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sean.hefty at intel.com Mon Jun 2 10:13:58 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 2 Jun 2008 10:13:58 -0700 Subject: [ofa-general] Update on features that are delayed In-Reply-To: <15ddcffd0806020953r4dac2a8p8e3a4bcfee7482a4@mail.gmail.com> References: <6C2C79E72C305246B504CBA17B5500C9041CE703@mtlexch01.mtl.com> <483BCDF9.2080207@voltaire.com> <000001c8c00b$50df8be0$ebc8180a@amr.corp.intel.com> <15ddcffd0806020953r4dac2a8p8e3a4bcfee7482a4@mail.gmail.com> Message-ID: <000501c8c4d4$0b795130$855b180a@amr.corp.intel.com> > OK. I assume that by convert from ipv6 address to gid you mean resolve to gid, > correct? so the way to go is apply the same methods used by the stack eg when > ping6 is used with ipoib? do you know if its the ipv6 Neighbor Discovery (ND) > protocol? yes - yes - and I haven't spent the time to see what's done in the stack today. - Sean From olga.shern at gmail.com Mon Jun 2 10:52:34 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Mon, 2 Jun 2008 20:52:34 +0300 Subject: [ofa-general] [PATCH 1/3] mcast loopback block In-Reply-To: <1212426598.22429.2.camel@chromite.mv.qlogic.com> References: <1212426598.22429.2.camel@chromite.mv.qlogic.com> Message-ID: On 6/2/08, Ralph Campbell wrote: > > What do you mean by "block loopback"? > Does this mean a posted send WR for a multicast packet > that matches the local LID is not sent at all or > the packet is sent on the wire but not sent to > the local QP? Or is it something else? It controls the ConnectX HCA multicast packets block loopback (blck_lb) for self QP. The patch is designed to enable or disable blocking of all multicast packets on self QP created on the ConnectX HCA On Mon, 2008-06-02 at 17:30 +0000, Ron Livne wrote: > > mcast block loopback patch [1/3] > > > > ib_verbs: add mcast block loopback capability to the ib device. > > > > This patch adds a capability flag for blocking mcast loopback > > packets in an ib device. > > > > > > Signed-off-by: Ron Livne > > > > > > Index: kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h > > =================================================================== > > --- kernels.orig/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-05-19 > 13:38:13.000000000 +0300 > > +++ kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-05-19 > 13:38:16.000000000 +0300 > > @@ -106,6 +106,7 @@ enum ib_device_cap_flags { > > IB_DEVICE_UD_IP_CSUM = (1<<18), > > IB_DEVICE_UD_TSO = (1<<19), > > IB_DEVICE_SEND_W_INV = (1<<21), > > + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), > > }; > > > > enum ib_atomic_cap { > > @@ -500,6 +501,7 @@ enum ib_qp_type { > > > > enum ib_qp_create_flags { > > IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, > > + IB_QP_BLOCK_LOOPBACK = 1 << 1, > > }; > > > > struct ib_qp_init_attr { > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Mon Jun 2 10:54:01 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 02 Jun 2008 12:54:01 -0500 Subject: [ofa-general] [PATCH 2.6.27] RDMA/Core: MEM_MGT_EXTENSIONS support In-Reply-To: <20080602174831.3510.63409.stgit@dell3.ogc.int> References: <20080602174831.3510.63409.stgit@dell3.ogc.int> Message-ID: <20080602175400.3510.77341.stgit@dell3.ogc.int> Support for the IB BMME and iWARP equivalent memory extensions to non shared memory regions. This includes: - allocation of an ib_mr for use in fast register work requests - device-specific alloc/free of physical buffer lists for use in fast register work requests. This allows devices to allocate this memory as needed (like via dma_alloc_coherent). - fast register memory region work request - invalidate local memory region work request - read with invalidate local memory region work request (iWARP only) Design details: - New device capability flag added: IB_DEVICE_MEM_MGT_EXTENSIONS indicates device support for this feature. - New send WR opcode IB_WR_FAST_REG_MR used to issue a fast_reg request. - New send WR opcode IB_WR_INVALIDATE_MR used to invalidate a fast_reg mr. - New API function, ib_alloc_mr() used to allocate fast_reg memory regions. - New API function, ib_alloc_fast_reg_page_list to allocate device-specific page lists. - New API function, ib_free_fast_reg_page_list to free said page lists. - New API function, ib_update_fast_reg_key to allow the key portion of the R_Key and L_Key of a fast_reg MR to be updated. Applications call this if desired before posting the IB_WR_FAST_REG_MR. Usage Model: - MR allocated with ib_alloc_mr() - Page lists allocated via ib_alloc_fast_reg_page_list(). - MR R_Key/L_Key "key" field updated with ib_update_fast_reg_key(). - MR made VALID and bound to a specific page list via ib_post_send(IB_WR_FAST_REG_MR) - MR made INVALID via ib_post_send(IB_WR_INVALIDATE_MR) - MR deallocated with ib_dereg_mr() - page lists dealloced via ib_free_fast_reg_page_list(). Applications can allocate a fast_reg mr once, and then can repeatedly bind the mr to different physical memory SGLs via posting work requests to the For each outstanding mr-to-pbl binding in the SQ pipe, a fast_reg_page_list needs to be allocated. Thus pipelining can be achieved while still allowing device-specific page_list processing. The 4B fast_reg rkey or stag is composed of a 3B index, and a 1B key. The application can change the key each time it fast-registers thus allowing more control over the peer's use of the rkey (ie it can effectively be changed each time the rkey is rebound to a page list). Signed-off-by: Steve Wise --- drivers/infiniband/core/verbs.c | 46 ++++++++++++++++++++++++ include/rdma/ib_verbs.h | 76 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 0504208..0a334b4 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -755,6 +755,52 @@ int ib_dereg_mr(struct ib_mr *mr) } EXPORT_SYMBOL(ib_dereg_mr); +struct ib_mr *ib_alloc_fast_reg_mr(struct ib_pd *pd, int max_page_list_len) +{ + struct ib_mr *mr; + + if (!pd->device->alloc_fast_reg_mr) + return ERR_PTR(-ENOSYS); + + mr = pd->device->alloc_fast_reg_mr(pd, max_page_list_len); + + if (!IS_ERR(mr)) { + mr->device = pd->device; + mr->pd = pd; + mr->uobject = NULL; + atomic_inc(&pd->usecnt); + atomic_set(&mr->usecnt, 0); + } + + return mr; +} +EXPORT_SYMBOL(ib_alloc_fast_reg_mr); + +struct ib_fast_reg_page_list *ib_alloc_fast_reg_page_list( + struct ib_device *device, int max_page_list_len) +{ + struct ib_fast_reg_page_list *page_list; + + if (!device->alloc_fast_reg_page_list) + return ERR_PTR(-ENOSYS); + + page_list = device->alloc_fast_reg_page_list(device, max_page_list_len); + + if (!IS_ERR(page_list)) { + page_list->device = device; + page_list->max_page_list_len = max_page_list_len; + } + + return page_list; +} +EXPORT_SYMBOL(ib_alloc_fast_reg_page_list); + +void ib_free_fast_reg_page_list(struct ib_fast_reg_page_list *page_list) +{ + page_list->device->free_fast_reg_page_list(page_list); +} +EXPORT_SYMBOL(ib_free_fast_reg_page_list); + /* Memory windows */ struct ib_mw *ib_alloc_mw(struct ib_pd *pd) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 911a661..4530ebd 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -106,6 +106,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_SEND_W_INV = (1<<21), + IB_DEVICE_MEM_MGT_EXTENSIONS = (1<<22), }; enum ib_atomic_cap { @@ -151,6 +152,7 @@ struct ib_device_attr { int max_srq; int max_srq_wr; int max_srq_sge; + unsigned int max_fast_reg_page_list_len; u16 max_pkeys; u8 local_ca_ack_delay; }; @@ -414,6 +416,8 @@ enum ib_wc_opcode { IB_WC_FETCH_ADD, IB_WC_BIND_MW, IB_WC_LSO, + IB_WC_FAST_REG_MR, + IB_WC_INVALIDATE_MR, /* * Set value of IB_WC_RECV so consumers can test if a completion is a * receive by testing (opcode & IB_WC_RECV). @@ -628,6 +632,9 @@ enum ib_wr_opcode { IB_WR_ATOMIC_FETCH_AND_ADD, IB_WR_LSO, IB_WR_SEND_WITH_INV, + IB_WR_FAST_REG_MR, + IB_WR_INVALIDATE_MR, + IB_WR_READ_WITH_INV, }; enum ib_send_flags { @@ -676,6 +683,19 @@ struct ib_send_wr { u16 pkey_index; /* valid for GSI only */ u8 port_num; /* valid for DR SMPs on switch only */ } ud; + struct { + u64 iova_start; + struct ib_fast_reg_page_list *page_list; + unsigned int page_shift; + unsigned int page_list_len; + unsigned int first_byte_offset; + u32 length; + int access_flags; + u32 rkey; + } fast_reg; + struct { + u32 rkey; + } local_inv; } wr; }; @@ -1014,6 +1034,10 @@ struct ib_device { int (*query_mr)(struct ib_mr *mr, struct ib_mr_attr *mr_attr); int (*dereg_mr)(struct ib_mr *mr); + struct ib_mr * (*alloc_fast_reg_mr)(struct ib_pd *pd, + int max_page_list_len); + struct ib_fast_reg_page_list * (*alloc_fast_reg_page_list)(struct ib_device *device, int page_list_len); + void (*free_fast_reg_page_list)(struct ib_fast_reg_page_list *page_list); int (*rereg_phys_mr)(struct ib_mr *mr, int mr_rereg_mask, struct ib_pd *pd, @@ -1808,6 +1832,58 @@ int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr); int ib_dereg_mr(struct ib_mr *mr); /** + * ib_alloc_fast_reg_mr - Allocates memory region usable with the + * IB_WR_FAST_REG_MR send work request. + * @pd: The protection domain associated with the region. + * @max_page_list_len: requested max physical buffer list size to be allocated. + */ +struct ib_mr *ib_alloc_fast_reg_mr(struct ib_pd *pd, int max_page_list_len); + +struct ib_fast_reg_page_list { + struct ib_device *device; + u64 *page_list; + unsigned int max_page_list_len; +}; + +/** + * ib_alloc_fast_reg_page_list - Allocates a page list array + * @device - ib device pointer. + * @page_list_len - size of the page list array to be allocated. + * + * This allocates and returns a struct ib_fast_reg_page_list * + * and a page_list array that is at least page_list_len in size. + * The actual size is returned in max_page_list_len. + * The caller is responsible for initializing the contents of the + * page_list array before posting a send work request with the + * IB_WC_FAST_REG_MR opcode. The page_list array entries must be + * translated using one of the ib_dma_*() functions similar to the + * addresses passed to ib_map_phys_fmr(). Once the ib_post_send() + * is issued, the struct ib_fast_reg_page_list must not be modified + * by the caller until a completion notice is returned by the device. + */ +struct ib_fast_reg_page_list *ib_alloc_fast_reg_page_list( + struct ib_device *device, int page_list_len); + +/** + * ib_free_fast_reg_page_list - Deallocates a previously allocated + * page list array. + * @page_list - struct ib_fast_reg_page_list pointer to be deallocated. + */ +void ib_free_fast_reg_page_list(struct ib_fast_reg_page_list *page_list); + +/** + * ib_update_fast_reg_key - updates the key portion of the fast_reg + * R_Key and L_Key. + * @mr - struct ib_mr pointer to be updated. + * @newkey - new key to be used. + */ +static inline void ib_update_fast_reg_key(struct ib_mr *mr, u8 newkey) +{ + mr->lkey = (mr->lkey & 0xffffff00) | newkey; + mr->rkey = (mr->rkey & 0xffffff00) | newkey; +} + +/** * ib_alloc_mw - Allocates a memory window. * @pd: The protection domain associated with the memory window. */ From swise at opengridcomputing.com Mon Jun 2 10:48:31 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 02 Jun 2008 12:48:31 -0500 Subject: [ofa-general] [PATCH 2.6.27] RDMA/Core:MEM_MGT_EXTENSIONS support Message-ID: <20080602174831.3510.63409.stgit@dell3.ogc.int> Roland, I think the core changes are ready for you to merge into your for-2.6.27 branch. I'm holding onto my cxgb3 changes until I get firmware and can test things out. If I hit other core issues, I'll post new patches against your for-2.6.27 branch. Thanks, Steve. From ralph.campbell at qlogic.com Mon Jun 2 10:57:32 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Mon, 02 Jun 2008 10:57:32 -0700 Subject: [ofa-general] [PATCH 1/3] mcast loopback block In-Reply-To: References: <1212426598.22429.2.camel@chromite.mv.qlogic.com> Message-ID: <1212429452.22429.19.camel@chromite.mv.qlogic.com> On Mon, 2008-06-02 at 20:52 +0300, Olga Shern (Voltaire) wrote: > > > On 6/2/08, Ralph Campbell wrote: > What do you mean by "block loopback"? > Does this mean a posted send WR for a multicast packet > that matches the local LID is not sent at all or > the packet is sent on the wire but not sent to > the local QP? Or is it something else? > > > It controls the ConnectX HCA multicast packets block loopback > (blck_lb) for self QP. The patch is designed to enable or > > disable blocking of all multicast packets on self QP created on the > ConnectX HCA I understand that the patch enables/disables the bit. You didn't really answer my question which is what does the bit really do. I'm pretty sure it means that multicast packets aren't forwarded to any locally attached QPs but I wanted to be sure that is what it does. From olga.shern at gmail.com Mon Jun 2 11:06:15 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Mon, 2 Jun 2008 21:06:15 +0300 Subject: [ofa-general] [PATCH 1/3] mcast loopback block In-Reply-To: <1212429452.22429.19.camel@chromite.mv.qlogic.com> References: <1212426598.22429.2.camel@chromite.mv.qlogic.com> <1212429452.22429.19.camel@chromite.mv.qlogic.com> Message-ID: On 6/2/08, Ralph Campbell wrote: > > On Mon, 2008-06-02 at 20:52 +0300, Olga Shern (Voltaire) wrote: > > > > > > On 6/2/08, Ralph Campbell wrote: > > What do you mean by "block loopback"? > > Does this mean a posted send WR for a multicast packet > > that matches the local LID is not sent at all or > > the packet is sent on the wire but not sent to > > the local QP? Or is it something else? > > > > > > It controls the ConnectX HCA multicast packets block loopback > > (blck_lb) for self QP. The patch is designed to enable or > > > > disable blocking of all multicast packets on self QP created on the > > ConnectX HCA > > I understand that the patch enables/disables the bit. > You didn't really answer my question which is what does > the bit really do. I'm pretty sure it means that multicast > packets aren't forwarded to any locally attached QPs > but I wanted to be sure that is what it does. > > No, It isn't disabled for every QP (by default loopback is enabled), when you attach QP to multicast group you can decide to disable it (the decision is done per QP) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Mon Jun 2 11:11:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 21:11:50 +0300 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> References: <20080523100634.GD4164@sashak.voltaire.com> <1211541313.13185.80.camel@hrosenstock-ws.xsigo.com> <20080523123414.GB4640@sashak.voltaire.com> <1211547161.13185.103.camel@hrosenstock-ws.xsigo.com> <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602181150.GB13775@sashak.voltaire.com> On 07:01 Mon 02 Jun , Hal Rosenstock wrote: > On Sat, 2008-05-31 at 20:28 +0300, Sasha Khapyorsky wrote: > > This adds possibility to specify SM_Key value with saquery. It should > > work with queries where OSM_DEFAULT_SM_KEY was used. > > > > If non-numeric string (like 'x') is provided with --smkey option then > > saquery will prompt to get SM_Key value. > > > > Signed-off-by: Sasha Khapyorsky > > --- > > > > SM_key value prompting was added as addition to v1 of the patch. > > > > infiniband-diags/src/saquery.c | 20 +++++++++++++++++--- > > man page change is needed for this too. Sure. > > @@ -1512,6 +1518,14 @@ main(int argc, char **argv) > > case 3: > > node_name_map_file = strdup(optarg); > > break; > > + case 4: > > + if (!isxdigit(*optarg) && > > + !(optarg = getpass("SM_Key: "))) { > > Man page for getpass indicates: > "This function is obsolete. Do not use it." Right, but glibc info page doesn't indicate this. Also I did some googling and looked at glibc code itself - found nothing suspicious yet. Finally it is how password handled in 'su'. Sasha From ralph.campbell at qlogic.com Mon Jun 2 11:25:17 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Mon, 02 Jun 2008 11:25:17 -0700 Subject: [ofa-general] [PATCH 1/3] mcast loopback block In-Reply-To: References: <1212426598.22429.2.camel@chromite.mv.qlogic.com> <1212429452.22429.19.camel@chromite.mv.qlogic.com> Message-ID: <1212431118.22429.24.camel@chromite.mv.qlogic.com> On Mon, 2008-06-02 at 21:06 +0300, Olga Shern (Voltaire) wrote: > > > On 6/2/08, Ralph Campbell wrote: > On Mon, 2008-06-02 at 20:52 +0300, Olga Shern (Voltaire) > wrote: > > > > > > On 6/2/08, Ralph Campbell wrote: > > What do you mean by "block loopback"? > > Does this mean a posted send WR for a multicast > packet > > that matches the local LID is not sent at all or > > the packet is sent on the wire but not sent to > > the local QP? Or is it something else? > > > > > > It controls the ConnectX HCA multicast packets block > loopback > > (blck_lb) for self QP. The patch is designed to enable or > > > > disable blocking of all multicast packets on self QP created > on the > > ConnectX HCA > > I understand that the patch enables/disables the bit. > You didn't really answer my question which is what does > the bit really do. I'm pretty sure it means that multicast > packets aren't forwarded to any locally attached QPs > but I wanted to be sure that is what it does. > > > No, > It isn't disabled for every QP (by default loopback is enabled), > when you attach QP to multicast group you can decide to disable it > (the decision is done per QP) So, in other words, if you create a QP with this option and then attach it to a multicast group and a multicast packet is posted on the send work request queue of the "disabled" QP, it won't receive a copy of the packet it sent. If some other QP on the same HCA is also attached to the same multicast group, it will receive a copy of the multicast packet. From sashak at voltaire.com Mon Jun 2 11:29:22 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 21:29:22 +0300 Subject: [ofa-general] Re: OSM_DEFAULT_SM_KEY byte order In-Reply-To: <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602182922.GC13775@sashak.voltaire.com> On 07:06 Mon 02 Jun , Hal Rosenstock wrote: > > This came from informal interop testing a while ago. It wasn't invented > out of thin air. IMHO there are not enough information about the case - finally the value of OSM_DEFAULT_SM_KEY was host byte order (which is obviously wrong), so it doesn't look for me that the case was fully analyzed. > > Our own backward compatibility could be solved by configuring sm key > > (this will work with OpenSM and saquery). > > > > Another opinions? > > I think that third party SMs are a side issue as this is not sanctioned > by IBTA and there is other evidence of a vendor SM using SM key. > > To me, key is back interop with older OpenSMs (at least for x86 as that > is the larger part of the installed base) and this is the aspect which > is sanctioned by IBTA. SM_Key value is configurable in OpenSM so we don't really break interoperability. And in longer term '1' seems as much more "friendly" value than '0x0100000000000000', which entered OpenSM code by mistake. Sasha From hrosenstock at xsigo.com Mon Jun 2 11:41:08 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 11:41:08 -0700 Subject: [ofa-general] Re: OSM_DEFAULT_SM_KEY byte order In-Reply-To: <20080602182922.GC13775@sashak.voltaire.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> <20080602182922.GC13775@sashak.voltaire.com> Message-ID: <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 21:29 +0300, Sasha Khapyorsky wrote: > On 07:06 Mon 02 Jun , Hal Rosenstock wrote: > > > > This came from informal interop testing a while ago. It wasn't invented > > out of thin air. > > IMHO there are not enough information about the case - finally the value > of OSM_DEFAULT_SM_KEY was host byte order (which is obviously wrong), so > it doesn't look for me that the case was fully analyzed. The value was observed on the IB wire with an analyzer. It was implemented in OpenSM incorrectly. > > > Our own backward compatibility could be solved by configuring sm key > > > (this will work with OpenSM and saquery). > > > > > > Another opinions? > > > > I think that third party SMs are a side issue as this is not sanctioned > > by IBTA and there is other evidence of a vendor SM using SM key. > > > > To me, key is back interop with older OpenSMs (at least for x86 as that > > is the larger part of the installed base) and this is the aspect which > > is sanctioned by IBTA. > > SM_Key value is configurable in OpenSM so we don't really break > interoperability. Well it does by default and that's the behavior we were discussing. This argument cuts the other way too in that it can be "fixed" when needed. > And in longer term '1' seems as much more "friendly" > value than '0x0100000000000000', which entered OpenSM code by mistake I don't think that friendliness is in the same category of factors. Also, it doesn't need to be typed when default except when "wrong". -- Hal > Sasha From hrosenstock at xsigo.com Mon Jun 2 11:44:37 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 11:44:37 -0700 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <20080602181150.GB13775@sashak.voltaire.com> References: <20080523100634.GD4164@sashak.voltaire.com> <1211541313.13185.80.camel@hrosenstock-ws.xsigo.com> <20080523123414.GB4640@sashak.voltaire.com> <1211547161.13185.103.camel@hrosenstock-ws.xsigo.com> <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> <20080602181150.GB13775@sashak.voltaire.com> Message-ID: <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 21:11 +0300, Sasha Khapyorsky wrote: > On 07:01 Mon 02 Jun , Hal Rosenstock wrote: > > On Sat, 2008-05-31 at 20:28 +0300, Sasha Khapyorsky wrote: > > > This adds possibility to specify SM_Key value with saquery. It should > > > work with queries where OSM_DEFAULT_SM_KEY was used. > > > > > > If non-numeric string (like 'x') is provided with --smkey option then > > > saquery will prompt to get SM_Key value. > > > > > > Signed-off-by: Sasha Khapyorsky > > > --- > > > > > > SM_key value prompting was added as addition to v1 of the patch. > > > > > > infiniband-diags/src/saquery.c | 20 +++++++++++++++++--- > > > > man page change is needed for this too. > > Sure. > > > > @@ -1512,6 +1518,14 @@ main(int argc, char **argv) > > > case 3: > > > node_name_map_file = strdup(optarg); > > > break; > > > + case 4: > > > + if (!isxdigit(*optarg) && > > > + !(optarg = getpass("SM_Key: "))) { > > > > Man page for getpass indicates: > > "This function is obsolete. Do not use it." > > Right, but glibc info page doesn't indicate this. I looked at the man pages on both RHEL4 and RHEL5 machines. -- Hal > Also I did some > googling and looked at glibc code itself - found nothing suspicious yet. > Finally it is how password handled in 'su'. > > Sasha From sashak at voltaire.com Mon Jun 2 11:51:51 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 21:51:51 +0300 Subject: [ofa-general] Re: OSM_DEFAULT_SM_KEY byte order In-Reply-To: <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> <20080602182922.GC13775@sashak.voltaire.com> <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602185151.GF13775@sashak.voltaire.com> On 11:41 Mon 02 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-02 at 21:29 +0300, Sasha Khapyorsky wrote: > > On 07:06 Mon 02 Jun , Hal Rosenstock wrote: > > > > > > This came from informal interop testing a while ago. It wasn't invented > > > out of thin air. > > > > IMHO there are not enough information about the case - finally the value > > of OSM_DEFAULT_SM_KEY was host byte order (which is obviously wrong), so > > it doesn't look for me that the case was fully analyzed. > > The value was observed on the IB wire with an analyzer. It was > implemented in OpenSM incorrectly. Right, and this leaves questions: for instance if grabbed value was exactly '0x0100000000000000', I wouldn't see a big chance for such mistake. Another story would be if grabbed value was something 'non-zero'. Again, it is unclear for me. > > > > Our own backward compatibility could be solved by configuring sm key > > > > (this will work with OpenSM and saquery). > > > > > > > > Another opinions? > > > > > > I think that third party SMs are a side issue as this is not sanctioned > > > by IBTA and there is other evidence of a vendor SM using SM key. > > > > > > To me, key is back interop with older OpenSMs (at least for x86 as that > > > is the larger part of the installed base) and this is the aspect which > > > is sanctioned by IBTA. > > > > SM_Key value is configurable in OpenSM so we don't really break > > interoperability. > > Well it does by default and that's the behavior we were discussing. This > argument cuts the other way too in that it can be "fixed" when needed. Yes, it can be "fixed" even now. I'm thinking about permanent solution and '1' looks like more reasonable default for me. > > And in longer term '1' seems as much more "friendly" > > value than '0x0100000000000000', which entered OpenSM code by mistake > > I don't think that friendliness is in the same category of factors. I think it is when we are about default values. > Also, it doesn't need to be typed when default except when "wrong". Right, and "wrong" default right now is on x86. Sasha From sashak at voltaire.com Mon Jun 2 11:55:52 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 21:55:52 +0300 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> References: <20080523123414.GB4640@sashak.voltaire.com> <1211547161.13185.103.camel@hrosenstock-ws.xsigo.com> <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> <20080602181150.GB13775@sashak.voltaire.com> <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602185552.GG13775@sashak.voltaire.com> On 11:44 Mon 02 Jun , Hal Rosenstock wrote: > > > > > > Man page for getpass indicates: > > > "This function is obsolete. Do not use it." > > > > Right, but glibc info page doesn't indicate this. > > I looked at the man pages on both RHEL4 and RHEL5 machines. I saw this, just don't understand what this should mean for us. Glibc's info page: info getpass doesn't indicate this. Maybe there could be portability problems, but it should not be worst with 'su'. Sasha From hrosenstock at xsigo.com Mon Jun 2 12:01:24 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 12:01:24 -0700 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <20080602185552.GG13775@sashak.voltaire.com> References: <20080523123414.GB4640@sashak.voltaire.com> <1211547161.13185.103.camel@hrosenstock-ws.xsigo.com> <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> <20080602181150.GB13775@sashak.voltaire.com> <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> <20080602185552.GG13775@sashak.voltaire.com> Message-ID: <1212433284.32691.88.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 21:55 +0300, Sasha Khapyorsky wrote: > On 11:44 Mon 02 Jun , Hal Rosenstock wrote: > > > > > > > > Man page for getpass indicates: > > > > "This function is obsolete. Do not use it." > > > > > > Right, but glibc info page doesn't indicate this. > > > > I looked at the man pages on both RHEL4 and RHEL5 machines. > > I saw this, just don't understand what this should mean for us. > Glibc's info page: > > info getpass > > doesn't indicate this. Maybe there could be portability problems, but it > should not be worst with 'su'. Just seems safer to me to implement without using this. It's only a "shortcut" anyway for turning echo off and on. -- Hal > Sasha From hnguyen at linux.vnet.ibm.com Mon Jun 2 12:01:02 2008 From: hnguyen at linux.vnet.ibm.com (Hoang-Nam Nguyen) Date: Mon, 2 Jun 2008 21:01:02 +0200 Subject: [ofa-general] IB/ipoib: no need to call skb_orphan() in cm_send()? Message-ID: <200806022101.02400.hnguyen@linux.vnet.ibm.com> Hello Eli! I'm referring to the thread below, where you explained the rational of calling skb_orphan() after post_send() in UD path: [ewg] Re: [PATCH] call skb_orphan() after sending an SKB http://lists.openfabrics.org/pipermail/ewg/2008-February/005739.html Looking at cm_send path I don't see that call. Is there any reason not to do that here? void ipoib_cm_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_cm_tx *tx) { ... if (unlikely(post_send(priv, tx, tx->tx_head & (ipoib_sendq_size - 1), addr, skb->len))) { ipoib_warn(priv, "post_send failed\n"); ++priv->stats.tx_errors; ib_dma_unmap_single(priv->ca, addr, skb->len, DMA_TO_DEVICE); dev_kfree_skb_any(skb); } else { dev->trans_start = jiffies; ++tx->tx_head; if (++priv->tx_outstanding == ipoib_sendq_size - 1) { ipoib_dbg(priv, "%s: TX ring 0x%x full," "stopping kernel net queue\n", __func__, tx->qp->qp_num); netif_stop_queue(dev); } } } Thanks and regards Nam From sashak at voltaire.com Mon Jun 2 12:12:27 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 22:12:27 +0300 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <1212433284.32691.88.camel@hrosenstock-ws.xsigo.com> References: <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> <20080602181150.GB13775@sashak.voltaire.com> <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> <20080602185552.GG13775@sashak.voltaire.com> <1212433284.32691.88.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602191227.GI13775@sashak.voltaire.com> On 12:01 Mon 02 Jun , Hal Rosenstock wrote: > > Just seems safer to me to implement without using this. Without using this we need to write getpass() function. Prefer to do it in the case if we will see some problem with glibc's one. > It's only a > "shortcut" anyway for turning echo off and on. Also it tries to use /dev/tty instead of stdin/stdout, locks the file, etc. Sasha From hrosenstock at xsigo.com Mon Jun 2 12:25:28 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 12:25:28 -0700 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <20080602191227.GI13775@sashak.voltaire.com> References: <20080527103341.GF12014@sashak.voltaire.com> <1211888036.13185.219.camel@hrosenstock-ws.xsigo.com> <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> <20080602181150.GB13775@sashak.voltaire.com> <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> <20080602185552.GG13775@sashak.voltaire.com> <1212433284.32691.88.camel@hrosenstock-ws.xsigo.com> <20080602191227.GI13775@sashak.voltaire.com> Message-ID: <1212434728.32691.100.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 22:12 +0300, Sasha Khapyorsky wrote: > On 12:01 Mon 02 Jun , Hal Rosenstock wrote: > > > > Just seems safer to me to implement without using this. > > Without using this we need to write getpass() function. Yes. > Prefer to do it in the case if we will see some problem with glibc's one. But why implement using a known obsoleted function ? -- Hal > > It's only a > > "shortcut" anyway for turning echo off and on. > > Also it tries to use /dev/tty instead of stdin/stdout, locks the file, > etc. > > Sasha From sashak at voltaire.com Mon Jun 2 12:22:47 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 22:22:47 +0300 Subject: [ofa-general] Re: [PATCH] OpenSM release notes: Update to 3.1.11 In-Reply-To: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> References: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602192247.GJ13775@sashak.voltaire.com> On 07:00 Mon 02 Jun , Hal Rosenstock wrote: > OpenSM release notes: Update to 3.1.11 > > Please apply to ofed_1_3 and master ofed_1_3 is closed and 3.1.11 is out already. I can apply to the master, but difference (except one mentioned below) is only version number and dates. Isn't this helpful there? > -1.4 Software Dependencies > - > -OpenSM depends on the installation of either OFED 1.3, OFED 1.2, OFED 1.1, > -OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD > -distribution), or Mellanox VAPI stacks. The qualified driver versions > -are provided in Table 2, "Qualified IB Stacks". > - > -Also building of QoS manager policy file parser requires flex, and either > -bison or byacc installed. Isn't this: commit f8e4b29248fd679d9678fe37acb72add1a48cfd6 Author: Yevgeny Kliteynik Date: Mon Mar 10 16:56:23 2008 +0200 opensm/osm_qos_parser: fixed compilation on byacc Hi Sasha. Fixing compilation with byacc (bug 932). Please apply to ofed_1_3 and master. -- Yevgeny Signed-off-by: Yevgeny Kliteynik Signed-off-by: Sasha Khapyorsky solve byacc issue ('byacc' is removed in the new text)? Sasha From sashak at voltaire.com Mon Jun 2 12:27:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 22:27:50 +0300 Subject: [ofa-general] Re: [PATCH v2] saquery: --smkey command line option In-Reply-To: <1212434728.32691.100.camel@hrosenstock-ws.xsigo.com> References: <20080527175343.GA14205@sashak.voltaire.com> <1211972791.13185.334.camel@hrosenstock-ws.xsigo.com> <20080531172815.GO22418@sashak.voltaire.com> <1212415276.32691.33.camel@hrosenstock-ws.xsigo.com> <20080602181150.GB13775@sashak.voltaire.com> <1212432277.32691.75.camel@hrosenstock-ws.xsigo.com> <20080602185552.GG13775@sashak.voltaire.com> <1212433284.32691.88.camel@hrosenstock-ws.xsigo.com> <20080602191227.GI13775@sashak.voltaire.com> <1212434728.32691.100.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602192750.GK13775@sashak.voltaire.com> On 12:25 Mon 02 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-02 at 22:12 +0300, Sasha Khapyorsky wrote: > > On 12:01 Mon 02 Jun , Hal Rosenstock wrote: > > > > > > Just seems safer to me to implement without using this. > > > > Without using this we need to write getpass() function. > > Yes. > > > Prefer to do it in the case if we will see some problem with glibc's one. > > But why implement using a known obsoleted function ? I cannot find *why* it was marked obsolete, it is not marked obsolete in glibc, finally why out own implementation will be "less obsolete" :). What wrong with doing this on demand if needed? Unless we know a reason. Sasha From hrosenstock at xsigo.com Mon Jun 2 12:32:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 12:32:38 -0700 Subject: [ofa-general] Re: [PATCH] OpenSM release notes: Update to 3.1.11 In-Reply-To: <20080602192247.GJ13775@sashak.voltaire.com> References: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> <20080602192247.GJ13775@sashak.voltaire.com> Message-ID: <1212435158.32691.109.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 22:22 +0300, Sasha Khapyorsky wrote: > On 07:00 Mon 02 Jun , Hal Rosenstock wrote: > > OpenSM release notes: Update to 3.1.11 > > > > Please apply to ofed_1_3 and master > > ofed_1_3 is closed and 3.1.11 is out already. ofed_1_3 is not closed. How do you know there won't be a 1.3.2 ? Why not fix it now ? > I can apply to the master, but difference (except one mentioned below) is > only version number and dates. Isn't this helpful there? Should have been done as part of 3.1.11. > > > > -1.4 Software Dependencies > > - > > -OpenSM depends on the installation of either OFED 1.3, OFED 1.2, OFED 1.1, > > -OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD > > -distribution), or Mellanox VAPI stacks. The qualified driver versions > > -are provided in Table 2, "Qualified IB Stacks". > > - > > -Also building of QoS manager policy file parser requires flex, and either > > -bison or byacc installed. > > Isn't this: > > commit f8e4b29248fd679d9678fe37acb72add1a48cfd6 > Author: Yevgeny Kliteynik > Date: Mon Mar 10 16:56:23 2008 +0200 > > opensm/osm_qos_parser: fixed compilation on byacc > > Hi Sasha. > > Fixing compilation with byacc (bug 932). > Please apply to ofed_1_3 and master. > > -- Yevgeny > > Signed-off-by: Yevgeny Kliteynik > Signed-off-by: Sasha Khapyorsky > > solve byacc issue ('byacc' is removed in the new text)? Yes, that's wrong. It should say "or byacc". It came from the OFED version :-( -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Mon Jun 2 12:46:08 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 22:46:08 +0300 Subject: [ofa-general] Re: [PATCH] OpenSM release notes: Update to 3.1.11 In-Reply-To: <1212435158.32691.109.camel@hrosenstock-ws.xsigo.com> References: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> <20080602192247.GJ13775@sashak.voltaire.com> <1212435158.32691.109.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602194608.GA14820@sashak.voltaire.com> On 12:32 Mon 02 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-02 at 22:22 +0300, Sasha Khapyorsky wrote: > > On 07:00 Mon 02 Jun , Hal Rosenstock wrote: > > > OpenSM release notes: Update to 3.1.11 > > > > > > Please apply to ofed_1_3 and master > > > > ofed_1_3 is closed and 3.1.11 is out already. > > ofed_1_3 is not closed. How do you know there won't be a 1.3.2 ? Why not > fix it now ? Because opensm-3.1.11 is released already (as tarball and git tree). So it should be next version, and doing whole new version for just updating date in RN looks as overkill for me. > > I can apply to the master, but difference (except one mentioned below) is > > only version number and dates. Isn't this helpful there? > > Should have been done as part of 3.1.11. Agree, but it is too late now. Sasha From hrosenstock at xsigo.com Mon Jun 2 12:50:43 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 12:50:43 -0700 Subject: [ofa-general] Re: [PATCH] OpenSM release notes: Update to 3.1.11 In-Reply-To: <20080602194608.GA14820@sashak.voltaire.com> References: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> <20080602192247.GJ13775@sashak.voltaire.com> <1212435158.32691.109.camel@hrosenstock-ws.xsigo.com> <20080602194608.GA14820@sashak.voltaire.com> Message-ID: <1212436243.32691.114.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 22:46 +0300, Sasha Khapyorsky wrote: > but it is too late now. Fine; then just update master. From sashak at voltaire.com Mon Jun 2 12:51:25 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 22:51:25 +0300 Subject: [ofa-general] Re: [PATCH] OpenSM/osm_sa_mcmember_record.c: Validate some more MGID bits for IPv6 SNM In-Reply-To: <1212415240.32691.32.camel@hrosenstock-ws.xsigo.com> References: <1212415240.32691.32.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602195125.GB14820@sashak.voltaire.com> On 07:00 Mon 02 Jun , Hal Rosenstock wrote: > OpenSM/osm_sa_mcmember_record.c: Include high order 24 bits of low 64 > MGID bits in validation as these bits are supposed to be 0 > > Signed-off-by: Hal Rosenstock > > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > index c14632d..73b3fc4 100644 > --- a/opensm/opensm/osm_sa_mcmember_record.c > +++ b/opensm/opensm/osm_sa_mcmember_record.c > @@ -1088,14 +1088,15 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > * YYYYYY is the last 24 bits of the port guid */ > #define PREFIX_MASK (0xff10ffff0000ffffULL) > #define PREFIX_SIGNATURE (0xff10601b00000000ULL) > -#define INT_ID_MASK (0x00000001ff000000ULL) > +#define INT_ID_MASK (0xffffff1ff000000ULL) Should be #define INT_ID_MASK (0xfffffff1ff000000ULL) (one more 'f' at begining), right? Sasha From hrosenstock at xsigo.com Mon Jun 2 12:57:08 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 02 Jun 2008 12:57:08 -0700 Subject: [ofa-general] Re: [PATCH] OpenSM/osm_sa_mcmember_record.c: Validate some more MGID bits for IPv6 SNM In-Reply-To: <20080602195125.GB14820@sashak.voltaire.com> References: <1212415240.32691.32.camel@hrosenstock-ws.xsigo.com> <20080602195125.GB14820@sashak.voltaire.com> Message-ID: <1212436628.32691.120.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 22:51 +0300, Sasha Khapyorsky wrote: > On 07:00 Mon 02 Jun , Hal Rosenstock wrote: > > OpenSM/osm_sa_mcmember_record.c: Include high order 24 bits of low 64 > > MGID bits in validation as these bits are supposed to be 0 > > > > Signed-off-by: Hal Rosenstock > > > > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > > index c14632d..73b3fc4 100644 > > --- a/opensm/opensm/osm_sa_mcmember_record.c > > +++ b/opensm/opensm/osm_sa_mcmember_record.c > > @@ -1088,14 +1088,15 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > > * YYYYYY is the last 24 bits of the port guid */ > > #define PREFIX_MASK (0xff10ffff0000ffffULL) > > #define PREFIX_SIGNATURE (0xff10601b00000000ULL) > > -#define INT_ID_MASK (0x00000001ff000000ULL) > > +#define INT_ID_MASK (0xffffff1ff000000ULL) > > Should be > > #define INT_ID_MASK (0xfffffff1ff000000ULL) > > (one more 'f' at begining), right? Yes. -- Hal > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Mon Jun 2 13:00:59 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 23:00:59 +0300 Subject: [ofa-general] Re: [PATCH] OpenSM release notes: Update to 3.1.11 In-Reply-To: <1212436243.32691.114.camel@hrosenstock-ws.xsigo.com> References: <1212415229.32691.30.camel@hrosenstock-ws.xsigo.com> <20080602192247.GJ13775@sashak.voltaire.com> <1212435158.32691.109.camel@hrosenstock-ws.xsigo.com> <20080602194608.GA14820@sashak.voltaire.com> <1212436243.32691.114.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602200059.GC14820@sashak.voltaire.com> On 12:50 Mon 02 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-02 at 22:46 +0300, Sasha Khapyorsky wrote: > > but it is too late now. > > Fine; then just update master. Added file there (after reverting byacc removing). Sasha From eli at dev.mellanox.co.il Mon Jun 2 13:04:42 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 2 Jun 2008 23:04:42 +0300 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Initialize DS field for stamping In-Reply-To: References: <1212403058.17351.7.camel@mtls03> Message-ID: <4e6a6b3c0806021304k5cb27de5oebb8e57ac007dc1a@mail.gmail.com> On Mon, Jun 2, 2008 at 7:26 PM, Roland Dreier wrote: > This is a fix for your "optimize stamping" patch, right? Right. > > I'll roll it into that patch. > I thought so too but didn't know what you'd prefer so I sent a different patch. From sashak at voltaire.com Mon Jun 2 13:20:15 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 23:20:15 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/sim_cmd.c: Cosmetic changes to help In-Reply-To: <1212426311.32691.60.camel@hrosenstock-ws.xsigo.com> References: <1212426311.32691.60.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602202015.GE14820@sashak.voltaire.com> On 10:05 Mon 02 Jun , Hal Rosenstock wrote: > ibsim/sim_cmd.c: Cosmetic changes to help > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Jun 2 13:31:14 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 23:31:14 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_lid_mgr.c: Eliminate some potential NULL pointer dereferences In-Reply-To: <1212416729.32691.43.camel@hrosenstock-ws.xsigo.com> References: <1212416729.32691.43.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602203114.GF14820@sashak.voltaire.com> On 07:25 Mon 02 Jun , Hal Rosenstock wrote: > opensm/osm_lid_mgr.c: Eliminate some potential NULL pointer dereferences > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Jun 2 13:36:28 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 23:36:28 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_lin_fwd_tbl.c: Eliminate potential NULL pointer dereference In-Reply-To: <1212426317.32691.62.camel@hrosenstock-ws.xsigo.com> References: <1212426317.32691.62.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602203628.GH14820@sashak.voltaire.com> On 10:05 Mon 02 Jun , Hal Rosenstock wrote: > opensm/osm_lin_fwd_tbl.c: Eliminate potential NULL pointer dereference > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Jun 2 13:45:26 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Jun 2008 23:45:26 +0300 Subject: [ofa-general] Re: [PATCH] OpenSM/osm_sa_mcmember_record.c: Validate some more MGID bits for IPv6 SNM In-Reply-To: <1212415240.32691.32.camel@hrosenstock-ws.xsigo.com> References: <1212415240.32691.32.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080602204526.GJ14820@sashak.voltaire.com> On 07:00 Mon 02 Jun , Hal Rosenstock wrote: > OpenSM/osm_sa_mcmember_record.c: Include high order 24 bits of low 64 > MGID bits in validation as these bits are supposed to be 0 > > Signed-off-by: Hal Rosenstock Applied (with discussed fix). Thanks. Sasha From arlin.r.davis at intel.com Mon Jun 2 14:14:40 2008 From: arlin.r.davis at intel.com (Arlin Davis) Date: Mon, 2 Jun 2008 14:14:40 -0700 Subject: [ofa-general] [PATCH 1/1] [v2.0] dapl extension: dapli_post_ext should always allocate cookie for requests. Message-ID: <001f01c8c4f5$abce8850$bfbf020a@amr.corp.intel.com> Request extensions didn't allocate cookie if completion was suppressed which resulted segfault during provider post call. Provider's expect cookie for wr_id, even with surpressed completions, to handle events during errors. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_cma/dapl_ib_extensions.c | 36 +++++++++++++-------------------- 1 files changed, 14 insertions(+), 22 deletions(-) diff --git a/dapl/openib_cma/dapl_ib_extensions.c b/dapl/openib_cma/dapl_ib_extensions.c index 52b238f..1402057 100755 --- a/dapl/openib_cma/dapl_ib_extensions.c +++ b/dapl/openib_cma/dapl_ib_extensions.c @@ -173,7 +173,7 @@ dapli_post_ext( IN DAT_EP_HANDLE ep_handle, { DAPL_EP *ep_ptr; ib_qp_handle_t qp_ptr; - DAPL_COOKIE *cookie; + DAPL_COOKIE *cookie = NULL; DAT_RETURN dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -191,25 +191,19 @@ dapli_post_ext( IN DAT_EP_HANDLE ep_handle, /* * Synchronization ok since this buffer is only used for send * requests, which aren't allowed to race with each other. - * only if completion is expected */ - if (!(DAT_COMPLETION_SUPPRESS_FLAG & flags)) { - - dat_status = dapls_dto_cookie_alloc( - &ep_ptr->req_buffer, - DAPL_DTO_TYPE_EXTENSION, - user_cookie, - &cookie ); - - if ( dat_status != DAT_SUCCESS ) - goto bail; + dat_status = dapls_dto_cookie_alloc(&ep_ptr->req_buffer, + DAPL_DTO_TYPE_EXTENSION, + user_cookie, + &cookie); + if (dat_status != DAT_SUCCESS) + goto bail; - /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc(&ep_ptr->req_count); - } + /* + * Take reference before posting to avoid race conditions with + * completions + */ + dapl_os_atomic_inc(&ep_ptr->req_count); /* * Invoke provider specific routine to post DTO @@ -226,10 +220,8 @@ dapli_post_ext( IN DAT_EP_HANDLE ep_handle, flags); if (dat_status != DAT_SUCCESS) { - if ( cookie != NULL ) { - dapl_os_atomic_dec(&ep_ptr->req_count); - dapls_cookie_dealloc(&ep_ptr->req_buffer, cookie); - } + dapl_os_atomic_dec(&ep_ptr->req_count); + dapls_cookie_dealloc(&ep_ptr->req_buffer, cookie); } bail: -- 1.5.2.5 From rdreier at cisco.com Mon Jun 2 14:47:51 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 14:47:51 -0700 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: (Ron Livne's message of "Mon, 2 Jun 2008 17:32:35 +0000 (UTC)") References: Message-ID: actually, looking at this further: > +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb) given that the real change is to multicast attach and is not a property of QPs, wouldn't it be simpler to simply add a flags argument to ib_attach_mcast() and use a flag bit to request that messages not be looped back per mcast attach? - R. From sashak at voltaire.com Mon Jun 2 18:20:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 3 Jun 2008 04:20:02 +0300 Subject: [ofa-general] Re: [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <1207861929.7695.165.camel@cardanus.llnl.gov> References: <1207861929.7695.165.camel@cardanus.llnl.gov> Message-ID: <20080603012002.GN14820@sashak.voltaire.com> Hi Al, Sorry about huge delay. Some questions are below. On 14:12 Thu 10 Apr , Al Chu wrote: > This is the primary patch that fiddles with the path recommendation > code. A few notes: > > 1) b/c I want to keep track of how many remote destinations there can > be, the 'remote_guids' array now stores all remote destinations, not > just the ones we have already forwarded to. > > 2) b/c I may need to free memory, I now "goto Exit" instead of just > calling 'return' many times. > > 3) Although the option is called 'port_offsetting', I actually "offset" > both the remote destination I send to and the port pointing towards that > remote destination. > > Al > > -- > Albert Chu > chu11 at llnl.gov > 925-422-5311 > Computer Scientist > High Performance Systems Division > Lawrence Livermore National Laboratory > From 57eb4d9bf55fbbbf39dc1c7ddfeeb2cae4776ef0 Mon Sep 17 00:00:00 2001 > From: Albert L. Chu > Date: Thu, 20 Mar 2008 16:23:13 -0700 > Subject: [PATCH] implement port_offsetting > > > Signed-off-by: Albert L. Chu > --- > opensm/include/opensm/osm_switch.h | 7 +- > opensm/opensm/osm_dump.c | 3 +- > opensm/opensm/osm_switch.c | 246 ++++++++++++++++++++++++++++++++---- > opensm/opensm/osm_ucast_mgr.c | 14 ++- > 4 files changed, 239 insertions(+), 31 deletions(-) > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > index 2624d5f..45f4718 100644 > --- a/opensm/include/opensm/osm_switch.h > +++ b/opensm/include/opensm/osm_switch.h > @@ -997,7 +997,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > IN const boolean_t dor, > IN OUT osm_switch_guid_count_t * remote_guids, > IN OUT uint16_t * p_num_remote_guids, > - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used); > + IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used, > + IN uint16_t port_offsetting_lids_per_port); > /* > * PARAMETERS > * p_sw > @@ -1031,6 +1032,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > * [in out] The specific osm_switch_guid_count_t used > * in switch recommendations. > * > +* port_offsetting_lids_per_port > +* [in] If > 0, indicates lids_per_port to use with > +* port_offsetting option. > +* > * RETURN VALUE > * Returns the recommended port on which to route this LID. > * > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > index 2bac75a..37842ac 100644 > --- a/opensm/opensm/osm_dump.c > +++ b/opensm/opensm/osm_dump.c > @@ -219,7 +219,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > /* No LMC Optimization */ > best_port = osm_switch_recommend_path(p_sw, p_port, > lid_ho, TRUE, dor, > - NULL, NULL, NULL); > + NULL, NULL, > + NULL, 0); > fprintf(file, "No %u hop path possible via port %u!", > best_hops, best_port); > } > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > index f346b25..8e3342b 100644 > --- a/opensm/opensm/osm_switch.c > +++ b/opensm/opensm/osm_switch.c > @@ -55,6 +55,15 @@ > #include > #include > > +/* Local structs */ > +struct osm_switch_remote_dest { > + uint32_t total_paths; > + uint32_t min_paths; > + uint8_t ports[256]; /* 256 b/c max_ports is a uint8_t */ > + unsigned int ports_count; > + osm_switch_guid_count_t *p_remote_guid; > +}; > + > /********************************************************************** > **********************************************************************/ > cl_status_t > @@ -304,9 +313,32 @@ osm_switch_find_guid_count(IN const osm_switch_t * const p_sw, > 1); > } > > +/********************************************************************** > + **********************************************************************/ > + > +/* greatest common divisor */ > +static unsigned int > +_gcd(unsigned int a, unsigned int b) > +{ > + unsigned int t; > + while (b != 0) { > + t = b; > + b = a % b; > + a = t; > + } > + return a; > +} > + > +/* least common multiple */ > +static unsigned int > +_lcm(unsigned int a, unsigned int b) > +{ > + return ((a*b) / _gcd(a,b)); > +} > > /********************************************************************** > **********************************************************************/ > + > uint8_t > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > IN osm_port_t * p_port, > @@ -315,7 +347,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > IN const boolean_t dor, > IN OUT osm_switch_guid_count_t * remote_guids, > IN OUT uint16_t * p_num_remote_guids, > - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used) > + IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used, > + IN uint16_t port_offsetting_lids_per_port) > { > /* > We support an enhanced LMC aware routing mode: > @@ -356,6 +389,20 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > osm_node_t *p_rem_node; > osm_node_t *p_rem_node_first = NULL; > osm_switch_guid_count_t *p_remote_guid = NULL; > + /* > + These vars track information for port offsetting. > + */ > + boolean_t port_offsetting = remote_guids && p_num_remote_guids > + && p_remote_guid_count_used && port_offsetting_lids_per_port; > + struct osm_switch_remote_dest * remote_dests = NULL; > + struct osm_switch_remote_dest * p_remote_dest = NULL; > + uint32_t num_remote_dests = 0; > + uint32_t total_paths_count = 0; > + uint32_t num_potential_ports = 0; > + uint32_t lcm = 0; > + uint32_t indx = 0; > + boolean_t dest_found = FALSE; > + unsigned int i; > > CL_ASSERT(lid_ho > 0); > > @@ -378,9 +425,22 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > num_ports = p_sw->num_ports; > > + if (port_offsetting) { > + remote_dests = malloc(sizeof(struct osm_switch_remote_dest) * num_ports); > + if (remote_dests == NULL) { > + osm_log(p_sw->p_log, OSM_LOG_ERROR, > + "osm_switch_recommend_path: " > + "Cannot allocate array. Insufficient memory: " > + "Disabling port_offsetting\n"); > + port_offsetting = 0; > + } > + } > + > least_hops = osm_switch_get_least_hops(p_sw, base_lid); > - if (least_hops == OSM_NO_PATH) > - return (OSM_NO_PATH); > + if (least_hops == OSM_NO_PATH) { > + best_port = OSM_NO_PATH; > + goto Exit; > + } > > /* > First, inquire with the forwarding table for an existing > @@ -417,8 +477,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > in the forwarding tables that he wants to be overridden by the > minimum hop function. > */ > - if (hops == least_hops) > - return (port_num); > + if (hops == least_hops) { > + best_port = port_num; > + goto Exit; > + } > } > } > } Maybe to move remote_dests allocation to be here and so minimize needed 'goto Exit' changes? > @@ -475,7 +537,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > port_num); > > /* If not update the least hops for this case */ > - if (!p_remote_guid) { > + if (!p_remote_guid > + || !p_remote_guid->forwarded_to) { > if (check_count < least_paths_other_sys) { > least_paths_other_sys = check_count; > best_port_other_sys = port_num; > @@ -489,7 +552,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > port_num); > > /* If not update the least hops for this case */ > - if (!p_remote_guid > + if ((!p_remote_guid > + || !p_remote_guid->forwarded_to) > && check_count < least_paths_other_nodes) { > least_paths_other_nodes = check_count; > best_port_other_node = port_num; > @@ -498,6 +562,50 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > /* else prior sys and node guid already used */ > > } /* same sys found */ > + > + /* Store the new sys/node guid that we haven't seen yet */ > + if (!p_remote_guid) { > + p_rem_physp = osm_physp_get_remote(p_physp); > + p_rem_node = osm_physp_get_node_ptr(p_rem_physp); > + memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), > + &(p_rem_node->node_info.sys_guid), > + sizeof(uint64_t)); > + memcpy(&(remote_guids[*p_num_remote_guids].node_guid), > + &(p_rem_node->node_info.node_guid), > + sizeof(uint64_t)); > + remote_guids[*p_num_remote_guids].forwarded_to = 0; > + p_remote_guid = &remote_guids[*p_num_remote_guids]; > + (*p_num_remote_guids)++; > + } > + } > + > + if (port_offsetting) { > + /* Keep track of the destinations we've seen so far */ > + p_remote_dest = NULL; > + for (i = 0; i < num_remote_dests; i++) { > + if (!memcmp(p_remote_guid, > + (&remote_dests[i])->p_remote_guid, > + sizeof(struct osm_switch_remote_dest))) { Should there be sizeof(osm_switch_guid_count_t)? Sasha > + p_remote_dest = &remote_dests[i]; > + break; > + } > + } > + if (!p_remote_dest) { > + p_remote_dest = &remote_dests[num_remote_dests]; > + p_remote_dest->p_remote_guid = p_remote_guid; > + p_remote_dest->total_paths = 0; > + p_remote_dest->min_paths = 0xFFFFFFFF; > + p_remote_dest->ports_count = 0; > + num_remote_dests++; > + } > + p_remote_dest->total_paths += check_count; > + if (check_count < p_remote_dest->min_paths) > + p_remote_dest->min_paths = check_count; > + p_remote_dest->ports[p_remote_dest->ports_count] = port_num; > + p_remote_dest->ports_count++; > + > + total_paths_count += check_count; > + num_potential_ports++; > } > > /* routing for LMC mode */ > @@ -523,21 +631,23 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > best_port = port_num; > least_paths = check_count; > if (routing_for_lmc > - && p_remote_guid > + && p_remote_guid->forwarded_to > && p_remote_guid->forwarded_to < least_forwarded_to) > least_forwarded_to = p_remote_guid->forwarded_to; > } > else if (routing_for_lmc > - && p_remote_guid > && check_count == least_paths > + && p_remote_guid->forwarded_to > && p_remote_guid->forwarded_to < least_forwarded_to) { > least_forwarded_to = p_remote_guid->forwarded_to; > best_port = port_num; > } > } > > - if (port_found == FALSE) > - return (OSM_NO_PATH); > + if (port_found == FALSE) { > + best_port = OSM_NO_PATH; > + goto Exit; > + } > > /* > if we are in enhanced routing mode and the best port is not > @@ -555,24 +665,110 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > remote_guids, > p_num_remote_guids, > best_port); > + /* Must be stored */ > + CL_ASSERT(p_remote_guid); > + *p_remote_guid_count_used = p_remote_guid; > + } > > - if (!p_remote_guid) { > - /* track the remote node and system of the port used. */ > - p_physp = osm_node_get_physp_ptr(p_sw->p_node, best_port); > - p_rem_physp = osm_physp_get_remote(p_physp); > - p_rem_node = osm_physp_get_node_ptr(p_rem_physp); > - memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), > - &(p_rem_node->node_info.sys_guid), > - sizeof(uint64_t)); > - memcpy(&(remote_guids[*p_num_remote_guids].node_guid), > - &(p_rem_node->node_info.node_guid), > - sizeof(uint64_t)); > - remote_guids[*p_num_remote_guids].forwarded_to = 0; > - (*p_num_remote_guids)++; > + /* > + * As an example of what we're trying to do with port > + * offsetting, assume LMC = 2 and we are trying to route > + * the lids of 4 ports. The lids of these 4 ports are: > + * > + * (1,2,3,4) > + * (5,6,7,8) > + * (9,10,11,12) > + * (13,14,15,16) > + * > + * Suppose forwarding to all these lids goes through > + * 4 specific switch ports. If we just cycle through > + * ports and lids in a normal iterative fashion, we would > + * normally forward out ports in this manner. > + * > + * switch port 1: 1, 5, 9, 13 > + * switch port 2: 2, 6, 10, 14 > + * switch port 3: 3, 7, 11, 15 > + * switch port 4: 4, 8, 12, 1 > + * > + * Note that the base lid of each port (lids 1, 5, 9, 13) > + * are all routed out of switch port 1. Thus, if the user > + * only uses the base lid of each port, they will get pretty > + * bad performance. We will try to get this layout instead. > + * > + * switch port 1: 1, 8, 11, 14 > + * switch port 2: 2, 5, 12, 15 > + * switch port 3: 3, 6, 9, 16 > + * switch port 4: 4, 7, 10, 13 > + * > + * where switch ports are distributed in a more even manner. > + * The base lid of each port is now distributed evenly > + * across all 4 switch ports. The remaining lids are still > + * distributed evenly across all the remaining switch ports. > + * > + * In order to accomplish this, we (effectively) will iterate > + * through all ports like before, but instead of iterating from > + * 0 to N-1 all the time, we will select the starting index > + * based on the number of paths we have routed thus far. > + */ > + > + /* We will not do port offsetting if num_potential_ports == 1 > + * b/c there is no offsetting that can be done. > + */ > + if (port_offsetting > + && best_port > + && num_potential_ports > 1) { > + /* Select which destination we want to forward to with our > + * offsetting loop. > + */ > + lcm = _lcm(port_offsetting_lids_per_port, > + num_remote_dests); > + indx = (total_paths_count / lcm) % num_remote_dests; > + for (i = 0; i < num_remote_dests; i++) { > + p_remote_dest = &remote_dests[indx]; > + p_remote_guid = p_remote_dest->p_remote_guid; > + > + if (p_remote_guid->forwarded_to == least_forwarded_to > + && p_remote_dest->min_paths == least_paths) { > + dest_found = TRUE; > + break; > + } > + > + indx++; > + if (indx >= num_remote_dests) > + indx = 0; > + } > + > + /* Then we "offset" within the potential ports we could > + * forward out of for this specific destination. > + */ > + if (dest_found && p_remote_dest) { > + lcm = _lcm(port_offsetting_lids_per_port, > + p_remote_dest->ports_count); > + indx = (p_remote_dest->total_paths / lcm) % p_remote_dest->ports_count; > + for (i = 0; i < p_remote_dest->ports_count; i++) { > + port_num = p_remote_dest->ports[indx]; > + check_count = > + osm_port_prof_path_count_get(&p_sw->p_prof[port_num]); > + if (check_count == least_paths) { > + if (best_port != port_num) > + osm_log(p_sw->p_log, OSM_LOG_DEBUG, > + "osm_switch_recommend_path: " > + "best port offsetted: %d to %d\n", > + best_port, port_num); > + best_port = port_num; > + *p_remote_guid_count_used = p_remote_dest->p_remote_guid; > + break; > + } > + indx++; > + if (indx >= p_remote_dest->ports_count) > + indx = 0; > + } > } > - *p_remote_guid_count_used = p_remote_guid; > } > > +Exit: > + if (remote_dests) > + free(remote_dests); > return (best_port); > } > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > index 938db84..501c2c7 100644 > --- a/opensm/opensm/osm_ucast_mgr.c > +++ b/opensm/opensm/osm_ucast_mgr.c > @@ -212,18 +212,23 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > osm_switch_guid_count_t *remote_guids = NULL; > uint16_t num_used_guids = 0; > osm_switch_guid_count_t *p_remote_guid_used = NULL; > + uint16_t port_offsetting_lids_per_port = 0; > > OSM_LOG_ENTER(p_mgr->p_log); > > if (lids_per_port > 1) { > - remote_guids = malloc(sizeof(osm_switch_guid_count_t) * lids_per_port); > + uint8_t num_ports = p_sw->num_ports; > + remote_guids = malloc(sizeof(osm_switch_guid_count_t) * num_ports); > if (remote_guids == NULL) { > osm_log(p_mgr->p_log, OSM_LOG_ERROR, > "__osm_ucast_mgr_process_port: ERR 3A09: " > "Cannot allocate array. Insufficient memory\n"); > goto Exit; > } > - memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * lids_per_port); > + memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * num_ports); > + > + if (p_mgr->p_subn->opt.port_offsetting) > + port_offsetting_lids_per_port = lids_per_port; > } > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > @@ -270,14 +275,15 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > p_mgr->is_dor, > remote_guids, > &num_used_guids, > - &p_remote_guid_used); > + &p_remote_guid_used, > + port_offsetting_lids_per_port); > } > else > port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > p_mgr->p_subn-> > ignore_existing_lfts, > p_mgr->is_dor, > - NULL, NULL, NULL); > + NULL, NULL, NULL, 0); > > /* > There might be no path to the target > -- > 1.5.1 > From rdreier at cisco.com Mon Jun 2 20:12:31 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 02 Jun 2008 20:12:31 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Initialize DS field for stamping In-Reply-To: <1212403058.17351.7.camel@mtls03> (Eli Cohen's message of "Mon, 02 Jun 2008 13:37:38 +0300") References: <1212403058.17351.7.camel@mtls03> Message-ID: OK, I rolled this into the optimize stamping patch, with one tweak: > + if (qp->sq_max_wqes_per_wr < 2) this seems like a really awkward way of saying if (qp->sq_max_wqes_per_wr == 1) so I converted it to test that. - R. From eli at dev.mellanox.co.il Mon Jun 2 21:15:01 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 3 Jun 2008 07:15:01 +0300 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Initialize DS field for stamping In-Reply-To: References: <1212403058.17351.7.camel@mtls03> Message-ID: <4e6a6b3c0806022115p158e00e2h3b687b9333632d85@mail.gmail.com> On Tue, Jun 3, 2008 at 6:12 AM, Roland Dreier wrote: > OK, I rolled this into the optimize stamping patch, with one tweak: > > > + if (qp->sq_max_wqes_per_wr < 2) > > this seems like a really awkward way of saying > > if (qp->sq_max_wqes_per_wr == 1) Agree > > so I converted it to test that. > From olga.shern at gmail.com Mon Jun 2 21:25:30 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Tue, 3 Jun 2008 07:25:30 +0300 Subject: [ofa-general] [PATCH 1/3] mcast loopback block In-Reply-To: <1212431118.22429.24.camel@chromite.mv.qlogic.com> References: <1212426598.22429.2.camel@chromite.mv.qlogic.com> <1212429452.22429.19.camel@chromite.mv.qlogic.com> <1212431118.22429.24.camel@chromite.mv.qlogic.com> Message-ID: On 6/2/08, Ralph Campbell wrote: > > On Mon, 2008-06-02 at 21:06 +0300, Olga Shern (Voltaire) wrote: > > > > > > On 6/2/08, Ralph Campbell wrote: > > On Mon, 2008-06-02 at 20:52 +0300, Olga Shern (Voltaire) > > wrote: > > > > > > > > > On 6/2/08, Ralph Campbell wrote: > > > What do you mean by "block loopback"? > > > Does this mean a posted send WR for a multicast > > packet > > > that matches the local LID is not sent at all or > > > the packet is sent on the wire but not sent to > > > the local QP? Or is it something else? > > > > > > > > > It controls the ConnectX HCA multicast packets block > > loopback > > > (blck_lb) for self QP. The patch is designed to enable or > > > > > > disable blocking of all multicast packets on self QP created > > on the > > > ConnectX HCA > > > > I understand that the patch enables/disables the bit. > > You didn't really answer my question which is what does > > the bit really do. I'm pretty sure it means that multicast > > packets aren't forwarded to any locally attached QPs > > but I wanted to be sure that is what it does. > > > > > > No, > > It isn't disabled for every QP (by default loopback is enabled), > > when you attach QP to multicast group you can decide to disable it > > (the decision is done per QP) > > So, in other words, if you create a QP with this option and then > attach it to a multicast group and a multicast packet is posted > on the send work request queue of the "disabled" QP, it won't > receive a copy of the packet it sent. If some other QP on the > same HCA is also attached to the same multicast group, it will > receive a copy of the multicast packet. > > Yes, you are rigth -------------- next part -------------- An HTML attachment was scrubbed... URL: From yevgenyp at mellanox.co.il Mon Jun 2 23:26:18 2008 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Tue, 03 Jun 2008 09:26:18 +0300 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> Message-ID: <4844E40A.9080209@mellanox.co.il> Roland Dreier wrote: > > Is there any update on this patch? > > Not that I have seen. > I resent the two patches that handle multiple completion vectors last week. What is the status for these patches? Are there further changes that need to be done, or will these patches be applied? Thanks, Yevgeny From ogerlitz at voltaire.com Mon Jun 2 23:55:43 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 03 Jun 2008 09:55:43 +0300 Subject: [ofa-general] [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <1207861929.7695.165.camel@cardanus.llnl.gov> References: <1207861929.7695.165.camel@cardanus.llnl.gov> Message-ID: <4844EAEF.8080605@voltaire.com> Al Chu wrote: > This is the primary patch that fiddles with the path recommendation > code. A few notes: > > 1) b/c I want to keep track of how many remote destinations there can > be, the 'remote_guids' array now stores all remote destinations, not > just the ones we have already forwarded to. > > 2) b/c I may need to free memory, I now "goto Exit" instead of just > calling 'return' many times. > > 3) Although the option is called 'port_offsetting', I actually "offset" > both the remote destination I send to and the port pointing towards that > remote destination. Hi Al, Sasha, May I ask what is port offsetting? also can you guys try harder such that patches to the management packages have an explanatory and not empty... change log that matches the standards applied in other domains (eg the Linux kernel)? Or. From 5info5 at cd-cool.com Tue Jun 3 01:16:30 2008 From: 5info5 at cd-cool.com (ajai ajay) Date: Tue, 03 Jun 2008 08:16:30 +0000 Subject: [ofa-general] timexx silver Message-ID: <000501c8c561$03ba4271$e0258897@mcehi> " My order arrived yesterday via registered mail in good order THE WATCH IS BEAUTIFUL AND EVEN BETTER THAN I EXPECTED." Try it for yourself - u will be amazed!! - The worlds largest online retailer of luxury products, including: Rolex Sports Models Rolex Datejusts Breitling Cartier Porsche Design Dolce & Gabbana Dior Gucci Hermes Watches Patek Philippe Visit - www.ccsiies.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at lists.openfabrics.org Tue Jun 3 03:09:12 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 3 Jun 2008 03:09:12 -0700 (PDT) Subject: [ofa-general] ofa_1_3_kernel 20080603-0200 daily build status Message-ID: <20080603100912.8A7DDE60DA8@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_3/linux-2.6.git git_branch: ofed_kernel Common build parameters: --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-user_mad-mod --with-user_access-mod --with-mthca-mod --with-mlx4-mod --with-core-mod --with-addr_trans-mod --with-rds-mod --with-cxgb3-mod --with-nes-mod Passed: Passed on i686 with 2.6.15-23-server Passed on i686 with linux-2.6.13 Passed on i686 with linux-2.6.12 Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.14 Passed on i686 with linux-2.6.15 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.12 Passed on x86_64 with linux-2.6.15 Passed on x86_64 with linux-2.6.13 Passed on x86_64 with linux-2.6.14 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.12 Passed on ia64 with linux-2.6.13 Passed on ia64 with linux-2.6.15 Passed on ia64 with linux-2.6.14 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.22 Passed on powerpc with linux-2.6.12 Passed on powerpc with linux-2.6.14 Passed on powerpc with linux-2.6.13 Passed on powerpc with linux-2.6.15 Passed on ppc64 with linux-2.6.13 Passed on ppc64 with linux-2.6.12 Passed on ppc64 with linux-2.6.14 Passed on ppc64 with linux-2.6.15 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.24 Passed on ppc64 with linux-2.6.19 Failed: From tziporet at dev.mellanox.co.il Tue Jun 3 04:20:24 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 03 Jun 2008 14:20:24 +0300 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: References: Message-ID: <484528F8.4010602@mellanox.co.il> Roland Dreier wrote: > > + IB_DEVICE_BLOCK_LOOPBACK; > > Does all firmware support this so it can be enabled unconditionally? > Yes Tziporet From sashak at voltaire.com Tue Jun 3 04:34:30 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 3 Jun 2008 14:34:30 +0300 Subject: [ofa-general] [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <4844EAEF.8080605@voltaire.com> References: <1207861929.7695.165.camel@cardanus.llnl.gov> <4844EAEF.8080605@voltaire.com> Message-ID: <20080603113430.GB4616@sashak.voltaire.com> Hi, On 09:55 Tue 03 Jun , Or Gerlitz wrote: > > May I ask what is port offsetting? It was described in great details in 0/3 message. > also can you guys try harder such that > patches to the management packages have an explanatory and not empty... I don't remember empty commits, but in general of course better log message is better. Sasha From ramachandra.kuchimanchi at qlogic.com Tue Jun 3 05:14:23 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 3 Jun 2008 17:44:23 +0530 Subject: [ofa-general] Re: [PATCH v3 01/13] QLogic VNIC: Driver - netdev implementation In-Reply-To: <20080529102752.584147ee@extreme> References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095423.9943.77528.stgit@localhost.localdomain> <20080529102752.584147ee@extreme> Message-ID: <71d336490806030514k580418e3n23790eb793e99381@mail.gmail.com> On Thu, May 29, 2008 at 10:57 PM, Stephen Hemminger wrote: > On Thu, 29 May 2008 15:24:23 +0530 > Ramachandra K wrote: > >> +module_param(vnic_debug, uint, 0444); >> +MODULE_PARM_DESC(vnic_debug, "Enable debug tracing if > 0"); > > maybe migrate this to ethtool msg_level? > We investigated this and found that we cannot map all of our Infiniband level communication errors etc to the ethtool message levels. One option could be to add support for Ethtool message levels in addition to the vnic_debug controlled messages. The Ethtool support would make this driver conform to ethtool similar to other Ethernet drivers and the vnic_debug messages will help in debugging the Infiniband related errors. Is ethtool support mandatory for a merge to 2.6.27 or can we consider it as a future enhancement while we first focus on moving this driver to use CONFIGFS ? Regards, Ram From ramachandra.kuchimanchi at qlogic.com Tue Jun 3 05:24:19 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 3 Jun 2008 17:54:19 +0530 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> Message-ID: <71d336490806030524r6aa6fed6rdb166efb11385969@mail.gmail.com> Roland, On Sun, Jun 1, 2008 at 11:16 AM, Roland Dreier wrote: > > And yes, multiple values per sysfs file are not allowed, sorry, please > > change this. If you need to configure your device through an interface > > like this, consider using configfs instead, that is what it is there > > for. > > Makes sense... I know that the SRP initiator uses the method of multiple > 'token=' entries passed into sysfs, but the excuse is that SRP > was merged before configfs. OK. We will investigate moving the qlgc_vnic driver to configfs. What is the deadline by which we need to be ready for a merge to 2.6.27 ? Regards, Ram From tziporet at mellanox.co.il Tue Jun 3 05:25:40 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Tue, 3 Jun 2008 15:25:40 +0300 Subject: [ofa-general] OFED meeting sumary on June 02, 08 In-Reply-To: <6C2C79E72C305246B504CBA17B5500C904282934@mtlexch01.mtl.com> Message-ID: <6C2C79E72C305246B504CBA17B5500C904283063@mtlexch01.mtl.com> OFED meeting summary on June 2, 08 Summary: -------- 1. OFED 1.3.1 release should be out today 2. OFED 1.4 development - ongoing 3. Decided to elect a new chair and co-chair to the EWG 4. Decided on libraries backward compatibility process Details: -------- 1. OFED 1.3.1 status: Release is expected on Tuesday June 3. 2. OFED 1.4: - Kernel rebase status: Backports are available for all kernels between 2.6.26-rc2 and kernel 2.6.16. Work is ongoing and expected to be completed this week. URL: git://git.openfabrics.org/ofed_1_4/linux-2.6.git ofed_kernel - There is a request to delay the feature freeze to July to implement CMA support for IPv6. Prefer not to delay the general feature freeze but take this one as an exception. 3. Electing a chair and co-chair to the EWG: Decided on the following process for election: a. Jim Ryan will email the XWG and EWG requesting nominations and explaining the process. b. Time frame: Nominees will present themselves at the next ewg meeting on June 18. Chair and co-chair selection will be made in 4 weeks, by June 30. 4. Backward compatibility of libraries: Problem: Qlogic found late in OFED 1.3 release process that one of opensm libraries had changed the binary API, and this brake some of the customers' application. Decisions: 1. Need to preserve binary API of libraries and not break backward compatibility (even if it requires extra efforts) 2. In case binary API must be changed, then both versions will be included in OFED to prevent broken applications. Also - a plan for deprecation of the old library should be published. 3. Changes in the API must be done before RC1. Tziporet From ogerlitz at voltaire.com Tue Jun 3 05:50:46 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 03 Jun 2008 15:50:46 +0300 Subject: [ofa-general] [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <20080603113430.GB4616@sashak.voltaire.com> References: <1207861929.7695.165.camel@cardanus.llnl.gov> <4844EAEF.8080605@voltaire.com> <20080603113430.GB4616@sashak.voltaire.com> Message-ID: <48453E26.9030900@voltaire.com> Sasha Khapyorsky wrote: >> May I ask what is port offsetting? > > It was described in great details in 0/3 message. oh, I see it now, very well documented, sorry. > I don't remember empty commits, but in general of course better log > message is better. this patch change-log is empty Or. From hrosenstock at xsigo.com Tue Jun 3 06:01:28 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 03 Jun 2008 06:01:28 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/saquery.c: Update for change to osm_mad_pool_init API Message-ID: <1212498088.31387.42.camel@hrosenstock-ws.xsigo.com> infiniband-diags/saquery.c: Update for change to osm_mad_pool_init API Signed-off-by: Hal Rosenstock diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c index 3d4ab24..e678ef8 100644 --- a/infiniband-diags/src/saquery.c +++ b/infiniband-diags/src/saquery.c @@ -1290,7 +1290,7 @@ get_bind_handle(void) vendor = osm_vendor_new(&log_osm, sa_timeout_ms); osm_mad_pool_construct(&mad_pool); - if ((status = osm_mad_pool_init(&mad_pool, &log_osm)) != IB_SUCCESS) { + if ((status = osm_mad_pool_init(&mad_pool)) != IB_SUCCESS) { fprintf(stderr, "Failed to init mad pool: %s\n", ib_get_err_str(status)); exit(-1); From sashak at voltaire.com Tue Jun 3 06:18:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 3 Jun 2008 16:18:21 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/saquery.c: Update for change to osm_mad_pool_init API In-Reply-To: <1212498088.31387.42.camel@hrosenstock-ws.xsigo.com> References: <1212498088.31387.42.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080603131821.GD4616@sashak.voltaire.com> On 06:01 Tue 03 Jun , Hal Rosenstock wrote: > infiniband-diags/saquery.c: Update for change to osm_mad_pool_init API > > Signed-off-by: Hal Rosenstock Nice catch. Applied. Thanks. Sasha From Thomas.Talpey at netapp.com Tue Jun 3 06:21:35 2008 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Tue, 03 Jun 2008 09:21:35 -0400 Subject: [ofa-general] [PATCH] rds: use separate ports for TCP and IB In-Reply-To: <200806021025.46789.olaf.kirch@oracle.com> References: <20080528225549.GC6288@opengridcomputing.com> <200806021025.46789.olaf.kirch@oracle.com> Message-ID: At 04:25 AM 6/2/2008, Olaf Kirch wrote: >On Friday 30 May 2008 18:29:01 Talpey, Thomas wrote: >> Yes, at the moment TCP to the iWARP NIC is like talking to a different >> host. But, RDMA-aware versions of a given protocol still need a second >> port, unless there is explicit upper layer support for initiating the MPA >> exchange. We have the same issue with NFSv3/RDMA, and we have >> applied for a second port (the application is still pending within IANA). > >This may also become a problem for RDS, but in a different sense. >If you bind an RDS socket to a specific IP address, this also selects >a transport. If you bind to an IP address owned by ib0, you select >the IB transport. If you bind to an IP address owned by say eth0, you >select the TCP transport. I think this is a bit of an overloading, and will lead to issues down the road. IP addresses really aren't about interfaces, they're about hosts. For example, on TCP (etc), hosts will respond to any of their IP addresses on any of their interfaces (IP routing often depends on this). I suspect it would be better to make the transport selection explicit. IPv4/IPv6 selection may fall into the same issue later. >Now if the one and only NIC in the system is an iWARP NIC, which transport >should we select? Both iWARP and TCP can be legitimate choices, depending >on the hosts we're talking to (and how do we find out whether the remote >node supports iWARP or not?) > >I think for now it's okay to default to iWARP and punt if the remote >doesn't support it. But in the long run this is something that needs >to be addressed - either in RDS, or in the way ofed handles iWARP NICs. Here I think you've made an even bigger assumption - that not only does your interface support iWARP, you're assuming the peer's does too! But because the kernel doesn't plumb iWARP/TCP into the stack I guess it's moot. Still... Is RDS able to use MPA negotiation to mutually agree to enable RDMA? If so, you might consider doing so, and simply use TCP unless you detect otherwise. The NFSv4.1 protocol session negotiation supports this, for example. FOr the existing NFS protocols, we rely on explicit instructions from the mount command (-o proto=rdma), and make the connection in RDMA mode. >In fact it may not be such a bad idea to treat an iWARP NIC as two >devices, and register one ethX device (owned exclusively by the normal >stack) and one ibX device (owned mostly by the ofed stack, maybe with >the exception of ARP and such). Would that work with protocols like >NFS/RDMA neogtiation - ie can you negotiate specify a secondary port >_and_ address? NFS/RDMA does, in fact, specify a secondary port. The reason is because NFSv4 (and v2 and v3) doesn't support negotiation, but v4 also does not use the RPC portmapper. When listening on an iWARP NIC therefore, it MUST use a different port from the default 2049, or else MPA would start seeing NFSv4 requests immediately on connect! So, a second well-known port is specified. I'm still waiting for the official assignment of this number from IANA. BTW, for simplicity, we simply use this port on all NFS/RDMA listens, even though it's technically not required on IB because the rdma listening port space is distinct from TCP's. Tom. From sashak at voltaire.com Tue Jun 3 06:23:11 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 3 Jun 2008 16:23:11 +0300 Subject: [ofa-general] [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <48453E26.9030900@voltaire.com> References: <1207861929.7695.165.camel@cardanus.llnl.gov> <4844EAEF.8080605@voltaire.com> <20080603113430.GB4616@sashak.voltaire.com> <48453E26.9030900@voltaire.com> Message-ID: <20080603132311.GE4616@sashak.voltaire.com> On 15:50 Tue 03 Jun , Or Gerlitz wrote: > this patch change-log is empty I see your concern now. I'm using git-am and it is how this patch looks in my local branch: commit 5575734bc33347ed87c625268c22502e66f9d97c Author: Al Chu Date: Thu Apr 10 14:12:09 2008 -0700 implement port_offsetting option This is the primary patch that fiddles with the path recommendation code. A few notes: 1) b/c I want to keep track of how many remote destinations there can be, the 'remote_guids' array now stores all remote destinations, not just the ones we have already forwarded to. 2) b/c I may need to free memory, I now "goto Exit" instead of just calling 'return' many times. 3) Although the option is called 'port_offsetting', I actually "offset" both the remote destination I send to and the port pointing towards that remote destination. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From 57eb4d9bf55fbbbf39dc1c7ddfeeb2cae4776ef0 Mon Sep 17 00:00:00 2001 From: Albert L. Chu Date: Thu, 20 Mar 2008 16:23:13 -0700 Subject: [PATCH] implement port_offsetting Signed-off-by: Albert L. Chu Signed-off-by: Sasha Khapyorsky Sasha From sashak at voltaire.com Tue Jun 3 06:31:33 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 3 Jun 2008 16:31:33 +0300 Subject: [ofa-general] Re: [ewg] OFED meeting sumary on June 02, 08 In-Reply-To: <6C2C79E72C305246B504CBA17B5500C904283063@mtlexch01.mtl.com> References: <6C2C79E72C305246B504CBA17B5500C904282934@mtlexch01.mtl.com> <6C2C79E72C305246B504CBA17B5500C904283063@mtlexch01.mtl.com> Message-ID: <20080603133133.GA29737@sashak.voltaire.com> On 15:25 Tue 03 Jun , Tziporet Koren wrote: > > 4. Backward compatibility of libraries: > Problem: Qlogic found late in OFED 1.3 release process that one of > opensm libraries had > changed the binary API, and this brake some of the > customers' application. > Decisions: > 1. Need to preserve binary API of libraries and not break backward > compatibility (even if it requires extra efforts) It could be impossible for some libraries. > 2. In case binary API must be changed, then both versions will be > included in OFED to prevent broken applications. Any ideas about how it should be done (technically)? Sasha From olaf.kirch at oracle.com Tue Jun 3 06:40:38 2008 From: olaf.kirch at oracle.com (Olaf Kirch) Date: Tue, 3 Jun 2008 15:40:38 +0200 Subject: [ofa-general] [PATCH] rds: use separate ports for TCP and IB In-Reply-To: References: <20080528225549.GC6288@opengridcomputing.com> <200806021025.46789.olaf.kirch@oracle.com> Message-ID: <200806031540.38945.olaf.kirch@oracle.com> On Tuesday 03 June 2008 15:21:35 Talpey, Thomas wrote: > >This may also become a problem for RDS, but in a different sense. > >If you bind an RDS socket to a specific IP address, this also selects > >a transport. If you bind to an IP address owned by ib0, you select > >the IB transport. If you bind to an IP address owned by say eth0, you > >select the TCP transport. > > I think this is a bit of an overloading, and will lead to issues down the road. > IP addresses really aren't about interfaces, they're about hosts. For example, > on TCP (etc), hosts will respond to any of their IP addresses on any of their > interfaces (IP routing often depends on this). I suspect it would be better to > make the transport selection explicit. IPv4/IPv6 selection may fall into the > same issue later. I assume the reason for doing it in this way is "historical", IOW the Oracle IPC layer expects things to work this way :-) But I agree it could be done better. In general, you shouldn't need to bind to any specific transport at all, you can make the decision on the fly by doing an IP flow lookup - and then you can look at the device to decide whether you're on TCP, IB, iWARP, whatever. > >I think for now it's okay to default to iWARP and punt if the remote > >doesn't support it. But in the long run this is something that needs > >to be addressed - either in RDS, or in the way ofed handles iWARP NICs. > > Here I think you've made an even bigger assumption - that not only does > your interface support iWARP, you're assuming the peer's does too! But Yes, I'm aware of that assumption. > Is RDS able to use MPA negotiation to mutually agree to enable RDMA? We could certainly modify RDS over TCP to include some kind of negotiation. RDS/IB also exchanges some information when the connection is established, and it helps in setting up things (eg the decision whether to use congestion control is based on this "negotiation"). Olaf -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play okir at lst.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax From hrosenstock at xsigo.com Tue Jun 3 06:57:05 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 03 Jun 2008 06:57:05 -0700 Subject: [ofa-general] [PATCH] ibutils/ibis/ibis.c: Update for change to osm_mad_pool_init API Message-ID: <1212501425.31387.46.camel@hrosenstock-ws.xsigo.com> ibutils/ibis/ibis.c: Update for change to osm_mad_pool_init API Signed-off-by: Hal Rosenstock diff --git a/ibis/src/ibis.c b/ibis/src/ibis.c index 0684b32..2864f64 100644 --- a/ibis/src/ibis.c +++ b/ibis/src/ibis.c @@ -120,9 +120,7 @@ ibis_init( IbisObj.p_opt = p_opt; - status = osm_mad_pool_init( - &(IbisObj.mad_pool), - &(IbisObj.log) ); + status = osm_mad_pool_init( &(IbisObj.mad_pool) ); if( status != IB_SUCCESS ) { osm_log( &IbisObj.log, OSM_LOG_ERROR, From hrosenstock at xsigo.com Tue Jun 3 07:30:25 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 03 Jun 2008 07:30:25 -0700 Subject: [ofa-general] Re: OSM_DEFAULT_SM_KEY byte order In-Reply-To: <20080602185151.GF13775@sashak.voltaire.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> <20080602182922.GC13775@sashak.voltaire.com> <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> <20080602185151.GF13775@sashak.voltaire.com> Message-ID: <1212503425.31387.54.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-02 at 21:51 +0300, Sasha Khapyorsky wrote: > On 11:41 Mon 02 Jun , Hal Rosenstock wrote: > > On Mon, 2008-06-02 at 21:29 +0300, Sasha Khapyorsky wrote: > > > On 07:06 Mon 02 Jun , Hal Rosenstock wrote: > > > > > > > > This came from informal interop testing a while ago. It wasn't invented > > > > out of thin air. > > > > > > IMHO there are not enough information about the case - finally the value > > > of OSM_DEFAULT_SM_KEY was host byte order (which is obviously wrong), so > > > it doesn't look for me that the case was fully analyzed. > > > > The value was observed on the IB wire with an analyzer. It was > > implemented in OpenSM incorrectly. > > Right, and this leaves questions: for instance if grabbed value was > exactly '0x0100000000000000', I wouldn't see a big chance for such > mistake. Another story would be if grabbed value was something 'non-zero'. > Again, it is unclear for me. I'm not following what you mean by this but I'm rendering this as moot at this point based on the below. > > > > > Our own backward compatibility could be solved by configuring sm key > > > > > (this will work with OpenSM and saquery). > > > > > > > > > > Another opinions? > > > > > > > > I think that third party SMs are a side issue as this is not sanctioned > > > > by IBTA and there is other evidence of a vendor SM using SM key. > > > > > > > > To me, key is back interop with older OpenSMs (at least for x86 as that > > > > is the larger part of the installed base) and this is the aspect which > > > > is sanctioned by IBTA. > > > > > > SM_Key value is configurable in OpenSM so we don't really break > > > interoperability. > > > > Well it does by default and that's the behavior we were discussing. This > > argument cuts the other way too in that it can be "fixed" when needed. > > Yes, it can be "fixed" even now. I'm thinking about permanent solution > and '1' looks like more reasonable default for me. > > > > And in longer term '1' seems as much more "friendly" > > > value than '0x0100000000000000', which entered OpenSM code by mistake > > > > I don't think that friendliness is in the same category of factors. > > I think it is when we are about default values. > > Also, it doesn't need to be typed when default except when "wrong". > > Right, and "wrong" default right now is on x86. OK, I wrote it backwards again. We're not converging in the least on this and I don't have the time right now to look deeper so I throw in the towel at this point but one last question as maybe I don't understand exactly what you are proposing as a change and also a request related to the answer to that question. Question: What is the expected effect on the wire of these changes (separate cases of little and big endian machines if different) ? Request: If your proposal changes default wire behavior, please make sure this is documented (in more than just the git log) so it stands less of a chance of being missed by users who might not follow all these discussions very closely. -- Hal > Sasha From hrosenstock at xsigo.com Tue Jun 3 07:44:03 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 03 Jun 2008 07:44:03 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibutils/README: Minor update Message-ID: <1212504243.31387.62.camel@hrosenstock-ws.xsigo.com> ibutils/README: Minor update Signed-off-by: Hal Rosenstock diff --git a/README b/README index 0f8152e..7454998 100644 --- a/README +++ b/README @@ -1,2 +1 @@ -This is the top level directory for the User Level Utilities -I hope one day it will be moved to the main truck of gen2 +This is the top level directory for additional User Level InfiniBand Management Utilities From rdreier at cisco.com Tue Jun 3 08:10:05 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 03 Jun 2008 08:10:05 -0700 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <71d336490806030524r6aa6fed6rdb166efb11385969@mail.gmail.com> (Ramachandra K.'s message of "Tue, 3 Jun 2008 17:54:19 +0530") References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <71d336490806030524r6aa6fed6rdb166efb11385969@mail.gmail.com> Message-ID: > OK. We will investigate moving the qlgc_vnic driver to configfs. What is the > deadline by which we need to be ready for a merge to 2.6.27 ? By the release of 2.6.26. From tziporet at mellanox.co.il Tue Jun 3 08:35:50 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Tue, 3 Jun 2008 18:35:50 +0300 Subject: [ofa-general] OFED 1.3.1 GA release is available Message-ID: <6C2C79E72C305246B504CBA17B5500C9042D4B51@mtlexch01.mtl.com> I am happy to announce the OFED 1.3.1 GA release. The release can be found under: http://www.openfabrics.org/builds/ofed-1.3.1/release/OFED-1.3.1.tgz It will soon be available on the OpenFabrics download page: http://www.openfabrics.org/downloads.htm This release was produced by a joint effort of all the companies in the EWG group. I wish to thank all who contributed its success. Please report any issues in Bugzilla https://bugs.openfabrics.org/ Tziporet ======================================================================== ======= OFED 1.3.1 GA Release summary: ============================== Release information: -------------------- Linux Operating Systems: - RedHat EL4 up4: 2.6.9-42.ELsmp - RedHat EL4 up5: 2.6.9-55.ELsmp - RedHat EL4 up6: 2.6.9-67.ELsmp - RedHat EL5: 2.6.18-8.el5 - RedHat EL5 up1: 2.6.18-53.el5 - RedHat EL5 up2 beta: 2.6.18-84.el5 * - Fedora C6: 2.6.18-8.fc6 * - SLES10: 2.6.16.21-0.8-smp - SLES10 SP1: 2.6.16.46-0.12-smp - SLES10 SP1 up1: 2.6.16.53-0.16-smp - SLES10 SP2: 2.6.16.60-0.21-smp * - OpenSuSE 10.3: 2.6.22-*-* * - kernel.org: 2.6.23 and 2.6.24 * OSes that are partially tested Systems: * x86_64 * x86 * ia64 * ppc64 Main changes from OFED 1.3 ========================== These are the main changes in OFED 1.3.1 release: 1. New OS support: - SLES 10 SP2 - RHEL 5 up2 2. Install scripts: - There is no default configuration for IPoIB interfaces: One should manually specify the full IP configuration or use the ofed_net.conf file. - A new option was added to the ofed_uninstall.sh script: --unload-modules. This option executes 'openibd stop' before removing the OFED RPMs. 3. Kernel changes: - IPoIB bug fixes - SRP bug fixes - RDS - RDMA API in GA level - Bonding fixes and support for SM failover. - Core: SA module fix 4. Libraries: - uDAPL v1 (1.2.7-1) and v2 (2.0.9-1) - librdmacm 1.0.7 5. Utilities: - opensm version 3.1.11 - mstflint - ibutils 6. Low level drivers: - cxgb3 (Chelsio) - ehca (IBM) - mlx4 (Mellanox) - nes (NetEffect) 7. MPI packages: - MVAPICH 1.0.1 - Open MPI 1.2.6 - MVAPICH2 1.0.3 Note: For details regarding the various fixed bugs, please see the release notes for each component in the docs directory. -------------- next part -------------- An HTML attachment was scrubbed... URL: From torvalds at linux-foundation.org Tue Jun 3 09:26:05 2008 From: torvalds at linux-foundation.org (Linus Torvalds) Date: Tue, 3 Jun 2008 09:26:05 -0700 (PDT) Subject: [ofa-general] Re: [PATCH 001/001] mmu-notifier-core v17 In-Reply-To: <20080509193230.GH7710@duo.random> References: <20080509193230.GH7710@duo.random> Message-ID: On Fri, 9 May 2008, Andrea Arcangeli wrote: > > At least for KVM without this patch it's impossible to swap guests > reliably. And having this feature and removing the page pin allows > several other optimizations that simplify life considerably. Ok, this looks ok as far as I'm concerned. I did not look at any details, so obviously other VM people need to ack the parts they care about, but at least I think this one is fine from a "big picture". I do have some small nits that are just about trivial stuff. > 1) Introduces list_del_init_rcu and documents it (fixes a comment for > list_del_rcu too) I think this should go in separately, and be split up into a patch of its own, just because it's really an independent area. So make it [1/3]. > 2) mm_take_all_locks() to register the mmu notifier when the whole VM > isn't doing anything with "mm". This allows mmu notifier users to > keep track if the VM is in the middle of the > invalidate_range_begin/end critical section with an atomic counter > incraese in range_begin and decreased in range_end. Similarly, even without any users, I think this can be posted as an independent patch, just for setting things up, and to make the whole thing easier to look through and review. So make this [2/3]. But before doing that, can you split up the low-level single-vma anon/file locking/unlocking, please? In other words, your 'mm_take_all_locks()' rigth now looks like it _works_ correctly, but it nests too deeply considering the complexity of it. There's really subtle things going on inside that for-loop, and I think it would be much better to split those low-level locking rules out. IOW, instead of: > +int mm_take_all_locks(struct mm_struct *mm) > +{ > + struct vm_area_struct *vma; > + int ret = -EINTR; > + > + BUG_ON(down_read_trylock(&mm->mmap_sem)); > + > + mutex_lock(&mm_all_locks_mutex); > + > + for (vma = mm->mmap; vma; vma = vma->vm_next) { > + struct file *filp; > + if (signal_pending(current)) > + goto out_unlock; > + if (vma->anon_vma && !test_bit(0, (unsigned long *) > + &vma->anon_vma->head.next)) { > + /* > + * The LSB of head.next can't change from > + * under us because we hold the > + * global_mm_spinlock. > + */ > + spin_lock(&vma->anon_vma->lock); ... ie, can you please make it be for (vma = mm->mmap; vma; vma = vma->vm_next) { if (signal_pending(current)) goto out_unlock; if (vma->anon_vma) vm_lock_anon_vma(vma->anon_vma); if (vma->vm_file && vma->vm_file->f_mapping) vm_lock_mapping(vma->vm_file->f_mapping); } and the same thing for unlocking.. Doesn't that look more obvious and easier to understand from a high-level standpoing (and then the individual locking rules for mappings/anon_vma's will also be more obvious, just because they are separated from the higher-level code). The comments are fine, but even with the comments I'd prefer you to write the code so that you don't need to break up the conditionals over multiple lines etc. Anyway - I didn't look very much at the actual _notifier_ stuff (ie the thing that I think should be [patch 3/3]), so I don't have any real comments about that part - but I don't really care either. Becasue as long as it doesn't mess up the core VM logic, I no longer have any real objections. I'd obviously want to see ack's by people like Andrew, Hugh and Nick, but as far as I am concerned, if you just do the trivial cleanup/split, you can add an "Acked-by: Linus Torvalds " to at least the two first patches of the split-up series. Linus From chu11 at llnl.gov Tue Jun 3 09:48:50 2008 From: chu11 at llnl.gov (Al Chu) Date: Tue, 03 Jun 2008 09:48:50 -0700 Subject: [ofa-general] Re: [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <20080603012002.GN14820@sashak.voltaire.com> References: <1207861929.7695.165.camel@cardanus.llnl.gov> <20080603012002.GN14820@sashak.voltaire.com> Message-ID: <1212511730.5369.58.camel@cardanus.llnl.gov> Hey Sasha, > Maybe to move remote_dests allocation to be here and so minimize needed > 'goto Exit' changes? Good idea. > Should there be sizeof(osm_switch_guid_count_t)? Good catch. At first glance, I was concerned that would have affected the routing. But since I'm memcmp-ing pointers to identical locations, the incorrect 'sizeof' wouldn't have had an affect. Thanks. I'll submit a follow up patch. Al On Tue, 2008-06-03 at 04:20 +0300, Sasha Khapyorsky wrote: > Hi Al, > > Sorry about huge delay. Some questions are below. > > On 14:12 Thu 10 Apr , Al Chu wrote: > > This is the primary patch that fiddles with the path recommendation > > code. A few notes: > > > > 1) b/c I want to keep track of how many remote destinations there can > > be, the 'remote_guids' array now stores all remote destinations, not > > just the ones we have already forwarded to. > > > > 2) b/c I may need to free memory, I now "goto Exit" instead of just > > calling 'return' many times. > > > > 3) Although the option is called 'port_offsetting', I actually "offset" > > both the remote destination I send to and the port pointing towards that > > remote destination. > > > > Al > > > > -- > > Albert Chu > > chu11 at llnl.gov > > 925-422-5311 > > Computer Scientist > > High Performance Systems Division > > Lawrence Livermore National Laboratory > > > From 57eb4d9bf55fbbbf39dc1c7ddfeeb2cae4776ef0 Mon Sep 17 00:00:00 2001 > > From: Albert L. Chu > > Date: Thu, 20 Mar 2008 16:23:13 -0700 > > Subject: [PATCH] implement port_offsetting > > > > > > Signed-off-by: Albert L. Chu > > --- > > opensm/include/opensm/osm_switch.h | 7 +- > > opensm/opensm/osm_dump.c | 3 +- > > opensm/opensm/osm_switch.c | 246 ++++++++++++++++++++++++++++++++---- > > opensm/opensm/osm_ucast_mgr.c | 14 ++- > > 4 files changed, 239 insertions(+), 31 deletions(-) > > > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > > index 2624d5f..45f4718 100644 > > --- a/opensm/include/opensm/osm_switch.h > > +++ b/opensm/include/opensm/osm_switch.h > > @@ -997,7 +997,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN const boolean_t dor, > > IN OUT osm_switch_guid_count_t * remote_guids, > > IN OUT uint16_t * p_num_remote_guids, > > - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used); > > + IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used, > > + IN uint16_t port_offsetting_lids_per_port); > > /* > > * PARAMETERS > > * p_sw > > @@ -1031,6 +1032,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > * [in out] The specific osm_switch_guid_count_t used > > * in switch recommendations. > > * > > +* port_offsetting_lids_per_port > > +* [in] If > 0, indicates lids_per_port to use with > > +* port_offsetting option. > > +* > > * RETURN VALUE > > * Returns the recommended port on which to route this LID. > > * > > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > > index 2bac75a..37842ac 100644 > > --- a/opensm/opensm/osm_dump.c > > +++ b/opensm/opensm/osm_dump.c > > @@ -219,7 +219,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > > /* No LMC Optimization */ > > best_port = osm_switch_recommend_path(p_sw, p_port, > > lid_ho, TRUE, dor, > > - NULL, NULL, NULL); > > + NULL, NULL, > > + NULL, 0); > > fprintf(file, "No %u hop path possible via port %u!", > > best_hops, best_port); > > } > > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > > index f346b25..8e3342b 100644 > > --- a/opensm/opensm/osm_switch.c > > +++ b/opensm/opensm/osm_switch.c > > @@ -55,6 +55,15 @@ > > #include > > #include > > > > +/* Local structs */ > > +struct osm_switch_remote_dest { > > + uint32_t total_paths; > > + uint32_t min_paths; > > + uint8_t ports[256]; /* 256 b/c max_ports is a uint8_t */ > > + unsigned int ports_count; > > + osm_switch_guid_count_t *p_remote_guid; > > +}; > > + > > /********************************************************************** > > **********************************************************************/ > > cl_status_t > > @@ -304,9 +313,32 @@ osm_switch_find_guid_count(IN const osm_switch_t * const p_sw, > > 1); > > } > > > > +/********************************************************************** > > + **********************************************************************/ > > + > > +/* greatest common divisor */ > > +static unsigned int > > +_gcd(unsigned int a, unsigned int b) > > +{ > > + unsigned int t; > > + while (b != 0) { > > + t = b; > > + b = a % b; > > + a = t; > > + } > > + return a; > > +} > > + > > +/* least common multiple */ > > +static unsigned int > > +_lcm(unsigned int a, unsigned int b) > > +{ > > + return ((a*b) / _gcd(a,b)); > > +} > > > > /********************************************************************** > > **********************************************************************/ > > + > > uint8_t > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN osm_port_t * p_port, > > @@ -315,7 +347,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN const boolean_t dor, > > IN OUT osm_switch_guid_count_t * remote_guids, > > IN OUT uint16_t * p_num_remote_guids, > > - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used) > > + IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used, > > + IN uint16_t port_offsetting_lids_per_port) > > { > > /* > > We support an enhanced LMC aware routing mode: > > @@ -356,6 +389,20 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > osm_node_t *p_rem_node; > > osm_node_t *p_rem_node_first = NULL; > > osm_switch_guid_count_t *p_remote_guid = NULL; > > + /* > > + These vars track information for port offsetting. > > + */ > > + boolean_t port_offsetting = remote_guids && p_num_remote_guids > > + && p_remote_guid_count_used && port_offsetting_lids_per_port; > > + struct osm_switch_remote_dest * remote_dests = NULL; > > + struct osm_switch_remote_dest * p_remote_dest = NULL; > > + uint32_t num_remote_dests = 0; > > + uint32_t total_paths_count = 0; > > + uint32_t num_potential_ports = 0; > > + uint32_t lcm = 0; > > + uint32_t indx = 0; > > + boolean_t dest_found = FALSE; > > + unsigned int i; > > > > CL_ASSERT(lid_ho > 0); > > > > @@ -378,9 +425,22 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > > num_ports = p_sw->num_ports; > > > > + if (port_offsetting) { > > + remote_dests = malloc(sizeof(struct osm_switch_remote_dest) * num_ports); > > + if (remote_dests == NULL) { > > + osm_log(p_sw->p_log, OSM_LOG_ERROR, > > + "osm_switch_recommend_path: " > > + "Cannot allocate array. Insufficient memory: " > > + "Disabling port_offsetting\n"); > > + port_offsetting = 0; > > + } > > + } > > + > > least_hops = osm_switch_get_least_hops(p_sw, base_lid); > > - if (least_hops == OSM_NO_PATH) > > - return (OSM_NO_PATH); > > + if (least_hops == OSM_NO_PATH) { > > + best_port = OSM_NO_PATH; > > + goto Exit; > > + } > > > > /* > > First, inquire with the forwarding table for an existing > > @@ -417,8 +477,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > in the forwarding tables that he wants to be overridden by the > > minimum hop function. > > */ > > - if (hops == least_hops) > > - return (port_num); > > + if (hops == least_hops) { > > + best_port = port_num; > > + goto Exit; > > + } > > } > > } > > } > > Maybe to move remote_dests allocation to be here and so minimize needed > 'goto Exit' changes? > > > @@ -475,7 +537,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > port_num); > > > > /* If not update the least hops for this case */ > > - if (!p_remote_guid) { > > + if (!p_remote_guid > > + || !p_remote_guid->forwarded_to) { > > if (check_count < least_paths_other_sys) { > > least_paths_other_sys = check_count; > > best_port_other_sys = port_num; > > @@ -489,7 +552,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > port_num); > > > > /* If not update the least hops for this case */ > > - if (!p_remote_guid > > + if ((!p_remote_guid > > + || !p_remote_guid->forwarded_to) > > && check_count < least_paths_other_nodes) { > > least_paths_other_nodes = check_count; > > best_port_other_node = port_num; > > @@ -498,6 +562,50 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > /* else prior sys and node guid already used */ > > > > } /* same sys found */ > > + > > + /* Store the new sys/node guid that we haven't seen yet */ > > + if (!p_remote_guid) { > > + p_rem_physp = osm_physp_get_remote(p_physp); > > + p_rem_node = osm_physp_get_node_ptr(p_rem_physp); > > + memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), > > + &(p_rem_node->node_info.sys_guid), > > + sizeof(uint64_t)); > > + memcpy(&(remote_guids[*p_num_remote_guids].node_guid), > > + &(p_rem_node->node_info.node_guid), > > + sizeof(uint64_t)); > > + remote_guids[*p_num_remote_guids].forwarded_to = 0; > > + p_remote_guid = &remote_guids[*p_num_remote_guids]; > > + (*p_num_remote_guids)++; > > + } > > + } > > + > > + if (port_offsetting) { > > + /* Keep track of the destinations we've seen so far */ > > + p_remote_dest = NULL; > > + for (i = 0; i < num_remote_dests; i++) { > > + if (!memcmp(p_remote_guid, > > + (&remote_dests[i])->p_remote_guid, > > + sizeof(struct osm_switch_remote_dest))) { > > Should there be sizeof(osm_switch_guid_count_t)? > > Sasha > > > + p_remote_dest = &remote_dests[i]; > > + break; > > + } > > + } > > + if (!p_remote_dest) { > > + p_remote_dest = &remote_dests[num_remote_dests]; > > + p_remote_dest->p_remote_guid = p_remote_guid; > > + p_remote_dest->total_paths = 0; > > + p_remote_dest->min_paths = 0xFFFFFFFF; > > + p_remote_dest->ports_count = 0; > > + num_remote_dests++; > > + } > > + p_remote_dest->total_paths += check_count; > > + if (check_count < p_remote_dest->min_paths) > > + p_remote_dest->min_paths = check_count; > > + p_remote_dest->ports[p_remote_dest->ports_count] = port_num; > > + p_remote_dest->ports_count++; > > + > > + total_paths_count += check_count; > > + num_potential_ports++; > > } > > > > /* routing for LMC mode */ > > @@ -523,21 +631,23 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > best_port = port_num; > > least_paths = check_count; > > if (routing_for_lmc > > - && p_remote_guid > > + && p_remote_guid->forwarded_to > > && p_remote_guid->forwarded_to < least_forwarded_to) > > least_forwarded_to = p_remote_guid->forwarded_to; > > } > > else if (routing_for_lmc > > - && p_remote_guid > > && check_count == least_paths > > + && p_remote_guid->forwarded_to > > && p_remote_guid->forwarded_to < least_forwarded_to) { > > least_forwarded_to = p_remote_guid->forwarded_to; > > best_port = port_num; > > } > > } > > > > - if (port_found == FALSE) > > - return (OSM_NO_PATH); > > + if (port_found == FALSE) { > > + best_port = OSM_NO_PATH; > > + goto Exit; > > + } > > > > /* > > if we are in enhanced routing mode and the best port is not > > @@ -555,24 +665,110 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > remote_guids, > > p_num_remote_guids, > > best_port); > > + /* Must be stored */ > > + CL_ASSERT(p_remote_guid); > > + *p_remote_guid_count_used = p_remote_guid; > > + } > > > > - if (!p_remote_guid) { > > - /* track the remote node and system of the port used. */ > > - p_physp = osm_node_get_physp_ptr(p_sw->p_node, best_port); > > - p_rem_physp = osm_physp_get_remote(p_physp); > > - p_rem_node = osm_physp_get_node_ptr(p_rem_physp); > > - memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), > > - &(p_rem_node->node_info.sys_guid), > > - sizeof(uint64_t)); > > - memcpy(&(remote_guids[*p_num_remote_guids].node_guid), > > - &(p_rem_node->node_info.node_guid), > > - sizeof(uint64_t)); > > - remote_guids[*p_num_remote_guids].forwarded_to = 0; > > - (*p_num_remote_guids)++; > > + /* > > + * As an example of what we're trying to do with port > > + * offsetting, assume LMC = 2 and we are trying to route > > + * the lids of 4 ports. The lids of these 4 ports are: > > + * > > + * (1,2,3,4) > > + * (5,6,7,8) > > + * (9,10,11,12) > > + * (13,14,15,16) > > + * > > + * Suppose forwarding to all these lids goes through > > + * 4 specific switch ports. If we just cycle through > > + * ports and lids in a normal iterative fashion, we would > > + * normally forward out ports in this manner. > > + * > > + * switch port 1: 1, 5, 9, 13 > > + * switch port 2: 2, 6, 10, 14 > > + * switch port 3: 3, 7, 11, 15 > > + * switch port 4: 4, 8, 12, 1 > > + * > > + * Note that the base lid of each port (lids 1, 5, 9, 13) > > + * are all routed out of switch port 1. Thus, if the user > > + * only uses the base lid of each port, they will get pretty > > + * bad performance. We will try to get this layout instead. > > + * > > + * switch port 1: 1, 8, 11, 14 > > + * switch port 2: 2, 5, 12, 15 > > + * switch port 3: 3, 6, 9, 16 > > + * switch port 4: 4, 7, 10, 13 > > + * > > + * where switch ports are distributed in a more even manner. > > + * The base lid of each port is now distributed evenly > > + * across all 4 switch ports. The remaining lids are still > > + * distributed evenly across all the remaining switch ports. > > + * > > + * In order to accomplish this, we (effectively) will iterate > > + * through all ports like before, but instead of iterating from > > + * 0 to N-1 all the time, we will select the starting index > > + * based on the number of paths we have routed thus far. > > + */ > > + > > + /* We will not do port offsetting if num_potential_ports == 1 > > + * b/c there is no offsetting that can be done. > > + */ > > + if (port_offsetting > > + && best_port > > + && num_potential_ports > 1) { > > + /* Select which destination we want to forward to with our > > + * offsetting loop. > > + */ > > + lcm = _lcm(port_offsetting_lids_per_port, > > + num_remote_dests); > > + indx = (total_paths_count / lcm) % num_remote_dests; > > + for (i = 0; i < num_remote_dests; i++) { > > + p_remote_dest = &remote_dests[indx]; > > + p_remote_guid = p_remote_dest->p_remote_guid; > > + > > + if (p_remote_guid->forwarded_to == least_forwarded_to > > + && p_remote_dest->min_paths == least_paths) { > > + dest_found = TRUE; > > + break; > > + } > > + > > + indx++; > > + if (indx >= num_remote_dests) > > + indx = 0; > > + } > > + > > + /* Then we "offset" within the potential ports we could > > + * forward out of for this specific destination. > > + */ > > + if (dest_found && p_remote_dest) { > > + lcm = _lcm(port_offsetting_lids_per_port, > > + p_remote_dest->ports_count); > > + indx = (p_remote_dest->total_paths / lcm) % p_remote_dest->ports_count; > > + for (i = 0; i < p_remote_dest->ports_count; i++) { > > + port_num = p_remote_dest->ports[indx]; > > + check_count = > > + osm_port_prof_path_count_get(&p_sw->p_prof[port_num]); > > + if (check_count == least_paths) { > > + if (best_port != port_num) > > + osm_log(p_sw->p_log, OSM_LOG_DEBUG, > > + "osm_switch_recommend_path: " > > + "best port offsetted: %d to %d\n", > > + best_port, port_num); > > + best_port = port_num; > > + *p_remote_guid_count_used = p_remote_dest->p_remote_guid; > > + break; > > + } > > + indx++; > > + if (indx >= p_remote_dest->ports_count) > > + indx = 0; > > + } > > } > > - *p_remote_guid_count_used = p_remote_guid; > > } > > > > +Exit: > > + if (remote_dests) > > + free(remote_dests); > > return (best_port); > > } > > > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > > index 938db84..501c2c7 100644 > > --- a/opensm/opensm/osm_ucast_mgr.c > > +++ b/opensm/opensm/osm_ucast_mgr.c > > @@ -212,18 +212,23 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > osm_switch_guid_count_t *remote_guids = NULL; > > uint16_t num_used_guids = 0; > > osm_switch_guid_count_t *p_remote_guid_used = NULL; > > + uint16_t port_offsetting_lids_per_port = 0; > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > if (lids_per_port > 1) { > > - remote_guids = malloc(sizeof(osm_switch_guid_count_t) * lids_per_port); > > + uint8_t num_ports = p_sw->num_ports; > > + remote_guids = malloc(sizeof(osm_switch_guid_count_t) * num_ports); > > if (remote_guids == NULL) { > > osm_log(p_mgr->p_log, OSM_LOG_ERROR, > > "__osm_ucast_mgr_process_port: ERR 3A09: " > > "Cannot allocate array. Insufficient memory\n"); > > goto Exit; > > } > > - memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * lids_per_port); > > + memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * num_ports); > > + > > + if (p_mgr->p_subn->opt.port_offsetting) > > + port_offsetting_lids_per_port = lids_per_port; > > } > > > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > > @@ -270,14 +275,15 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > p_mgr->is_dor, > > remote_guids, > > &num_used_guids, > > - &p_remote_guid_used); > > + &p_remote_guid_used, > > + port_offsetting_lids_per_port); > > } > > else > > port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > p_mgr->p_subn-> > > ignore_existing_lfts, > > p_mgr->is_dor, > > - NULL, NULL, NULL); > > + NULL, NULL, NULL, 0); > > > > /* > > There might be no path to the target > > -- > > 1.5.1 > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From andrea at qumranet.com Tue Jun 3 10:35:02 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Tue, 3 Jun 2008 19:35:02 +0200 Subject: [ofa-general] Re: [PATCH 001/001] mmu-notifier-core v17 In-Reply-To: References: <20080509193230.GH7710@duo.random> Message-ID: <20080603173502.GR8158@duo.random> On Tue, Jun 03, 2008 at 09:26:05AM -0700, Linus Torvalds wrote: > I'd obviously want to see ack's by people like Andrew, Hugh and Nick, but > as far as I am concerned, if you just do the trivial cleanup/split, you I'll do all cleanups/splits very shortly and submit a v18 with 3 patches, all good, thanks a lot! From shemminger at vyatta.com Tue Jun 3 10:51:24 2008 From: shemminger at vyatta.com (Stephen Hemminger) Date: Tue, 3 Jun 2008 10:51:24 -0700 Subject: [ofa-general] Re: [PATCH v3 01/13] QLogic VNIC: Driver - netdev implementation In-Reply-To: <71d336490806030514k580418e3n23790eb793e99381@mail.gmail.com> References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095423.9943.77528.stgit@localhost.localdomain> <20080529102752.584147ee@extreme> <71d336490806030514k580418e3n23790eb793e99381@mail.gmail.com> Message-ID: <20080603105124.2f663297@extreme> On Tue, 3 Jun 2008 17:44:23 +0530 "Ramachandra K" wrote: > On Thu, May 29, 2008 at 10:57 PM, Stephen Hemminger > wrote: > > On Thu, 29 May 2008 15:24:23 +0530 > > Ramachandra K wrote: > > > > > >> +module_param(vnic_debug, uint, 0444); > >> +MODULE_PARM_DESC(vnic_debug, "Enable debug tracing if > 0"); > > > > maybe migrate this to ethtool msg_level? > > > > We investigated this and found that we cannot map all of our Infiniband > level communication errors etc to the ethtool message levels. One option could > be to add support for Ethtool message levels in addition to the vnic_debug > controlled messages. The Ethtool support would make this driver conform > to ethtool similar to other Ethernet drivers and the vnic_debug messages will > help in debugging the Infiniband related errors. > > Is ethtool support mandatory for a merge to 2.6.27 or can we consider > it as a future > enhancement while we first focus on moving this driver to use CONFIGFS ? > > Regards, > Ram No problem, just a suggestion of how to make it easier to manage. From amar.mudrankit at qlogic.com Tue Jun 3 11:25:00 2008 From: amar.mudrankit at qlogic.com (Amar Mudrankit) Date: Tue, 3 Jun 2008 23:55:00 +0530 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <20080529174805.GA10903@kroah.com> References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> Message-ID: On Thu, May 29, 2008 at 11:18 PM, Greg KH wrote: > On Thu, May 29, 2008 at 10:30:03AM -0700, Stephen Hemminger wrote: > > On Thu, 29 May 2008 15:27:54 +0530 > > Ramachandra K wrote: > > > > > From: Amar Mudrankit > > > > > > The sysfs interface for the QLogic VNIC driver is implemented through > > > this patch. > > > > > > Signed-off-by: Amar Mudrankit > > > Signed-off-by: Ramachandra K > > > Signed-off-by: Poornima Kamath > > > --- > > > > > > drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c | 1133 > +++++++++++++++++++++++++++ > > > drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h | 51 + > > > 2 files changed, 1184 insertions(+), 0 deletions(-) > > > create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c > > > create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h > > > > > > diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c > b/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c > > > new file mode 100644 > > > index 0000000..40b3c77 > > > --- /dev/null > > > +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c > > > @@ -0,0 +1,1133 @@ > > > +/* > > > + * Copyright (c) 2006 QLogic, Inc. All rights reserved. > > > + * > > > + * This software is available to you under a choice of one of two > > > + * licenses. You may choose to be licensed under the terms of the GNU > > > + * General Public License (GPL) Version 2, available from the file > > > + * COPYING in the main directory of this source tree, or the > > > + * OpenIB.org BSD license below: > > > + * > > > + * Redistribution and use in source and binary forms, with or > > > + * without modification, are permitted provided that the following > > > + * conditions are met: > > > + * > > > + * - Redistributions of source code must retain the above > > > + * copyright notice, this list of conditions and the following > > > + * disclaimer. > > > + * > > > + * - Redistributions in binary form must reproduce the above > > > + * copyright notice, this list of conditions and the following > > > + * disclaimer in the documentation and/or other materials > > > + * provided with the distribution. > > > + * > > > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > > > + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > > > + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > > > + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > > > + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > > > + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > > > + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > > > + * SOFTWARE. > > > + */ > > > + > > > +#include > > > +#include > > > +#include > > > + > > > +#include "vnic_util.h" > > > +#include "vnic_config.h" > > > +#include "vnic_ib.h" > > > +#include "vnic_viport.h" > > > +#include "vnic_main.h" > > > +#include "vnic_stats.h" > > > + > > > +/* > > > + * target eiocs are added by writing > > > + * > > > + * ioc_guid=,dgid= GID>,pkey=,name= > > > + * to the create_primary sysfs attribute. > > > + */ > > > +enum { > > > + VNIC_OPT_ERR = 0, > > > + VNIC_OPT_IOC_GUID = 1 << 0, > > > + VNIC_OPT_DGID = 1 << 1, > > > + VNIC_OPT_PKEY = 1 << 2, > > > + VNIC_OPT_NAME = 1 << 3, > > > + VNIC_OPT_INSTANCE = 1 << 4, > > > + VNIC_OPT_RXCSUM = 1 << 5, > > > + VNIC_OPT_TXCSUM = 1 << 6, > > > + VNIC_OPT_HEARTBEAT = 1 << 7, > > > + VNIC_OPT_IOC_STRING = 1 << 8, > > > + VNIC_OPT_IB_MULTICAST = 1 << 9, > > > + VNIC_OPT_ALL = (VNIC_OPT_IOC_GUID | > > > + VNIC_OPT_DGID | VNIC_OPT_NAME | VNIC_OPT_PKEY), > > > +}; > > > + > > > +static match_table_t vnic_opt_tokens = { > > > + {VNIC_OPT_IOC_GUID, "ioc_guid=%s"}, > > > + {VNIC_OPT_DGID, "dgid=%s"}, > > > + {VNIC_OPT_PKEY, "pkey=%x"}, > > > + {VNIC_OPT_NAME, "name=%s"}, > > > + {VNIC_OPT_INSTANCE, "instance=%d"}, > > > + {VNIC_OPT_RXCSUM, "rx_csum=%s"}, > > > + {VNIC_OPT_TXCSUM, "tx_csum=%s"}, > > > + {VNIC_OPT_HEARTBEAT, "heartbeat=%d"}, > > > + {VNIC_OPT_IOC_STRING, "ioc_string=\"%s"}, > > > + {VNIC_OPT_IB_MULTICAST, "ib_multicast=%s"}, > > > + {VNIC_OPT_ERR, NULL} > > > +}; > > > > > > > No sysfs is supposed to be one value per file use separate attributes > > for each one. This also eliminates the parsing code. > > And yes, multiple values per sysfs file are not allowed, sorry, please > change this. If you need to configure your device through an interface > like this, consider using configfs instead, that is what it is there > for. Documentation/filesystems/configfs/configfs.txt <> [Using configfs] Like sysfs, attributes should be ASCII text files, preferably with only one value per file. The same efficiency caveats from sysfs apply. Don't mix more than one attribute in one attribute file. It looks like configfs also does not allow multiple attributes per file. So, in that case, shall we modify our sysfs code to support one value per sysfs file? With this approach, the impact on user space applications that we have which depend on the qlgc_vnic driver sysfs interface will be less. Thanks and Regards, Amar > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at kroah.com Tue Jun 3 11:30:39 2008 From: greg at kroah.com (Greg KH) Date: Tue, 3 Jun 2008 11:30:39 -0700 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> Message-ID: <20080603183039.GA30454@kroah.com> On Tue, Jun 03, 2008 at 11:55:00PM +0530, Amar Mudrankit wrote: > Documentation/filesystems/configfs/configfs.txt <> > > [Using configfs] > > Like sysfs, attributes should be ASCII text files, preferably > with only one value per file. The same efficiency caveats from sysfs > apply. Don't mix more than one attribute in one attribute file. > > It looks like configfs also does not allow multiple attributes per file. But you have attribute/value pairs, right? Just create multiple attribute files then in configfs for them. > So, in that case, shall we modify our sysfs code to support one value per > sysfs file? With this approach, the impact on user space applications that > we have which depend on the qlgc_vnic driver sysfs interface will be less. If you want to, but as this looks like it is used for configuration, I suggest using configfs instead. thanks, greg k-h From dillowda at ornl.gov Tue Jun 3 11:45:03 2008 From: dillowda at ornl.gov (David Dillow) Date: Tue, 03 Jun 2008 14:45:03 -0400 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> Message-ID: <1212518703.2217.22.camel@lap75545.ornl.gov> On Tue, 2008-06-03 at 23:55 +0530, Amar Mudrankit wrote: > On Thu, May 29, 2008 at 11:18 PM, Greg KH wrote: > > On Thu, May 29, 2008 at 10:30:03AM -0700, Stephen Hemminger wrote: > > > No sysfs is supposed to be one value per file use separate > > > attributes for each one. This also eliminates the parsing code. > > > And yes, multiple values per sysfs file are not allowed, sorry, > > please change this. If you need to configure your device through an > > interface like this, consider using configfs instead, that is what > > it is there for. > > Documentation/filesystems/configfs/configfs.txt <> > > [Using configfs] > > Like sysfs, attributes should be ASCII text files, preferably > with only one value per file. The same efficiency caveats from sysfs > apply. Don't mix more than one attribute in one attribute file. > > > It looks like configfs also does not allow multiple attributes per > file. So, in that case, shall we modify our sysfs code to support one > value per sysfs file? With this approach, the impact on user space > applications that we have which depend on the qlgc_vnic driver sysfs > interface will be less. The idea would be to have a separate file per attribute, ie ioc_guid in a file, dgid in a file, pkey in a file, and so on. Userspace would make a directory for the new connection in /configfs/qlogic-device (or what ever you call it), and configfs will populate it with default values for those attributes. Or so the theory goes. Unfortunately, you need all that information before you can create the connection. The configfs guys have thought about that, but not implemented yet: > [Committable Items] > NOTE: Committable items are currently unimplemented. > > Some config_items cannot have a valid initial state. That is, no > default values can be specified for the item's attributes such that > the item can do its work. Userspace must configure one or more > attributes, after which the subsystem can start whatever entity this > item represents. > [snip] > Any group that provides the ct_group_ops->commit_item() method has > committable items. When this group appears in configfs, mkdir(2) will > not work directly in the group. Instead, the group will have two > subdirectories: "live" and "pending". The "live" directory does not > support mkdir(2) or rmdir(2) either. It only allows rename(2). The > "pending" directory does allow mkdir(2) and rmdir(2). An item is > created in the "pending" directory. Its attributes can be modified at > will. Userspace commits the item by renaming it into the "live" > directory. At this point, the subsystem receives the ->commit_item() > callback. If all required attributes are filled to satisfaction, the > method returns zero and the item is moved to the "live" directory. It looks like the design for what you'd need is there, but no code -- ag grep for 'commit' in the fs/configfs directory doesn't give me anything, though I just did a cursory glance when it was suggested that SRP use this. So, I don't think configfs is ready for your use case just yet. Perhaps you can contact the configfs guys and see where they are at on this. I suppose you could also implement a special "activate" attribute that tells your code that all the attributes are set when it gets written, but that seems counter to the intended usage, and there are some possible obj lifetime rules to deal with then. The commitable item concept seems like a much better fit. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From or.gerlitz at gmail.com Tue Jun 3 11:59:50 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Tue, 3 Jun 2008 21:59:50 +0300 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: References: Message-ID: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> On 6/3/08, Roland Dreier wrote: > > given that the real change is to multicast attach and is not a property > of QPs, wouldn't it be simpler to simply add a flags argument to > ib_attach_mcast() and use a flag bit to request that messages not be > looped back per mcast attach? > The approach here was that (A) from application point of view, setting this property per QP and not per QP per attach to group makes enough sense, specifically, IPoIB wants this flag to be set for all its attaches since the network stack does loopback in memory. and (B) moving forward with the patches, we'd like to export this feature to user space, and from the previous threads on "special QP features" (eg XRC, Low Latency) it sounds like adding some sort of creation flags (even in the price of new verb such that the existing ABI for qp create still works) might be the way to go. Or -------------- next part -------------- An HTML attachment was scrubbed... URL: From or.gerlitz at gmail.com Tue Jun 3 12:02:48 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Tue, 3 Jun 2008 22:02:48 +0300 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: References: Message-ID: <15ddcffd0806031202l248b4fdbgcd6cd98f5c22ce65@mail.gmail.com> On 6/2/08, Roland Dreier wrote: > > Do we really need to burn a capability bit for this? Would it cause any > problems to make IB_QP_BLOCK_LOOPBACK a hint that the underlying HCA may > or may not support? > For IPoIB a hint might be enough, since the code to handle such packets is there anyway and what this code does is simply to drop them. For other applications, I think that their design might rely deeper on whether this functionality is provided by their QP or not. Or -------------- next part -------------- An HTML attachment was scrubbed... URL: From gatfdg at bradford-medical.com Tue Jun 3 12:25:41 2008 From: gatfdg at bradford-medical.com (Norman Moseley) Date: Tue, 3 Jun 2008 21:25:41 +0200 Subject: [ofa-general] Blue sexy pill - $0.{_2SYMBCHAR} Message-ID: <01c8c5c0$5fb2b880$7e7e6255@gatfdg> Blue sexy pill - $0.{_2SYMBCHAR}Visit our shop -------------- next part -------------- An HTML attachment was scrubbed... URL: From pw at osc.edu Tue Jun 3 13:44:07 2008 From: pw at osc.edu (Pete Wyckoff) Date: Tue, 3 Jun 2008 16:44:07 -0400 Subject: [ofa-general] Re: mthca MR attrs userspace change In-Reply-To: References: <20080527180004.GA15444@osc.edu> Message-ID: <20080603204407.GA7840@osc.edu> rdreier at cisco.com wrote on Tue, 27 May 2008 15:53 -0700: > libmthca-1.0.5 is now in Fedora 9 proposed updates -- > http://koji.fedoraproject.org/koji/buildinfo?buildID=50682 > > I'm not sure exactly how it makes it into real F-9. Nor am I. It never showed up in updates-testing or the real updates, a week after you built it. I can't find any developer docs on how this is supposed to work. Maybe someone on the list knows what button to push? -- Pete From chu11 at llnl.gov Tue Jun 3 15:46:03 2008 From: chu11 at llnl.gov (Al Chu) Date: Tue, 03 Jun 2008 15:46:03 -0700 Subject: [ofa-general] Re: [OpenSM] [PATCH 3/3] implement port_offsetting option In-Reply-To: <20080603012002.GN14820@sashak.voltaire.com> References: <1207861929.7695.165.camel@cardanus.llnl.gov> <20080603012002.GN14820@sashak.voltaire.com> Message-ID: <1212533163.5369.72.camel@cardanus.llnl.gov> Hey Sasha, Here are two patches to fix the minor comments you made earlier. Sanity tested on my small cluster, but did not have the chance to test on a big one. However, these patches are tiny, so I doubt there'd be an issue. Al On Tue, 2008-06-03 at 04:20 +0300, Sasha Khapyorsky wrote: > Hi Al, > > Sorry about huge delay. Some questions are below. > > On 14:12 Thu 10 Apr , Al Chu wrote: > > This is the primary patch that fiddles with the path recommendation > > code. A few notes: > > > > 1) b/c I want to keep track of how many remote destinations there can > > be, the 'remote_guids' array now stores all remote destinations, not > > just the ones we have already forwarded to. > > > > 2) b/c I may need to free memory, I now "goto Exit" instead of just > > calling 'return' many times. > > > > 3) Although the option is called 'port_offsetting', I actually "offset" > > both the remote destination I send to and the port pointing towards that > > remote destination. > > > > Al > > > > -- > > Albert Chu > > chu11 at llnl.gov > > 925-422-5311 > > Computer Scientist > > High Performance Systems Division > > Lawrence Livermore National Laboratory > > > From 57eb4d9bf55fbbbf39dc1c7ddfeeb2cae4776ef0 Mon Sep 17 00:00:00 2001 > > From: Albert L. Chu > > Date: Thu, 20 Mar 2008 16:23:13 -0700 > > Subject: [PATCH] implement port_offsetting > > > > > > Signed-off-by: Albert L. Chu > > --- > > opensm/include/opensm/osm_switch.h | 7 +- > > opensm/opensm/osm_dump.c | 3 +- > > opensm/opensm/osm_switch.c | 246 ++++++++++++++++++++++++++++++++---- > > opensm/opensm/osm_ucast_mgr.c | 14 ++- > > 4 files changed, 239 insertions(+), 31 deletions(-) > > > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > > index 2624d5f..45f4718 100644 > > --- a/opensm/include/opensm/osm_switch.h > > +++ b/opensm/include/opensm/osm_switch.h > > @@ -997,7 +997,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN const boolean_t dor, > > IN OUT osm_switch_guid_count_t * remote_guids, > > IN OUT uint16_t * p_num_remote_guids, > > - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used); > > + IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used, > > + IN uint16_t port_offsetting_lids_per_port); > > /* > > * PARAMETERS > > * p_sw > > @@ -1031,6 +1032,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > * [in out] The specific osm_switch_guid_count_t used > > * in switch recommendations. > > * > > +* port_offsetting_lids_per_port > > +* [in] If > 0, indicates lids_per_port to use with > > +* port_offsetting option. > > +* > > * RETURN VALUE > > * Returns the recommended port on which to route this LID. > > * > > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > > index 2bac75a..37842ac 100644 > > --- a/opensm/opensm/osm_dump.c > > +++ b/opensm/opensm/osm_dump.c > > @@ -219,7 +219,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > > /* No LMC Optimization */ > > best_port = osm_switch_recommend_path(p_sw, p_port, > > lid_ho, TRUE, dor, > > - NULL, NULL, NULL); > > + NULL, NULL, > > + NULL, 0); > > fprintf(file, "No %u hop path possible via port %u!", > > best_hops, best_port); > > } > > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > > index f346b25..8e3342b 100644 > > --- a/opensm/opensm/osm_switch.c > > +++ b/opensm/opensm/osm_switch.c > > @@ -55,6 +55,15 @@ > > #include > > #include > > > > +/* Local structs */ > > +struct osm_switch_remote_dest { > > + uint32_t total_paths; > > + uint32_t min_paths; > > + uint8_t ports[256]; /* 256 b/c max_ports is a uint8_t */ > > + unsigned int ports_count; > > + osm_switch_guid_count_t *p_remote_guid; > > +}; > > + > > /********************************************************************** > > **********************************************************************/ > > cl_status_t > > @@ -304,9 +313,32 @@ osm_switch_find_guid_count(IN const osm_switch_t * const p_sw, > > 1); > > } > > > > +/********************************************************************** > > + **********************************************************************/ > > + > > +/* greatest common divisor */ > > +static unsigned int > > +_gcd(unsigned int a, unsigned int b) > > +{ > > + unsigned int t; > > + while (b != 0) { > > + t = b; > > + b = a % b; > > + a = t; > > + } > > + return a; > > +} > > + > > +/* least common multiple */ > > +static unsigned int > > +_lcm(unsigned int a, unsigned int b) > > +{ > > + return ((a*b) / _gcd(a,b)); > > +} > > > > /********************************************************************** > > **********************************************************************/ > > + > > uint8_t > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN osm_port_t * p_port, > > @@ -315,7 +347,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN const boolean_t dor, > > IN OUT osm_switch_guid_count_t * remote_guids, > > IN OUT uint16_t * p_num_remote_guids, > > - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used) > > + IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used, > > + IN uint16_t port_offsetting_lids_per_port) > > { > > /* > > We support an enhanced LMC aware routing mode: > > @@ -356,6 +389,20 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > osm_node_t *p_rem_node; > > osm_node_t *p_rem_node_first = NULL; > > osm_switch_guid_count_t *p_remote_guid = NULL; > > + /* > > + These vars track information for port offsetting. > > + */ > > + boolean_t port_offsetting = remote_guids && p_num_remote_guids > > + && p_remote_guid_count_used && port_offsetting_lids_per_port; > > + struct osm_switch_remote_dest * remote_dests = NULL; > > + struct osm_switch_remote_dest * p_remote_dest = NULL; > > + uint32_t num_remote_dests = 0; > > + uint32_t total_paths_count = 0; > > + uint32_t num_potential_ports = 0; > > + uint32_t lcm = 0; > > + uint32_t indx = 0; > > + boolean_t dest_found = FALSE; > > + unsigned int i; > > > > CL_ASSERT(lid_ho > 0); > > > > @@ -378,9 +425,22 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > > num_ports = p_sw->num_ports; > > > > + if (port_offsetting) { > > + remote_dests = malloc(sizeof(struct osm_switch_remote_dest) * num_ports); > > + if (remote_dests == NULL) { > > + osm_log(p_sw->p_log, OSM_LOG_ERROR, > > + "osm_switch_recommend_path: " > > + "Cannot allocate array. Insufficient memory: " > > + "Disabling port_offsetting\n"); > > + port_offsetting = 0; > > + } > > + } > > + > > least_hops = osm_switch_get_least_hops(p_sw, base_lid); > > - if (least_hops == OSM_NO_PATH) > > - return (OSM_NO_PATH); > > + if (least_hops == OSM_NO_PATH) { > > + best_port = OSM_NO_PATH; > > + goto Exit; > > + } > > > > /* > > First, inquire with the forwarding table for an existing > > @@ -417,8 +477,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > in the forwarding tables that he wants to be overridden by the > > minimum hop function. > > */ > > - if (hops == least_hops) > > - return (port_num); > > + if (hops == least_hops) { > > + best_port = port_num; > > + goto Exit; > > + } > > } > > } > > } > > Maybe to move remote_dests allocation to be here and so minimize needed > 'goto Exit' changes? > > > @@ -475,7 +537,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > port_num); > > > > /* If not update the least hops for this case */ > > - if (!p_remote_guid) { > > + if (!p_remote_guid > > + || !p_remote_guid->forwarded_to) { > > if (check_count < least_paths_other_sys) { > > least_paths_other_sys = check_count; > > best_port_other_sys = port_num; > > @@ -489,7 +552,8 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > port_num); > > > > /* If not update the least hops for this case */ > > - if (!p_remote_guid > > + if ((!p_remote_guid > > + || !p_remote_guid->forwarded_to) > > && check_count < least_paths_other_nodes) { > > least_paths_other_nodes = check_count; > > best_port_other_node = port_num; > > @@ -498,6 +562,50 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > /* else prior sys and node guid already used */ > > > > } /* same sys found */ > > + > > + /* Store the new sys/node guid that we haven't seen yet */ > > + if (!p_remote_guid) { > > + p_rem_physp = osm_physp_get_remote(p_physp); > > + p_rem_node = osm_physp_get_node_ptr(p_rem_physp); > > + memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), > > + &(p_rem_node->node_info.sys_guid), > > + sizeof(uint64_t)); > > + memcpy(&(remote_guids[*p_num_remote_guids].node_guid), > > + &(p_rem_node->node_info.node_guid), > > + sizeof(uint64_t)); > > + remote_guids[*p_num_remote_guids].forwarded_to = 0; > > + p_remote_guid = &remote_guids[*p_num_remote_guids]; > > + (*p_num_remote_guids)++; > > + } > > + } > > + > > + if (port_offsetting) { > > + /* Keep track of the destinations we've seen so far */ > > + p_remote_dest = NULL; > > + for (i = 0; i < num_remote_dests; i++) { > > + if (!memcmp(p_remote_guid, > > + (&remote_dests[i])->p_remote_guid, > > + sizeof(struct osm_switch_remote_dest))) { > > Should there be sizeof(osm_switch_guid_count_t)? > > Sasha > > > + p_remote_dest = &remote_dests[i]; > > + break; > > + } > > + } > > + if (!p_remote_dest) { > > + p_remote_dest = &remote_dests[num_remote_dests]; > > + p_remote_dest->p_remote_guid = p_remote_guid; > > + p_remote_dest->total_paths = 0; > > + p_remote_dest->min_paths = 0xFFFFFFFF; > > + p_remote_dest->ports_count = 0; > > + num_remote_dests++; > > + } > > + p_remote_dest->total_paths += check_count; > > + if (check_count < p_remote_dest->min_paths) > > + p_remote_dest->min_paths = check_count; > > + p_remote_dest->ports[p_remote_dest->ports_count] = port_num; > > + p_remote_dest->ports_count++; > > + > > + total_paths_count += check_count; > > + num_potential_ports++; > > } > > > > /* routing for LMC mode */ > > @@ -523,21 +631,23 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > best_port = port_num; > > least_paths = check_count; > > if (routing_for_lmc > > - && p_remote_guid > > + && p_remote_guid->forwarded_to > > && p_remote_guid->forwarded_to < least_forwarded_to) > > least_forwarded_to = p_remote_guid->forwarded_to; > > } > > else if (routing_for_lmc > > - && p_remote_guid > > && check_count == least_paths > > + && p_remote_guid->forwarded_to > > && p_remote_guid->forwarded_to < least_forwarded_to) { > > least_forwarded_to = p_remote_guid->forwarded_to; > > best_port = port_num; > > } > > } > > > > - if (port_found == FALSE) > > - return (OSM_NO_PATH); > > + if (port_found == FALSE) { > > + best_port = OSM_NO_PATH; > > + goto Exit; > > + } > > > > /* > > if we are in enhanced routing mode and the best port is not > > @@ -555,24 +665,110 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > remote_guids, > > p_num_remote_guids, > > best_port); > > + /* Must be stored */ > > + CL_ASSERT(p_remote_guid); > > + *p_remote_guid_count_used = p_remote_guid; > > + } > > > > - if (!p_remote_guid) { > > - /* track the remote node and system of the port used. */ > > - p_physp = osm_node_get_physp_ptr(p_sw->p_node, best_port); > > - p_rem_physp = osm_physp_get_remote(p_physp); > > - p_rem_node = osm_physp_get_node_ptr(p_rem_physp); > > - memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), > > - &(p_rem_node->node_info.sys_guid), > > - sizeof(uint64_t)); > > - memcpy(&(remote_guids[*p_num_remote_guids].node_guid), > > - &(p_rem_node->node_info.node_guid), > > - sizeof(uint64_t)); > > - remote_guids[*p_num_remote_guids].forwarded_to = 0; > > - (*p_num_remote_guids)++; > > + /* > > + * As an example of what we're trying to do with port > > + * offsetting, assume LMC = 2 and we are trying to route > > + * the lids of 4 ports. The lids of these 4 ports are: > > + * > > + * (1,2,3,4) > > + * (5,6,7,8) > > + * (9,10,11,12) > > + * (13,14,15,16) > > + * > > + * Suppose forwarding to all these lids goes through > > + * 4 specific switch ports. If we just cycle through > > + * ports and lids in a normal iterative fashion, we would > > + * normally forward out ports in this manner. > > + * > > + * switch port 1: 1, 5, 9, 13 > > + * switch port 2: 2, 6, 10, 14 > > + * switch port 3: 3, 7, 11, 15 > > + * switch port 4: 4, 8, 12, 1 > > + * > > + * Note that the base lid of each port (lids 1, 5, 9, 13) > > + * are all routed out of switch port 1. Thus, if the user > > + * only uses the base lid of each port, they will get pretty > > + * bad performance. We will try to get this layout instead. > > + * > > + * switch port 1: 1, 8, 11, 14 > > + * switch port 2: 2, 5, 12, 15 > > + * switch port 3: 3, 6, 9, 16 > > + * switch port 4: 4, 7, 10, 13 > > + * > > + * where switch ports are distributed in a more even manner. > > + * The base lid of each port is now distributed evenly > > + * across all 4 switch ports. The remaining lids are still > > + * distributed evenly across all the remaining switch ports. > > + * > > + * In order to accomplish this, we (effectively) will iterate > > + * through all ports like before, but instead of iterating from > > + * 0 to N-1 all the time, we will select the starting index > > + * based on the number of paths we have routed thus far. > > + */ > > + > > + /* We will not do port offsetting if num_potential_ports == 1 > > + * b/c there is no offsetting that can be done. > > + */ > > + if (port_offsetting > > + && best_port > > + && num_potential_ports > 1) { > > + /* Select which destination we want to forward to with our > > + * offsetting loop. > > + */ > > + lcm = _lcm(port_offsetting_lids_per_port, > > + num_remote_dests); > > + indx = (total_paths_count / lcm) % num_remote_dests; > > + for (i = 0; i < num_remote_dests; i++) { > > + p_remote_dest = &remote_dests[indx]; > > + p_remote_guid = p_remote_dest->p_remote_guid; > > + > > + if (p_remote_guid->forwarded_to == least_forwarded_to > > + && p_remote_dest->min_paths == least_paths) { > > + dest_found = TRUE; > > + break; > > + } > > + > > + indx++; > > + if (indx >= num_remote_dests) > > + indx = 0; > > + } > > + > > + /* Then we "offset" within the potential ports we could > > + * forward out of for this specific destination. > > + */ > > + if (dest_found && p_remote_dest) { > > + lcm = _lcm(port_offsetting_lids_per_port, > > + p_remote_dest->ports_count); > > + indx = (p_remote_dest->total_paths / lcm) % p_remote_dest->ports_count; > > + for (i = 0; i < p_remote_dest->ports_count; i++) { > > + port_num = p_remote_dest->ports[indx]; > > + check_count = > > + osm_port_prof_path_count_get(&p_sw->p_prof[port_num]); > > + if (check_count == least_paths) { > > + if (best_port != port_num) > > + osm_log(p_sw->p_log, OSM_LOG_DEBUG, > > + "osm_switch_recommend_path: " > > + "best port offsetted: %d to %d\n", > > + best_port, port_num); > > + best_port = port_num; > > + *p_remote_guid_count_used = p_remote_dest->p_remote_guid; > > + break; > > + } > > + indx++; > > + if (indx >= p_remote_dest->ports_count) > > + indx = 0; > > + } > > } > > - *p_remote_guid_count_used = p_remote_guid; > > } > > > > +Exit: > > + if (remote_dests) > > + free(remote_dests); > > return (best_port); > > } > > > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > > index 938db84..501c2c7 100644 > > --- a/opensm/opensm/osm_ucast_mgr.c > > +++ b/opensm/opensm/osm_ucast_mgr.c > > @@ -212,18 +212,23 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > osm_switch_guid_count_t *remote_guids = NULL; > > uint16_t num_used_guids = 0; > > osm_switch_guid_count_t *p_remote_guid_used = NULL; > > + uint16_t port_offsetting_lids_per_port = 0; > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > if (lids_per_port > 1) { > > - remote_guids = malloc(sizeof(osm_switch_guid_count_t) * lids_per_port); > > + uint8_t num_ports = p_sw->num_ports; > > + remote_guids = malloc(sizeof(osm_switch_guid_count_t) * num_ports); > > if (remote_guids == NULL) { > > osm_log(p_mgr->p_log, OSM_LOG_ERROR, > > "__osm_ucast_mgr_process_port: ERR 3A09: " > > "Cannot allocate array. Insufficient memory\n"); > > goto Exit; > > } > > - memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * lids_per_port); > > + memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * num_ports); > > + > > + if (p_mgr->p_subn->opt.port_offsetting) > > + port_offsetting_lids_per_port = lids_per_port; > > } > > > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > > @@ -270,14 +275,15 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > p_mgr->is_dor, > > remote_guids, > > &num_used_guids, > > - &p_remote_guid_used); > > + &p_remote_guid_used, > > + port_offsetting_lids_per_port); > > } > > else > > port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > p_mgr->p_subn-> > > ignore_existing_lfts, > > p_mgr->is_dor, > > - NULL, NULL, NULL); > > + NULL, NULL, NULL, 0); > > > > /* > > There might be no path to the target > > -- > > 1.5.1 > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-move-remote_dests-array-allocation-down-to-remove-go.patch Type: text/x-patch Size: 2091 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-fix-incorrect-memcmp-length.patch Type: text/x-patch Size: 870 bytes Desc: not available URL: From wxuylnnognp at bmgseltec.com Tue Jun 3 23:47:57 2008 From: wxuylnnognp at bmgseltec.com (Claudia Martinez) Date: Wed, 4 Jun 2008 12:17:57 +0530 Subject: [ofa-general] Blue sexy pill - $0.{_2SYMBCHAR} Message-ID: <01c8c63d$05a40880$ed0b2e5c@wxuylnnognp> Blue sexy pill - $0.{_2SYMBCHAR}Have a look at our site -------------- next part -------------- An HTML attachment was scrubbed... URL: From staturevrg835 at danegg.com Wed Jun 4 01:47:16 2008 From: staturevrg835 at danegg.com (Clair Gomes) Date: Wed, 4 Jun 2008 10:47:16 +0200 Subject: [ofa-general] Award: We are waiting for your nomination Message-ID: <01c8c630$5a8d5200$9699af4e@staturevrg835> Bacheelor, MasteerMBA, and Doctoraate diplomas available in the field of your choice that's right, you can even become a Doctor and receive all the benefits that comes with it! Our Diplomas/Certificates are recognised in most countries No required examination, tests, classes, books, or interviews. ** No one is turned down ** Confidentiality assured CALL US 24 HOURS A DAY, 7 DAYS A WEEK For US: 1-801-504-2132 Outside US: +1-801-504-2132 "Just leave your NAME & PHONE NO. (with CountryCode)" in the voicemail our staff will get back to you in next few days -------------- next part -------------- An HTML attachment was scrubbed... URL: From chronicbmu58 at quintekgroup.com Wed Jun 4 01:52:12 2008 From: chronicbmu58 at quintekgroup.com (Adolph Presley) Date: Wed, 4 Jun 2008 12:22:12 +0330 Subject: [ofa-general] Now you can get it before anyone. Message-ID: <01c8c63d$9da1f200$be55625b@chronicbmu58> Bacheelor, MasteerMBA, and Doctoraate diplomas available in the field of your choice that's right, you can even become a Doctor and receive all the benefits that comes with it! Our Diplomas/Certificates are recognised in most countries No required examination, tests, classes, books, or interviews. ** No one is turned down ** Confidentiality assured CALL US 24 HOURS A DAY, 7 DAYS A WEEK For US: 1-801-504-2132 Outside US: +1-801-504-2132 "Just leave your NAME & PHONE NO. (with CountryCode)" in the voicemail our staff will get back to you in next few days -------------- next part -------------- An HTML attachment was scrubbed... URL: From PHF at zurich.ibm.com Wed Jun 4 04:24:04 2008 From: PHF at zurich.ibm.com (Philip Frey1) Date: Wed, 4 Jun 2008 13:24:04 +0200 Subject: [ofa-general] RNIC resource limits Message-ID: Hello, I have asked this question about RNIC resource limits before: > > Could you give me some insight in what the limits of the Chelsio RNIC > > are? (Max MRs, QPs, PDs etc) > > > > Many thanks and kind regards, > > Philip > > Try running ibv_devinfo -v to see driver/hw limits. > > However, how are you limited? Are you getting failures registering > memory? Did you try setting your ulimit -l to unlimited or at least as > large as the memory region you want to register? > > Steve. Steve, thanks for the answer! When running 'ibv_devinfo -v' on my Chelsio RNIC (T3) with OFED 1.3 (FW 5.0) I get the following: [root at achilles ~]# ibv_devinfo -v hca_id: cxgb3_0 fw_ver: 0.0.0 node_guid: 0007:4301:33f7:0000 sys_image_guid: 0007:4301:33f7:0000 vendor_id: 0x1425 vendor_part_id: 49 hw_ver: 0x0 board_id: 1425.31 phys_port_cnt: 2 max_mr_size: 0xffffffffffffffff page_size_cap: 0x0 max_qp: 32736 max_qp_wr: 16777215 device_cap_flags: 0x00038000 max_sge: 4 max_sge_rd: 1 max_cq: 32767 max_cqe: 16777215 max_mr: 32768 max_pd: 32767 max_qp_rd_atom: 8 max_ee_rd_atom: 0 max_res_rd_atom: 0 max_qp_init_rd_atom: 8 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_NONE (0) max_ee: 0 max_rdd: 0 max_mw: 0 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 0 max_mcast_grp: 0 max_mcast_qp_attach: 0 max_total_mcast_qp_attach: 0 max_ah: 0 max_fmr: 0 max_srq: 0 max_pkeys: 0 local_ca_ack_delay: 0 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: invalid MTU (225) sm_lid: 0 port_lid: 0 port_lmc: 0x00 max_msg_sz: 0xffffffff port_cap_flags: 0x009f0000 max_vl_num: invalid value (255) bad_pkey_cntr: 0x213 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 1 gid_tbl_len: 1 subnet_timeout: 146 init_type_reply: 39 active_width: 4X (2) active_speed: 5.0 Gbps (2) phys_state: invalid physical state (0) port: 2 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: invalid MTU (225) sm_lid: 0 port_lid: 0 port_lmc: 0x00 max_msg_sz: 0xffffffff port_cap_flags: 0x009f0000 max_vl_num: invalid value (255) bad_pkey_cntr: 0x213 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 1 gid_tbl_len: 1 subnet_timeout: 146 init_type_reply: 39 active_width: 4X (2) active_speed: 5.0 Gbps (2) phys_state: invalid physical state (0) When creating a QP, I need to specify some capacity information: struct ibv_qp_cap { uint32_t max_send_wr; uint32_t max_recv_wr; uint32_t max_send_sge; uint32_t max_recv_sge; uint32_t max_inline_data; }; According to the above listing, I should be able to use: 16777215 WRs (max_qp_wr: 16777215) [is this per qp or total?] 4 SGEs (max_sge: 4) It does not say anything about the max_inline_data. I have tried to create a QP with as many resources as possible and found the following: max_send_wr cannot exceed 16384 max_recv_wr cannot exceed 1023 max_send_sge cannot exceed 4294967295 (maximum a uint32_t can hold) max_recv_sge cannot exceed 4294967295 (maximum a uint32_t can hold) max_inline_data cannot exceed 64 If the stated limits are exceeded, the call to ibv_qp_create() fails. I am wondering now, if I can really use as many WRs, SGEs and inline data as the figures above or not. It is also not clear to me if these figures represent per-QP values or if they are global max values across all QPs. Many thanks for you advice and best regards, Philip -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Wed Jun 4 04:37:00 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 04 Jun 2008 04:37:00 -0700 Subject: [ofa-general] [PATCH][MINOR] opensm/osm_sa_mcmember_record.c: Minor logic change in __get_new_mlid Message-ID: <1212579420.6127.57.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Minor logic change in __get_new_mlid Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index c982f40..3cfd5f7 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -181,10 +181,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) /* track all used mlids in the array (by mlid index) */ used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); - if (used_mlids_array) - memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); if (!used_mlids_array) return 0; + memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); /* scan all available multicast groups in the DB and fill in the table */ while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { From hrosenstock at xsigo.com Wed Jun 4 04:37:16 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 04 Jun 2008 04:37:16 -0700 Subject: [ofa-general] [PATCH][MINOR] opensm/osm_ucast_ftree.c: Eliminate unnecessary check in __osm_ftree_sw_tbl_element_create Message-ID: <1212579436.6127.58.camel@hrosenstock-ws.xsigo.com> opensm/osm_ucast_ftree.c: Eliminate unnecessary check in __osm_ftree_sw_tbl_element_create Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index e7104e5..caf231c 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -349,8 +349,7 @@ static ftree_sw_tbl_element_t *__osm_ftree_sw_tbl_element_create(IN ftree_sw_t * return NULL; memset(p_element, 0, sizeof(ftree_sw_tbl_element_t)); - if (p_element) - p_element->p_sw = p_sw; + p_element->p_sw = p_sw; return p_element; } From hrosenstock at xsigo.com Wed Jun 4 04:37:57 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 04 Jun 2008 04:37:57 -0700 Subject: [ofa-general] [PATCH] opensm/osm_pkey.c: Eliminate potential NULL pointer dereference Message-ID: <1212579477.6127.60.camel@hrosenstock-ws.xsigo.com> opensm/osm_pkey.c: Eliminate potential NULL pointer dereference Also, a comment reformat Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_pkey.c b/opensm/opensm/osm_pkey.c index 9b43669..c3b8394 100644 --- a/opensm/opensm/osm_pkey.c +++ b/opensm/opensm/osm_pkey.c @@ -151,8 +151,9 @@ osm_pkey_tbl_set(IN osm_pkey_tbl_t * p_pkey_tbl, if (!p_pkey_block) { p_pkey_block = (ib_pkey_table_t *) malloc(sizeof(ib_pkey_table_t)); - if (p_pkey_block) - memset(p_pkey_block, 0, sizeof(ib_pkey_table_t)); + if (!p_pkey_block) + return (IB_ERROR); + memset(p_pkey_block, 0, sizeof(ib_pkey_table_t)); cl_ptr_vector_set(&p_pkey_tbl->blocks, block, p_pkey_block); } @@ -163,8 +164,8 @@ osm_pkey_tbl_set(IN osm_pkey_tbl_t * p_pkey_tbl, NOTE: as the spec does not require uniqueness of PKeys in tables there is no other way but to refresh the entire keys map. - Moreover, if the same key exists but with full membership it should have - precedence on the key with limited membership ! + Moreover, if the same key exists but with full membership it should + have precedence on the key with limited membership ! */ cl_map_remove_all(&p_pkey_tbl->keys); From tenjoy at qdhuaxia.com Wed Jun 4 05:43:36 2008 From: tenjoy at qdhuaxia.com (Mia Brunson) Date: Wed, 4 Jun 2008 15:43:36 +0300 Subject: [ofa-general] Blue sexy pill - $0.{_2SYMBCHAR} Message-ID: <01c8c659$c0420400$a9becc58@tenjoy> Blue sexy pill - $0.{_2SYMBCHAR}Click the link -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Wed Jun 4 05:44:27 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 04 Jun 2008 05:44:27 -0700 Subject: [ofa-general] [PATCH] opensm/osm_port.c: Eliminate potential NULL pointer dereferences Message-ID: <1212583467.6127.64.camel@hrosenstock-ws.xsigo.com> opensm/osm_port.c: Eliminate potential NULL pointer dereferences Also, comment format change Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_port.c b/opensm/opensm/osm_port.c index 3398d04..d66e6be 100644 --- a/opensm/opensm/osm_port.c +++ b/opensm/opensm/osm_port.c @@ -127,8 +127,9 @@ osm_physp_init(IN osm_physp_t * const p_physp, cl_ptr_vector_init(&p_physp->slvl_by_port, num_slvl, 1); for (i = 0; i < num_slvl; i++) { p_slvl = (ib_slvl_table_t *) malloc(sizeof(ib_slvl_table_t)); - if (p_slvl) - memset(p_slvl, 0, sizeof(ib_slvl_table_t)); + if (!p_slvl) + break; + memset(p_slvl, 0, sizeof(ib_slvl_table_t)); cl_ptr_vector_set(&p_physp->slvl_by_port, i, p_slvl); } @@ -594,6 +595,10 @@ osm_physp_replace_dr_path_with_alternate_dr_path(IN osm_log_t * p_log, boolean_t next_list_is_full = TRUE, reached_dest = FALSE; uint8_t num_ports, port_num; + p_nextPortsList = (cl_list_t *) malloc(sizeof(cl_list_t)); + if (!p_nextPortsList) + return; + /* initialize the map of all port participating in current dr path not including first and last switches @@ -609,7 +614,6 @@ osm_physp_replace_dr_path_with_alternate_dr_path(IN osm_log_t * p_log, BFS from OSM port until we find the target physp but avoid going through mapped ports */ - p_nextPortsList = (cl_list_t *) malloc(sizeof(cl_list_t)); cl_list_construct(p_nextPortsList); cl_list_init(p_nextPortsList, 10); @@ -638,12 +642,16 @@ osm_physp_replace_dr_path_with_alternate_dr_path(IN osm_log_t * p_log, next_list_is_full = FALSE; p_currPortsList = p_nextPortsList; p_nextPortsList = (cl_list_t *) malloc(sizeof(cl_list_t)); + if (!p_nextPortsList) { + p_nextPortsList = p_currPortsList; + goto Exit; + } cl_list_construct(p_nextPortsList); cl_list_init(p_nextPortsList, 10); p_physp = (osm_physp_t *) cl_list_remove_head(p_currPortsList); while (p_physp != NULL) { - /* If we are in a switch - need to go out through all the other - physical ports of the switch */ + /* If we are in a switch - need to go out through all + the other physical ports of the switch */ num_ports = osm_node_get_num_physp(p_physp->p_node); for (port_num = 1; port_num < num_ports; port_num++) { From swise at opengridcomputing.com Wed Jun 4 07:22:48 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 04 Jun 2008 09:22:48 -0500 Subject: [ofa-general] RNIC resource limits In-Reply-To: References: Message-ID: <4846A538.9040608@opengridcomputing.com> > [root at achilles ~]# ibv_devinfo -v > hca_id: cxgb3_0 > fw_ver: 0.0.0 > node_guid: 0007:4301:33f7:0000 > sys_image_guid: 0007:4301:33f7:0000 > vendor_id: 0x1425 > vendor_part_id: 49 > hw_ver: 0x0 > board_id: 1425.31 > phys_port_cnt: 2 > max_mr_size: 0xffffffffffffffff > page_size_cap: 0x0 > max_qp: 32736 > max_qp_wr: 16777215 > device_cap_flags: 0x00038000 > max_sge: 4 > max_sge_rd: 1 > max_cq: 32767 > max_cqe: 16777215 > max_mr: 32768 > max_pd: 32767 > max_qp_rd_atom: 8 > max_ee_rd_atom: 0 > max_res_rd_atom: 0 > max_qp_init_rd_atom: 8 > max_ee_init_rd_atom: 0 > atomic_cap: ATOMIC_NONE (0) > max_ee: 0 > max_rdd: 0 > max_mw: 0 > max_raw_ipv6_qp: 0 > max_raw_ethy_qp: 0 > max_mcast_grp: 0 > max_mcast_qp_attach: 0 > max_total_mcast_qp_attach: 0 > max_ah: 0 > max_fmr: 0 > max_srq: 0 > max_pkeys: 0 > local_ca_ack_delay: 0 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: invalid MTU (225) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > max_msg_sz: 0xffffffff > port_cap_flags: 0x009f0000 > max_vl_num: invalid value (255) > bad_pkey_cntr: 0x213 > qkey_viol_cntr: 0x0 > sm_sl: 0 > pkey_tbl_len: 1 > gid_tbl_len: 1 > subnet_timeout: 146 > init_type_reply: 39 > active_width: 4X (2) > active_speed: 5.0 Gbps (2) > phys_state: invalid physical state (0) > > port: 2 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: invalid MTU (225) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > max_msg_sz: 0xffffffff > port_cap_flags: 0x009f0000 > max_vl_num: invalid value (255) > bad_pkey_cntr: 0x213 > qkey_viol_cntr: 0x0 > sm_sl: 0 > pkey_tbl_len: 1 > gid_tbl_len: 1 > subnet_timeout: 146 > init_type_reply: 39 > active_width: 4X (2) > active_speed: 5.0 Gbps (2) > phys_state: invalid physical state (0) > > When creating a QP, I need to specify some capacity information: > struct ibv_qp_cap { > uint32_t max_send_wr; > uint32_t max_recv_wr; > uint32_t max_send_sge; > uint32_t max_recv_sge; > uint32_t max_inline_data; > }; > > According to the above listing, I should be able to use: > 16777215 WRs (max_qp_wr: 16777215) [is this per qp > or total?] > 4 SGEs (max_sge: 4) > > It does not say anything about the max_inline_data. > max inline supported is 64. > I have tried to create a QP with as many resources as possible and found > the following: > max_send_wr cannot exceed 16384 > max_recv_wr cannot exceed 1023 > max_send_sge cannot exceed 4294967295 (maximum a uint32_t can hold) > max_recv_sge cannot exceed 4294967295 (maximum a uint32_t can hold) > max_inline_data cannot exceed 64 > The limits described in the device attributes are max supported by the HW. The OS limits these to much lower. For example, the chelsio work queue must be in contiguous dma-coherent memory. Linux pretty much limits that to a max size of about 128KB. That boils down to the max_send_wr and max_recv_wr limits you are experiencing. Note that the chelsio device uses a single work queue to implement both the SQ and RQ. So if you set your max_send_wr to say 16, then you will be able to set max_recv_wr much larger than 1023. So the two max depths inter-relate, and are combined to allocate a single work queue. Hope that makes sense. send_sge and recv_sge are 4 max. the chelsio driver should be failing the create_qp with the values you are passing in. That's a bug. > If the stated limits are exceeded, the call to ibv_qp_create() fails. > > I am wondering now, if I can really use as many WRs, SGEs and inline > data as the figures > above or not. It is also not clear to me if these figures represent > per-QP values or > if they are global max values across all QPs. > You should use 4 for SGE depths and I would keep the SQ and RQ as shallow as possible. Steve. From swise at opengridcomputing.com Wed Jun 4 07:28:05 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 04 Jun 2008 09:28:05 -0500 Subject: [ofa-general] device attributes Message-ID: <4846A675.4050909@opengridcomputing.com> Roland/All, Should the device attributes (for instance max_send_wr) be the max supported by the HW or the max supported by the OS or something else? For instance: Chelsio's HW can handle very large work queues, but since Linux limits the size of contiguous dma coherent memory allocations, the actual limits are much smaller. Which should I be using for the device attributes? Also, the chelsio device uses a single work queue to implement the SQ and RQ abstractions. So the max SQ depth depends on the RQ depth and vice versa. This leads to device max attributes that aren't that useful. I'm wondering what application writes should glean from these attributes... Steve From tom at opengridcomputing.com Wed Jun 4 08:02:18 2008 From: tom at opengridcomputing.com (Tom Tucker) Date: Wed, 04 Jun 2008 10:02:18 -0500 Subject: [ofa-general] device attributes In-Reply-To: <4846A675.4050909@opengridcomputing.com> References: <4846A675.4050909@opengridcomputing.com> Message-ID: <1212591738.19840.18.camel@trinity.ogc.int> On Wed, 2008-06-04 at 09:28 -0500, Steve Wise wrote: > Roland/All, > > Should the device attributes (for instance max_send_wr) be the max > supported by the HW or the max supported by the OS or something else? > Something else. > For instance: Chelsio's HW can handle very large work queues, but since > Linux limits the size of contiguous dma coherent memory allocations, the > actual limits are much smaller. Basing the limit on an OS resource seems arbitrary and dangerous. Applications using advertised adapter resource limits will unnecessarily consume the maximum. > Which should I be using for the device > attributes? > Arbitrary knee jerk == 512. However, surveying the current app usage as well as the other manufacturers advertised limits will make it less arbitrary. > Also, the chelsio device uses a single work queue to implement the SQ > and RQ abstractions. So the max SQ depth depends on the RQ depth and > vice versa. This leads to device max attributes that aren't that useful. > So the "real" limit is the HW WQ max and therefore max SQ = HW WQ max - RQ max? Setting RQ and SQ to 512 solves this "problem." > I'm wondering what application writes should glean from these attributes... > I guess you mean "application writer"? Here's what I suggest: - Set the RQ and SQ max to some reasonable default limit (e.g. 512). - Add an escape hatch by providing module options to override the default max. Tom > Steve > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From swise at opengridcomputing.com Wed Jun 4 08:01:58 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 04 Jun 2008 10:01:58 -0500 Subject: [ofa-general] device attributes In-Reply-To: <1212591738.19840.18.camel@trinity.ogc.int> References: <4846A675.4050909@opengridcomputing.com> <1212591738.19840.18.camel@trinity.ogc.int> Message-ID: <4846AE66.5080800@opengridcomputing.com> > Here's what I suggest: > > - Set the RQ and SQ max to some reasonable default limit (e.g. 512). > - Add an escape hatch by providing module options to override > the default max. > Or don't actually enforce the 512 max. IE set the attrs to what we think apps should use or max out at, but allow larger ones if the OS can handle it. Stevo. From noxiousei8 at jd-photography.com Wed Jun 4 14:09:37 2008 From: noxiousei8 at jd-photography.com (Terrance Middleton) Date: Wed, 4 Jun 2008 22:09:37 +0100 Subject: [ofa-general] How many CEOs Went to School? Message-ID: <01c8c68f$ad4a2680$f8400453@noxiousei8> Bacheelor, MasteerMBA, and Doctoraate diplomas available in the field of your choice that's right, you can even become a Doctor and receive all the benefits that comes with it! Our Diplomas/Certificates are recognised in most countries No required examination, tests, classes, books, or interviews. ** No one is turned down ** Confidentiality assured CALL US 24 HOURS A DAY, 7 DAYS A WEEK For US: 1-801-504-2132 Outside US: +1-801-504-2132 "Just leave your NAME & PHONE NO. (with CountryCode)" in the voicemail our staff will get back to you in next few days -------------- next part -------------- An HTML attachment was scrubbed... URL: From dotanba at gmail.com Thu Jun 5 14:39:49 2008 From: dotanba at gmail.com (Dotan Barak) Date: Thu, 05 Jun 2008 23:39:49 +0200 Subject: ***SPAM*** Re: [ofa-general] RNIC resource limits In-Reply-To: References: Message-ID: <48485D25.5020003@gmail.com> Hi. Philip Frey1 wrote: > > Hello, > > I have asked this question about RNIC resource limits before: > > > > > > Could you give me some insight in what the limits of the Chelsio RNIC > > > are? (Max MRs, QPs, PDs etc) > > > > > > Many thanks and kind regards, > > > Philip > > > > > > > Try running ibv_devinfo -v to see driver/hw limits. > > > > However, how are you limited? Are you getting failures registering > > memory? Did you try setting your ulimit -l to unlimited or at least as > > large as the memory region you want to register? > > > > Steve. > > Steve, thanks for the answer! > > When running 'ibv_devinfo -v' on my Chelsio RNIC (T3) with OFED 1.3 > (FW 5.0) > I get the following: > > [root at achilles ~]# ibv_devinfo -v > hca_id: cxgb3_0 > fw_ver: 0.0.0 > node_guid: 0007:4301:33f7:0000 > sys_image_guid: 0007:4301:33f7:0000 > vendor_id: 0x1425 > vendor_part_id: 49 > hw_ver: 0x0 > board_id: 1425.31 > phys_port_cnt: 2 > max_mr_size: 0xffffffffffffffff > page_size_cap: 0x0 > max_qp: 32736 > max_qp_wr: 16777215 > device_cap_flags: 0x00038000 > max_sge: 4 > max_sge_rd: 1 > max_cq: 32767 > max_cqe: 16777215 > max_mr: 32768 > max_pd: 32767 > max_qp_rd_atom: 8 > max_ee_rd_atom: 0 > max_res_rd_atom: 0 > max_qp_init_rd_atom: 8 > max_ee_init_rd_atom: 0 > atomic_cap: ATOMIC_NONE (0) > max_ee: 0 > max_rdd: 0 > max_mw: 0 > max_raw_ipv6_qp: 0 > max_raw_ethy_qp: 0 > max_mcast_grp: 0 > max_mcast_qp_attach: 0 > max_total_mcast_qp_attach: 0 > max_ah: 0 > max_fmr: 0 > max_srq: 0 > max_pkeys: 0 > local_ca_ack_delay: 0 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: invalid MTU (225) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > max_msg_sz: 0xffffffff > port_cap_flags: 0x009f0000 > max_vl_num: invalid value (255) > bad_pkey_cntr: 0x213 > qkey_viol_cntr: 0x0 > sm_sl: 0 > pkey_tbl_len: 1 > gid_tbl_len: 1 > subnet_timeout: 146 > init_type_reply: 39 > active_width: 4X (2) > active_speed: 5.0 Gbps (2) > phys_state: invalid physical state > (0) > > port: 2 > state: PORT_ACTIVE (4) > max_mtu: 4096 (5) > active_mtu: invalid MTU (225) > sm_lid: 0 > port_lid: 0 > port_lmc: 0x00 > max_msg_sz: 0xffffffff > port_cap_flags: 0x009f0000 > max_vl_num: invalid value (255) > bad_pkey_cntr: 0x213 > qkey_viol_cntr: 0x0 > sm_sl: 0 > pkey_tbl_len: 1 > gid_tbl_len: 1 > subnet_timeout: 146 > init_type_reply: 39 > active_width: 4X (2) > active_speed: 5.0 Gbps (2) > phys_state: invalid physical state > (0) > > When creating a QP, I need to specify some capacity information: > struct ibv_qp_cap { > uint32_t max_send_wr; > uint32_t max_recv_wr; > uint32_t max_send_sge; > uint32_t max_recv_sge; > uint32_t max_inline_data; > }; > > According to the above listing, I should be able to use: > 16777215 WRs (max_qp_wr: 16777215) [is this per qp > or total?] > 4 SGEs (max_sge: 4) > > It does not say anything about the max_inline_data. If you don't use inline data, you should zero this attribute. > > I have tried to create a QP with as many resources as possible and > found the following: > max_send_wr cannot exceed 16384 > max_recv_wr cannot exceed 1023 > max_send_sge cannot exceed 4294967295 (maximum a uint32_t can > hold) > max_recv_sge cannot exceed 4294967295 (maximum a uint32_t can > hold) > max_inline_data cannot exceed 64 > > If the stated limits are exceeded, the call to ibv_qp_create() fails. > > I am wondering now, if I can really use as many WRs, SGEs and inline > data as the figures > above or not. It is also not clear to me if these figures represent > per-QP values or > if they are global max values across all QPs. The values in the HCA capability should be per QP, for example if max_qp_wr = 0xffff, every QP that you open can have this amount of WR (unless there is a memory problem). I think that the mentioned value for max_qp_wr is just too high (many a driver bug?). > > Many thanks for you advice and best regards, > Philip Dotan From tennvuaqwl at csa.com Wed Jun 4 19:49:22 2008 From: tennvuaqwl at csa.com (Hattie Metcalf) Date: Thu, 5 Jun 2008 05:49:22 +0300 Subject: [ofa-general] Blue sexy pill - $0.81 Message-ID: <01c8c6cf$e73b1d00$9aa18c55@tennvuaqwl> Blue sexy pill - $0.81Click the link -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Wed Jun 4 21:07:31 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Jun 2008 21:07:31 -0700 Subject: [ofa-general] Re: device attributes In-Reply-To: <4846A675.4050909@opengridcomputing.com> (Steve Wise's message of "Wed, 04 Jun 2008 09:28:05 -0500") References: <4846A675.4050909@opengridcomputing.com> Message-ID: > Should the device attributes (for instance max_send_wr) be the max > supported by the HW or the max supported by the OS or something else? I think they should be what the consumer can expect to work -- ie the upper limits on the current system, whether those limits are imposed by the HW, driver, or whatever. > For instance: Chelsio's HW can handle very large work queues, but > since Linux limits the size of contiguous dma coherent memory > allocations, the actual limits are much smaller. Which should I be > using for the device attributes? The smaller limit I think. > Also, the chelsio device uses a single work queue to implement the SQ > and RQ abstractions. So the max SQ depth depends on the RQ depth and > vice versa. This leads to device max attributes that aren't that > useful. Not nice... I guess you could return half of the work queue depth for each max since that is guaranteed to work? > I'm wondering what application writes should glean from these attributes... I think the limits are pointless unless they give something that an application can actually request. Some of the limits are pretty useless, eg max # of CQs, since another app may have already used up all the CQs. - R. From rdreier at cisco.com Wed Jun 4 21:10:26 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Jun 2008 21:10:26 -0700 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <1212518703.2217.22.camel@lap75545.ornl.gov> (David Dillow's message of "Tue, 03 Jun 2008 14:45:03 -0400") References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> Message-ID: > Or so the theory goes. Unfortunately, you need all that information > before you can create the connection. The configfs guys have thought > about that, but not implemented yet: > > > [Committable Items] > > NOTE: Committable items are currently unimplemented. The netconsole code in-tree has a separate "enabled" attribute that serves the purpose of "committing" something. Seems good enough for SRP to use to me... the rename to commit idea seems cute but I don't see that it buys much beyond this. From rdreier at cisco.com Wed Jun 4 21:12:41 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Jun 2008 21:12:41 -0700 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: <4844E40A.9080209@mellanox.co.il> (Yevgeny Petrilin's message of "Tue, 03 Jun 2008 09:26:18 +0300") References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> Message-ID: > I resent the two patches that handle multiple completion vectors last week. > What is the status for these patches? Are there further changes that need to be done, > or will these patches be applied? I guess the patches look mostly OK, but are there any results that give motivation for applying this? I'm not asking about "multicore systems need multiple EQs so we should do this" level of vagueness, but "we have application and/or ULP X that shows improvement Z using this." Also any thoughts on how an app/ULP can sanely choose which vector to use? - R. From rdreier at cisco.com Wed Jun 4 21:13:26 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Jun 2008 21:13:26 -0700 Subject: [ofa-general] Re: [PATCH] IB/ipoib: increase ring sizes In-Reply-To: <1211786440.13769.54.camel@mtls03> (Eli Cohen's message of "Mon, 26 May 2008 10:20:40 +0300") References: <1211786440.13769.54.camel@mtls03> Message-ID: > Increase IPoIB ring sizes to twice the original size to act as > a shock observer for high traffic picks. Looks fine but I would like to include a little more motivation in the changelog. What type of workload benefits from this change? From rdreier at cisco.com Wed Jun 4 21:20:35 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Jun 2008 21:20:35 -0700 Subject: [ofa-general][PATCH 8/12 v1] mlx4: Dynamic port configuration In-Reply-To: <480F5026.9070400@mellanox.co.il> (Yevgeny Petrilin's message of "Wed, 23 Apr 2008 18:05:10 +0300") References: <480F5026.9070400@mellanox.co.il> Message-ID: > +static int mlx4_change_port_types(struct mlx4_dev *dev, > + enum mlx4_port_type *port_types) > +{ > + int i; > + int err = 0; > + int change = 0; > + int port; > + > + for (i = 0; i < MLX4_MAX_PORTS; i++) { > + if (port_types[i] != dev->caps.port_type[i + 1]) { > + change = 1; > + dev->caps.port_type[i + 1] = port_types[i]; > + } > + } > + if (change) { > + mlx4_unregister_device(dev); > + for (port = 1; port <= dev->caps.num_ports; port++) { > + mlx4_CLOSE_PORT(dev, port); > + err = mlx4_SET_PORT(dev, port); > + if (err) { > + mlx4_err(dev, "Failed to set port %d, " > + "aborting\n", port); > + return err; > + } > + } > + err = mlx4_register_device(dev); > + } > + return err; > +} If I read the code correctly, there is no locking around this, so multiple processes could race and cause all sorts of problems. And also you do the assignment > + dev->caps.port_type[i + 1] = port_types[i]; before unregistering the device -- so there is a window where caps.port_type has the wrong data -- not sure if this is a real issue. > +static ssize_t show_port_type(struct device *dev, > + struct device_attribute *attr, > + char *buf) > +{ > + struct pci_dev *pdev = to_pci_dev(dev); > + struct mlx4_dev *mdev = pci_get_drvdata(pdev); > + int i; > + > + sprintf(buf, "Current port types:\n"); > + for (i = 1; i <= MLX4_MAX_PORTS; i++) { > + sprintf(buf, "%sPort%d: %s\n", buf, i, > + (mdev->caps.port_type[i] == MLX4_PORT_TYPE_IB)? > + "ib": "eth"); > + } > + return strlen(buf); > +} This violates the one-value-per-file rule for sysfs if I am reading the code correctly. - R. From dillowda at ornl.gov Wed Jun 4 22:40:35 2008 From: dillowda at ornl.gov (Dave Dillow) Date: Thu, 5 Jun 2008 01:40:35 -0400 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> Message-ID: <20080605054035.GA8059@ornl.gov> On Wed, Jun 04, 2008 at 09:10:26PM -0700, Roland Dreier wrote: > > Or so the theory goes. Unfortunately, you need all that information > > before you can create the connection. The configfs guys have thought > > about that, but not implemented yet: > > > > > [Committable Items] > > > NOTE: Committable items are currently unimplemented. > > The netconsole code in-tree has a separate "enabled" attribute that > serves the purpose of "committing" something. Seems good enough for SRP > to use to me... the rename to commit idea seems cute but I don't see > that it buys much beyond this. But... But... I've got nothing. I mentioned the enable attribute as a possible way to do it, though it is counter to the configfs's documented preference. But it's there, it works perfectly well, and the configfs guys have had over 2 years to implement their alternate commit feature. That said, given that SRP's been using sysfs since it went in, is there a reason to move to configfs other than it's the new preferred way to do it? Given the desire to not break ABI's -- and IIRC sysfs was declared to be under that unbrella -- wouldn't we have to at least carry both interfaces for a while, assuming we can even get rid of the sysfs one? Carrying both adds a bit of a interesting twist -- targets added using the sysfs add-target wouldn't show up under configfs. It may not be a real problem, but it could be a bit of a surprise to an admin. I'm not opposed to configfs, but the more I think about it, it doesn't seem to bring much to the table for the SRP initiator other more code and data structure size. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From tziporet at dev.mellanox.co.il Thu Jun 5 00:49:57 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 05 Jun 2008 10:49:57 +0300 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> Message-ID: <48479AA5.3010303@mellanox.co.il> Roland Dreier wrote: > > I guess the patches look mostly OK, but are there any results that give > motivation for applying this? I'm not asking about "multicore systems > need multiple EQs so we should do this" level of vagueness, but "we have > application and/or ULP X that shows improvement Z using this." Also any > thoughts on how an app/ULP can sanely choose which vector to use? > > We will post the mlnx_en shortly that use these vectors. Also when IPoIB will use RSS it will need multiple vectors. So this can wait for 2.6.27 Tziporet From eli at dev.mellanox.co.il Thu Jun 5 01:20:37 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 05 Jun 2008 11:20:37 +0300 Subject: [ofa-general] Re: [PATCH] IB/ipoib: increase ring sizes In-Reply-To: References: <1211786440.13769.54.camel@mtls03> Message-ID: <1212654037.31801.13.camel@mtls03> On Wed, 2008-06-04 at 21:13 -0700, Roland Dreier wrote: > > Increase IPoIB ring sizes to twice the original size to act as > > a shock observer for high traffic picks. First let me correct myself, absorber and not observer. > > Looks fine but I would like to include a little more motivation in the > changelog. What type of workload benefits from this change? So this is what I suggest for the changelog: Increase IPoIB ring sizes to twice the original size to act as a shock absorber for high traffic picks. With the current settings, we have seen cases that there are many calls to netif_stop_queue()/netif_wake_queue(). This causes degradation in throughput. Also larger receive buffer sizes help IPoIB in CM mode to avoid experiencing RNR NAK conditions due to insufficient receive buffers at the SRQ. This increase is still modest compared to other Ethernet drivers -- bnx2 uses 256 for it HW queue length (on 4K page size machines). From roybclee at sympatico.ca Thu Jun 5 05:01:07 2008 From: roybclee at sympatico.ca (UK LOTTERY INTL.) Date: Thu, 5 Jun 2008 8:01:07 -0400 Subject: [ofa-general] Contact Mr.Pinkett Griffin Message-ID: <6qh28a$1u4r3h@alconsout.srvr.bell.ca> Contact Mr.Pinkett Griffin for the claim of £1.000.000pounds which youhave won in UK-LOTTERY COMPANY. Provide your Names,Address,Age,Occupation, Tel,Country.Email:mailsalert at y7mail.com Sincerely, Mrs.Rose Woods. From hrosenstock at xsigo.com Thu Jun 5 05:36:59 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 05:36:59 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/sim_mad.c: Cosmetic change to some debug messages Message-ID: <1212669419.6127.129.camel@hrosenstock-ws.xsigo.com> ibsim/sim_mad.c: Cosmetic change to some debug messages Signed-off-by: Hal Rosenstock diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c old mode 100644 new mode 100755 index da96f35..675d95b --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1058,7 +1058,7 @@ static Port *direct_route_in_MAD(Port * port, ib_dr_path_t * path) { unsigned ptr; - DEBUG("route_in: path %s hops %d\n", pathstr(0, path), path->cnt); + DEBUG("route_in: path %s hops %d", pathstr(0, path), path->cnt); for (ptr = path->cnt; ptr; ptr--) { if (ptr < path->cnt && port->node->type != SWITCH_NODE) @@ -1068,7 +1068,7 @@ static Port *direct_route_in_MAD(Port * port, ib_dr_path_t * path) return NULL; } - DEBUG("routed in to node %s port %d (%p)\n", + DEBUG("routed in to node %s port %d (%p)", port->node->nodeid, port->portnum, port); return port; @@ -1078,7 +1078,7 @@ static Port *direct_route_out_MAD(Port * port, ib_dr_path_t * path) { unsigned ptr = 0; - DEBUG("route_out: path %s hops %d\n", pathstr(0, path), path->cnt); + DEBUG("route_out: path %s hops %d", pathstr(0, path), path->cnt); while (ptr < path->cnt) { if (ptr && port->node->type != SWITCH_NODE) @@ -1090,7 +1090,7 @@ static Port *direct_route_out_MAD(Port * port, ib_dr_path_t * path) } path->p[ptr++] = port->portnum; - DEBUG("routed out to node %s port %d (%p) return path %s\n", + DEBUG("routed out to node %s port %d (%p) return path %s", port->node->nodeid, port->portnum, port, pathstr(0, path)); return port; @@ -1144,7 +1144,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) *dcl = cl; - DEBUG("client id %d, size %d", cl->id, size); + DEBUG("client %d, size %d", cl->id, size); if (size != sizeof(*r)) { IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); return -1; From hrosenstock at xsigo.com Thu Jun 5 05:37:05 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 05:37:05 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/ibsim.c: Cosmetic change to some debug messages Message-ID: <1212669425.6127.131.camel@hrosenstock-ws.xsigo.com> ibsim/ibsim.c: Cosmetic change to some debug messages Signed-off-by: Hal Rosenstock diff --git a/ibsim/ibsim.c b/ibsim/ibsim.c old mode 100644 new mode 100755 index 8e97e21..e4ab6bc --- a/ibsim/ibsim.c +++ b/ibsim/ibsim.c @@ -221,10 +221,10 @@ static int sim_ctl_new_client(Client * cl, struct sim_ctl * ctl, union name_t *f return -1; } cl->port = node_get_port(node, 0); - VERB("Attaching client %d at node \"%s\"/port 0x%" PRIx64, + VERB("Attaching client %d at node \"%s\" port 0x%" PRIx64, i, node->nodeid, cl->port->portguid); } else { - VERB("Attaching client %d at default node \"%s\"/port 0x%" + VERB("Attaching client %d at default node \"%s\" port 0x%" PRIx64, i, default_port->node->nodeid, default_port->portguid); cl->port = default_port; @@ -275,7 +275,7 @@ static int sim_ctl_disconnect_client(Client * cl, struct sim_ctl * ctl) return 0; // ? } - DEBUG("Detaching client %d from node \"%s\"/port 0x%" PRIx64, + DEBUG("Detaching client %d from node \"%s\" port 0x%" PRIx64, client, cl->port->node->nodeid, cl->port->portguid); cl->pid = 0; From hrosenstock at xsigo.com Thu Jun 5 05:51:47 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 05:51:47 -0700 Subject: [ofa-general] [PATCHv2][TRIVIAL] ibsim/ibsim.c: Cosmetic change to some debug messages Message-ID: <1212670307.6127.136.camel@hrosenstock-ws.xsigo.com> ibsim/ibsim.c: Cosmetic change to some debug messages Signed-off-by: Hal Rosenstock diff --git a/ibsim/ibsim.c b/ibsim/ibsim.c index 8e97e21..e4ab6bc --- a/ibsim/ibsim.c +++ b/ibsim/ibsim.c @@ -221,10 +221,10 @@ static int sim_ctl_new_client(Client * cl, struct sim_ctl * ctl, union name_t *f return -1; } cl->port = node_get_port(node, 0); - VERB("Attaching client %d at node \"%s\"/port 0x%" PRIx64, + VERB("Attaching client %d at node \"%s\" port 0x%" PRIx64, i, node->nodeid, cl->port->portguid); } else { - VERB("Attaching client %d at default node \"%s\"/port 0x%" + VERB("Attaching client %d at default node \"%s\" port 0x%" PRIx64, i, default_port->node->nodeid, default_port->portguid); cl->port = default_port; @@ -275,7 +275,7 @@ static int sim_ctl_disconnect_client(Client * cl, struct sim_ctl * ctl) return 0; // ? } - DEBUG("Detaching client %d from node \"%s\"/port 0x%" PRIx64, + DEBUG("Detaching client %d from node \"%s\" port 0x%" PRIx64, client, cl->port->node->nodeid, cl->port->portguid); cl->pid = 0; From hrosenstock at xsigo.com Thu Jun 5 05:51:45 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 05:51:45 -0700 Subject: [ofa-general] [PATCHv2][TRIVIAL] ibsim/sim_mad.c: Cosmetic change to some debug messages Message-ID: <1212670305.6127.135.camel@hrosenstock-ws.xsigo.com> ibsim/sim_mad.c: Cosmetic change to some debug messages Signed-off-by: Hal Rosenstock diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index da96f35..675d95b --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1058,7 +1058,7 @@ static Port *direct_route_in_MAD(Port * port, ib_dr_path_t * path) { unsigned ptr; - DEBUG("route_in: path %s hops %d\n", pathstr(0, path), path->cnt); + DEBUG("route_in: path %s hops %d", pathstr(0, path), path->cnt); for (ptr = path->cnt; ptr; ptr--) { if (ptr < path->cnt && port->node->type != SWITCH_NODE) @@ -1068,7 +1068,7 @@ static Port *direct_route_in_MAD(Port * port, ib_dr_path_t * path) return NULL; } - DEBUG("routed in to node %s port %d (%p)\n", + DEBUG("routed in to node %s port %d (%p)", port->node->nodeid, port->portnum, port); return port; @@ -1078,7 +1078,7 @@ static Port *direct_route_out_MAD(Port * port, ib_dr_path_t * path) { unsigned ptr = 0; - DEBUG("route_out: path %s hops %d\n", pathstr(0, path), path->cnt); + DEBUG("route_out: path %s hops %d", pathstr(0, path), path->cnt); while (ptr < path->cnt) { if (ptr && port->node->type != SWITCH_NODE) @@ -1090,7 +1090,7 @@ static Port *direct_route_out_MAD(Port * port, ib_dr_path_t * path) } path->p[ptr++] = port->portnum; - DEBUG("routed out to node %s port %d (%p) return path %s\n", + DEBUG("routed out to node %s port %d (%p) return path %s", port->node->nodeid, port->portnum, port, pathstr(0, path)); return port; @@ -1144,7 +1144,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) *dcl = cl; - DEBUG("client id %d, size %d", cl->id, size); + DEBUG("client %d, size %d", cl->id, size); if (size != sizeof(*r)) { IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); return -1; From yevgenyp at mellanox.co.il Thu Jun 5 06:46:39 2008 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Thu, 05 Jun 2008 16:46:39 +0300 Subject: [ofa-general][PATCH 8/12 v1] mlx4: Dynamic port configuration In-Reply-To: References: <480F5026.9070400@mellanox.co.il> Message-ID: <4847EE3F.6090000@mellanox.co.il> Roland Dreier wrote: > > +static ssize_t show_port_type(struct device *dev, > > + struct device_attribute *attr, > > + char *buf) > > +{ > > + struct pci_dev *pdev = to_pci_dev(dev); > > + struct mlx4_dev *mdev = pci_get_drvdata(pdev); > > + int i; > > + > > + sprintf(buf, "Current port types:\n"); > > + for (i = 1; i <= MLX4_MAX_PORTS; i++) { > > + sprintf(buf, "%sPort%d: %s\n", buf, i, > > + (mdev->caps.port_type[i] == MLX4_PORT_TYPE_IB)? > > + "ib": "eth"); > > + } > > + return strlen(buf); > > +} > > This violates the one-value-per-file rule for sysfs if I am reading the > code correctly. > > - R. > The purpose of this sysfs file is to show the port configuration of the device, so I would refer to this as a single value. We could present the configuration as a single, comma separated string. Is it a reasonable solution, or should we have a sysfs file for each port? Thanks, Yevgeny -- Yevgeny Petrilin Mellanox Technalogies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext.7677) Mobile: +972 (54) 783 9222 E-mail: yevgenyp at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From chas at cmf.nrl.navy.mil Thu Jun 5 07:06:25 2008 From: chas at cmf.nrl.navy.mil (Chas Williams (CONTRACTOR)) Date: Thu, 05 Jun 2008 10:06:25 -0400 Subject: [ofa-general] Re: [PATCH] IB/ipoib: increase ring sizes In-Reply-To: <1212654037.31801.13.camel@mtls03> Message-ID: <200806051406.m55E6Qct027411@cmf.nrl.navy.mil> In message <1212654037.31801.13.camel at mtls03>,Eli Cohen writes: >Increase IPoIB ring sizes to twice the original size to act as >a shock absorber for high traffic picks. With the current settings, we >have seen cases that there are many calls to >netif_stop_queue()/netif_wake_queue(). This causes degradation in with a tx ring size of 64, and datagram mode, the 'window size' you will get is approximately 64*1500, or 96k. using the bandwidth delay product, this implies that your interhost delay should be about .096ms to get full rate with tcp (ignoring the basic problems with UD mode). From yevgenyp at mellanox.co.il Thu Jun 5 07:03:05 2008 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Thu, 05 Jun 2008 17:03:05 +0300 Subject: [ofa-general][PATCH 8/12 v1] mlx4: Dynamic port configuration In-Reply-To: References: <480F5026.9070400@mellanox.co.il> Message-ID: <4847F219.5060305@mellanox.co.il> Roland Dreier wrote: > If I read the code correctly, there is no locking around this, so > multiple processes could race and cause all sorts of problems. I will add locking in the "mlx4_change_port_types" function and re-submit the patch. > And also you do the assignment > > > + dev->caps.port_type[i + 1] = port_types[i]; > > before unregistering the device -- so there is a window where > caps.port_type has the wrong data -- not sure if this is a real issue. > If the port type changes, we unregister and then register the device, so I don't think that the temporary wrong data in caps.port_type has any influence. Thanks, Yevgeny -- Yevgeny Petrilin Mellanox Technalogies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext.7677) Mobile: +972 (54) 783 9222 E-mail: yevgenyp at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From andrea at qumranet.com Thu Jun 5 08:36:08 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 05 Jun 2008 17:36:08 +0200 Subject: [ofa-general] [PATCH 1 of 3] list_del_init_rcu In-Reply-To: Message-ID: From: Andrea Arcangeli Introduces list_del_init_rcu and documents it (fixes a comment for list_del_rcu too). Signed-off-by: Andrea Arcangeli Acked-by: Linus Torvalds --- diff -r 86e3ce291996 -r becc195598f8 include/linux/list.h --- a/include/linux/list.h Wed Jun 04 10:35:03 2008 -0700 +++ b/include/linux/list.h Thu Jun 05 17:30:14 2008 +0200 @@ -747,7 +747,7 @@ static inline void hlist_del(struct hlis * or hlist_del_rcu(), running on this same list. * However, it is perfectly legal to run concurrently with * the _rcu list-traversal primitives, such as - * hlist_for_each_entry(). + * hlist_for_each_entry_rcu(). */ static inline void hlist_del_rcu(struct hlist_node *n) { @@ -760,6 +760,34 @@ static inline void hlist_del_init(struct if (!hlist_unhashed(n)) { __hlist_del(n); INIT_HLIST_NODE(n); + } +} + +/** + * hlist_del_init_rcu - deletes entry from hash list with re-initialization + * @n: the element to delete from the hash list. + * + * Note: list_unhashed() on the node return true after this. It is + * useful for RCU based read lockfree traversal if the writer side + * must know if the list entry is still hashed or already unhashed. + * + * In particular, it means that we can not poison the forward pointers + * that may still be used for walking the hash list and we can only + * zero the pprev pointer so list_unhashed() will return true after + * this. + * + * The caller must take whatever precautions are necessary (such as + * holding appropriate locks) to avoid racing with another + * list-mutation primitive, such as hlist_add_head_rcu() or + * hlist_del_rcu(), running on this same list. However, it is + * perfectly legal to run concurrently with the _rcu list-traversal + * primitives, such as hlist_for_each_entry_rcu(). + */ +static inline void hlist_del_init_rcu(struct hlist_node *n) +{ + if (!hlist_unhashed(n)) { + __hlist_del(n); + n->pprev = NULL; } } From andrea at qumranet.com Thu Jun 5 08:36:09 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 05 Jun 2008 17:36:09 +0200 Subject: [ofa-general] [PATCH 2 of 3] mm_take_all_locks In-Reply-To: Message-ID: <082f312bc6821733b1c3.1212680169@duo.random> From: Andrea Arcangeli mm_take_all_locks holds off reclaim from an entire mm_struct. This allows mmu notifiers to register into the mm at any time with the guarantee that no mmu operation is in progress on the mm. This operation locks against the VM for all pte/vma/mm related operations that could ever happen on a certain mm. This includes vmtruncate, try_to_unmap, and all page faults. The caller must take the mmap_sem in write mode before calling mm_take_all_locks(). The caller isn't allowed to release the mmap_sem until mm_drop_all_locks() returns. mmap_sem in write mode is required in order to block all operations that could modify pagetables and free pages without need of altering the vma layout (for example populate_range() with nonlinear vmas). It's also needed in write mode to avoid new anon_vmas to be associated with existing vmas. A single task can't take more than one mm_take_all_locks() in a row or it would deadlock. mm_take_all_locks() and mm_drop_all_locks are expensive operations that may have to take thousand of locks. mm_take_all_locks() can fail if it's interrupted by signals. When mmu_notifier_register returns, we must be sure that the driver is notified if some task is in the middle of a vmtruncate for the 'mm' where the mmu notifier was registered (mmu_notifier_invalidate_range_start/end is run around the vmtruncation but mmu_notifier_register can run after mmu_notifier_invalidate_range_start and before mmu_notifier_invalidate_range_end). Same problem for rmap paths. And we've to remove page pinning to avoid replicating the tlb_gather logic inside KVM (and GRU doesn't work well with page pinning regardless of needing tlb_gather), so without mm_take_all_locks when vmtruncate frees the page, kvm would have no way to notice that it mapped into sptes a page that is going into the freelist without a chance of any further mmu_notifier notification. Signed-off-by: Andrea Arcangeli Acked-by: Linus Torvalds --- diff -r becc195598f8 -r 082f312bc682 include/linux/mm.h --- a/include/linux/mm.h Thu Jun 05 17:30:14 2008 +0200 +++ b/include/linux/mm.h Thu Jun 05 17:30:17 2008 +0200 @@ -1067,6 +1067,9 @@ extern struct vm_area_struct *copy_vma(s unsigned long addr, unsigned long len, pgoff_t pgoff); extern void exit_mmap(struct mm_struct *); +extern int mm_take_all_locks(struct mm_struct *mm); +extern void mm_drop_all_locks(struct mm_struct *mm); + #ifdef CONFIG_PROC_FS /* From fs/proc/base.c. callers must _not_ hold the mm's exe_file_lock */ extern void added_exe_file_vma(struct mm_struct *mm); diff -r becc195598f8 -r 082f312bc682 include/linux/pagemap.h --- a/include/linux/pagemap.h Thu Jun 05 17:30:14 2008 +0200 +++ b/include/linux/pagemap.h Thu Jun 05 17:30:17 2008 +0200 @@ -19,6 +19,7 @@ */ #define AS_EIO (__GFP_BITS_SHIFT + 0) /* IO error on async write */ #define AS_ENOSPC (__GFP_BITS_SHIFT + 1) /* ENOSPC on async write */ +#define AS_MM_ALL_LOCKS (__GFP_BITS_SHIFT + 2) /* under mm_take_all_locks() */ static inline void mapping_set_error(struct address_space *mapping, int error) { diff -r becc195598f8 -r 082f312bc682 include/linux/rmap.h --- a/include/linux/rmap.h Thu Jun 05 17:30:14 2008 +0200 +++ b/include/linux/rmap.h Thu Jun 05 17:30:17 2008 +0200 @@ -26,6 +26,14 @@ */ struct anon_vma { spinlock_t lock; /* Serialize access to vma list */ + /* + * NOTE: the LSB of the head.next is set by + * mm_take_all_locks() _after_ taking the above lock. So the + * head must only be read/written after taking the above lock + * to be sure to see a valid next pointer. The LSB bit itself + * is serialized by a system wide lock only visible to + * mm_take_all_locks() (mm_all_locks_mutex). + */ struct list_head head; /* List of private "related" vmas */ }; diff -r becc195598f8 -r 082f312bc682 mm/mmap.c --- a/mm/mmap.c Thu Jun 05 17:30:14 2008 +0200 +++ b/mm/mmap.c Thu Jun 05 17:30:17 2008 +0200 @@ -2255,3 +2255,161 @@ int install_special_mapping(struct mm_st return 0; } + +static DEFINE_MUTEX(mm_all_locks_mutex); + +static void vm_lock_anon_vma(struct anon_vma *anon_vma) +{ + if (!test_bit(0, (unsigned long *) &anon_vma->head.next)) { + /* + * The LSB of head.next can't change from under us + * because we hold the global_mm_spinlock. + */ + spin_lock(&anon_vma->lock); + /* + * We can safely modify head.next after taking the + * anon_vma->lock. If some other vma in this mm shares + * the same anon_vma we won't take it again. + * + * No need of atomic instructions here, head.next + * can't change from under us thanks to the + * anon_vma->lock. + */ + if (__test_and_set_bit(0, (unsigned long *) + &anon_vma->head.next)) + BUG(); + } +} + +static void vm_lock_mapping(struct address_space *mapping) +{ + if (!test_bit(AS_MM_ALL_LOCKS, &mapping->flags)) { + /* + * AS_MM_ALL_LOCKS can't change from under us because + * we hold the global_mm_spinlock. + * + * Operations on ->flags have to be atomic because + * even if AS_MM_ALL_LOCKS is stable thanks to the + * global_mm_spinlock, there may be other cpus + * changing other bitflags in parallel to us. + */ + if (test_and_set_bit(AS_MM_ALL_LOCKS, &mapping->flags)) + BUG(); + spin_lock(&mapping->i_mmap_lock); + } +} + +/* + * This operation locks against the VM for all pte/vma/mm related + * operations that could ever happen on a certain mm. This includes + * vmtruncate, try_to_unmap, and all page faults. + * + * The caller must take the mmap_sem in write mode before calling + * mm_take_all_locks(). The caller isn't allowed to release the + * mmap_sem until mm_drop_all_locks() returns. + * + * mmap_sem in write mode is required in order to block all operations + * that could modify pagetables and free pages without need of + * altering the vma layout (for example populate_range() with + * nonlinear vmas). It's also needed in write mode to avoid new + * anon_vmas to be associated with existing vmas. + * + * A single task can't take more than one mm_take_all_locks() in a row + * or it would deadlock. + * + * The LSB in anon_vma->head.next and the AS_MM_ALL_LOCKS bitflag in + * mapping->flags avoid to take the same lock twice, if more than one + * vma in this mm is backed by the same anon_vma or address_space. + * + * We can take all the locks in random order because the VM code + * taking i_mmap_lock or anon_vma->lock outside the mmap_sem never + * takes more than one of them in a row. Secondly we're protected + * against a concurrent mm_take_all_locks() by the mm_all_locks_mutex. + * + * mm_take_all_locks() and mm_drop_all_locks are expensive operations + * that may have to take thousand of locks. + * + * mm_take_all_locks() can fail if it's interrupted by signals. + */ +int mm_take_all_locks(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + int ret = -EINTR; + + BUG_ON(down_read_trylock(&mm->mmap_sem)); + + mutex_lock(&mm_all_locks_mutex); + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (signal_pending(current)) + goto out_unlock; + if (vma->anon_vma) + vm_lock_anon_vma(vma->anon_vma); + if (vma->vm_file && vma->vm_file->f_mapping) + vm_lock_mapping(vma->vm_file->f_mapping); + } + ret = 0; + +out_unlock: + if (ret) + mm_drop_all_locks(mm); + + return ret; +} + +static void vm_unlock_anon_vma(struct anon_vma *anon_vma) +{ + if (test_bit(0, (unsigned long *) &anon_vma->head.next)) { + /* + * The LSB of head.next can't change to 0 from under + * us because we hold the global_mm_spinlock. + * + * We must however clear the bitflag before unlocking + * the vma so the users using the anon_vma->head will + * never see our bitflag. + * + * No need of atomic instructions here, head.next + * can't change from under us until we release the + * anon_vma->lock. + */ + if (!__test_and_clear_bit(0, (unsigned long *) + &anon_vma->head.next)) + BUG(); + spin_unlock(&anon_vma->lock); + } +} + +static void vm_unlock_mapping(struct address_space *mapping) +{ + if (test_bit(AS_MM_ALL_LOCKS, &mapping->flags)) { + /* + * AS_MM_ALL_LOCKS can't change to 0 from under us + * because we hold the global_mm_spinlock. + */ + spin_unlock(&mapping->i_mmap_lock); + if (!test_and_clear_bit(AS_MM_ALL_LOCKS, + &mapping->flags)) + BUG(); + } +} + +/* + * The mmap_sem cannot be released by the caller until + * mm_drop_all_locks() returns. + */ +void mm_drop_all_locks(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + + BUG_ON(down_read_trylock(&mm->mmap_sem)); + BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (vma->anon_vma) + vm_unlock_anon_vma(vma->anon_vma); + if (vma->vm_file && vma->vm_file->f_mapping) + vm_unlock_mapping(vma->vm_file->f_mapping); + } + + mutex_unlock(&mm_all_locks_mutex); +} From andrea at qumranet.com Thu Jun 5 08:36:10 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 05 Jun 2008 17:36:10 +0200 Subject: [ofa-general] [PATCH 3 of 3] mmu-notifier-core In-Reply-To: Message-ID: <9cf86778092b94ac8c5b.1212680170@duo.random> From: Andrea Arcangeli With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages. There are secondary MMUs (with secondary sptes and secondary tlbs) too. sptes in the kvm case are shadow pagetables, but when I say spte in mmu-notifier context, I mean "secondary pte". In GRU case there's no actual secondary pte and there's only a secondary tlb because the GRU secondary MMU has no knowledge about sptes and every secondary tlb miss event in the MMU always generates a page fault that has to be resolved by the CPU (this is not the case of KVM where the a secondary tlb miss will walk sptes in hardware and it will refill the secondary tlb transparently to software if the corresponding spte is present). The same way zap_page_range has to invalidate the pte before freeing the page, the spte (and secondary tlb) must also be invalidated before any page is freed and reused. Currently we take a page_count pin on every page mapped by sptes, but that means the pages can't be swapped whenever they're mapped by any spte because they're part of the guest working set. Furthermore a spte unmap event can immediately lead to a page to be freed when the pin is released (so requiring the same complex and relatively slow tlb_gather smp safe logic we have in zap_page_range and that can be avoided completely if the spte unmap event doesn't require an unpin of the page previously mapped in the secondary MMU). The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know when the VM is swapping or freeing or doing anything on the primary MMU so that the secondary MMU code can drop sptes before the pages are freed, avoiding all page pinning and allowing 100% reliable swapping of guest physical address space. Furthermore it avoids the code that teardown the mappings of the secondary MMU, to implement a logic like tlb_gather in zap_page_range that would require many IPI to flush other cpu tlbs, for each fixed number of spte unmapped. To make an example: if what happens on the primary MMU is a protection downgrade (from writeable to wrprotect) the secondary MMU mappings will be invalidated, and the next secondary-mmu-page-fault will call get_user_pages and trigger a do_wp_page through get_user_pages if it called get_user_pages with write=1, and it'll re-establishing an updated spte or secondary-tlb-mapping on the copied page. Or it will setup a readonly spte or readonly tlb mapping if it's a guest-read, if it calls get_user_pages with write=0. This is just an example. This allows to map any page pointed by any pte (and in turn visible in the primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an full MMU with both sptes and secondary-tlb like the shadow-pagetable layer with kvm), or a remote DMA in software like XPMEM (hence needing of schedule in XPMEM code to send the invalidate to the remote node, while no need to schedule in kvm/gru as it's an immediate event like invalidating primary-mmu pte). At least for KVM without this patch it's impossible to swap guests reliably. And having this feature and removing the page pin allows several other optimizations that simplify life considerably. Dependencies: 1) mm_take_all_locks() to register the mmu notifier when the whole VM isn't doing anything with "mm". This allows mmu notifier users to keep track if the VM is in the middle of the invalidate_range_begin/end critical section with an atomic counter incraese in range_begin and decreased in range_end. No secondary MMU page fault is allowed to map any spte or secondary tlb reference, while the VM is in the middle of range_begin/end as any page returned by get_user_pages in that critical section could later immediately be freed without any further ->invalidate_page notification (invalidate_range_begin/end works on ranges and ->invalidate_page isn't called immediately before freeing the page). To stop all page freeing and pagetable overwrites the mmap_sem must be taken in write mode and all other anon_vma/i_mmap locks must be taken too. 2) It'd be a waste to add branches in the VM if nobody could possibly run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of mmu notifiers, but this already allows to compile a KVM external module against a kernel with mmu notifiers enabled and from the next pull from kvm.git we'll start using them. And GRU/XPMEM will also be able to continue the development by enabling KVM=m in their config, until they submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n). This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM are all =n. The mmu_notifier_register call can fail because mm_take_all_locks may be interrupted by a signal and return -EINTR. Because mmu_notifier_reigster is used when a driver startup, a failure can be gracefully handled. Here an example of the change applied to kvm to register the mmu notifiers. Usually when a driver startups other allocations are required anyway and -ENOMEM failure paths exists already. struct kvm *kvm_arch_create_vm(void) { struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL); + int err; if (!kvm) return ERR_PTR(-ENOMEM); INIT_LIST_HEAD(&kvm->arch.active_mmu_pages); + kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops; + err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm); + if (err) { + kfree(kvm); + return ERR_PTR(err); + } + return kvm; } mmu_notifier_unregister returns void and it's reliable. The patch also adds a few needed but missing includes that would prevent kernel to compile after these changes on non-x86 archs (x86 didn't need them by luck). Signed-off-by: Andrea Arcangeli Signed-off-by: Nick Piggin Signed-off-by: Christoph Lameter --- diff -r 082f312bc682 -r 9cf86778092b arch/x86/kvm/Kconfig --- a/arch/x86/kvm/Kconfig Thu Jun 05 17:30:17 2008 +0200 +++ b/arch/x86/kvm/Kconfig Thu Jun 05 17:31:21 2008 +0200 @@ -21,6 +21,7 @@ config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM select PREEMPT_NOTIFIERS + select MMU_NOTIFIER select ANON_INODES ---help--- Support hosting fully virtualized guest machines using hardware diff -r 082f312bc682 -r 9cf86778092b include/linux/mm_types.h --- a/include/linux/mm_types.h Thu Jun 05 17:30:17 2008 +0200 +++ b/include/linux/mm_types.h Thu Jun 05 17:31:21 2008 +0200 @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -244,6 +245,9 @@ struct mm_struct { struct file *exe_file; unsigned long num_exe_file_vmas; #endif +#ifdef CONFIG_MMU_NOTIFIER + struct mmu_notifier_mm *mmu_notifier_mm; +#endif }; #endif /* _LINUX_MM_TYPES_H */ diff -r 082f312bc682 -r 9cf86778092b include/linux/mmu_notifier.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/include/linux/mmu_notifier.h Thu Jun 05 17:31:21 2008 +0200 @@ -0,0 +1,279 @@ +#ifndef _LINUX_MMU_NOTIFIER_H +#define _LINUX_MMU_NOTIFIER_H + +#include +#include +#include + +struct mmu_notifier; +struct mmu_notifier_ops; + +#ifdef CONFIG_MMU_NOTIFIER + +/* + * The mmu notifier_mm structure is allocated and installed in + * mm->mmu_notifier_mm inside the mm_take_all_locks() protected + * critical section and it's released only when mm_count reaches zero + * in mmdrop(). + */ +struct mmu_notifier_mm { + /* all mmu notifiers registerd in this mm are queued in this list */ + struct hlist_head list; + /* to serialize the list modifications and hlist_unhashed */ + spinlock_t lock; +}; + +struct mmu_notifier_ops { + /* + * Called either by mmu_notifier_unregister or when the mm is + * being destroyed by exit_mmap, always before all pages are + * freed. This can run concurrently with other mmu notifier + * methods (the ones invoked outside the mm context) and it + * should tear down all secondary mmu mappings and freeze the + * secondary mmu. If this method isn't implemented you've to + * be sure that nothing could possibly write to the pages + * through the secondary mmu by the time the last thread with + * tsk->mm == mm exits. + * + * As side note: the pages freed after ->release returns could + * be immediately reallocated by the gart at an alias physical + * address with a different cache model, so if ->release isn't + * implemented because all _software_ driven memory accesses + * through the secondary mmu are terminated by the time the + * last thread of this mm quits, you've also to be sure that + * speculative _hardware_ operations can't allocate dirty + * cachelines in the cpu that could not be snooped and made + * coherent with the other read and write operations happening + * through the gart alias address, so leading to memory + * corruption. + */ + void (*release)(struct mmu_notifier *mn, + struct mm_struct *mm); + + /* + * clear_flush_young is called after the VM is + * test-and-clearing the young/accessed bitflag in the + * pte. This way the VM will provide proper aging to the + * accesses to the page through the secondary MMUs and not + * only to the ones through the Linux pte. + */ + int (*clear_flush_young)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long address); + + /* + * Before this is invoked any secondary MMU is still ok to + * read/write to the page previously pointed to by the Linux + * pte because the page hasn't been freed yet and it won't be + * freed until this returns. If required set_page_dirty has to + * be called internally to this method. + */ + void (*invalidate_page)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long address); + + /* + * invalidate_range_start() and invalidate_range_end() must be + * paired and are called only when the mmap_sem and/or the + * locks protecting the reverse maps are held. The subsystem + * must guarantee that no additional references are taken to + * the pages in the range established between the call to + * invalidate_range_start() and the matching call to + * invalidate_range_end(). + * + * Invalidation of multiple concurrent ranges may be + * optionally permitted by the driver. Either way the + * establishment of sptes is forbidden in the range passed to + * invalidate_range_begin/end for the whole duration of the + * invalidate_range_begin/end critical section. + * + * invalidate_range_start() is called when all pages in the + * range are still mapped and have at least a refcount of one. + * + * invalidate_range_end() is called when all pages in the + * range have been unmapped and the pages have been freed by + * the VM. + * + * The VM will remove the page table entries and potentially + * the page between invalidate_range_start() and + * invalidate_range_end(). If the page must not be freed + * because of pending I/O or other circumstances then the + * invalidate_range_start() callback (or the initial mapping + * by the driver) must make sure that the refcount is kept + * elevated. + * + * If the driver increases the refcount when the pages are + * initially mapped into an address space then either + * invalidate_range_start() or invalidate_range_end() may + * decrease the refcount. If the refcount is decreased on + * invalidate_range_start() then the VM can free pages as page + * table entries are removed. If the refcount is only + * droppped on invalidate_range_end() then the driver itself + * will drop the last refcount but it must take care to flush + * any secondary tlb before doing the final free on the + * page. Pages will no longer be referenced by the linux + * address space but may still be referenced by sptes until + * the last refcount is dropped. + */ + void (*invalidate_range_start)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end); + void (*invalidate_range_end)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end); +}; + +/* + * The notifier chains are protected by mmap_sem and/or the reverse map + * semaphores. Notifier chains are only changed when all reverse maps and + * the mmap_sem locks are taken. + * + * Therefore notifier chains can only be traversed when either + * + * 1. mmap_sem is held. + * 2. One of the reverse map locks is held (i_mmap_lock or anon_vma->lock). + * 3. No other concurrent thread can access the list (release) + */ +struct mmu_notifier { + struct hlist_node hlist; + const struct mmu_notifier_ops *ops; +}; + +static inline int mm_has_notifiers(struct mm_struct *mm) +{ + return unlikely(mm->mmu_notifier_mm); +} + +extern int mmu_notifier_register(struct mmu_notifier *mn, + struct mm_struct *mm); +extern int __mmu_notifier_register(struct mmu_notifier *mn, + struct mm_struct *mm); +extern void mmu_notifier_unregister(struct mmu_notifier *mn, + struct mm_struct *mm); +extern void __mmu_notifier_mm_destroy(struct mm_struct *mm); +extern void __mmu_notifier_release(struct mm_struct *mm); +extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address); +extern void __mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address); +extern void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end); +extern void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end); + +static inline void mmu_notifier_release(struct mm_struct *mm) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_release(mm); +} + +static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_flush_young(mm, address); + return 0; +} + +static inline void mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_page(mm, address); +} + +static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_range_start(mm, start, end); +} + +static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_range_end(mm, start, end); +} + +static inline void mmu_notifier_mm_init(struct mm_struct *mm) +{ + mm->mmu_notifier_mm = NULL; +} + +static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_mm_destroy(mm); +} + +/* + * These two macros will sometime replace ptep_clear_flush. + * ptep_clear_flush is impleemnted as macro itself, so this also is + * implemented as a macro until ptep_clear_flush will converted to an + * inline function, to diminish the risk of compilation failure. The + * invalidate_page method over time can be moved outside the PT lock + * and these two macros can be later removed. + */ +#define ptep_clear_flush_notify(__vma, __address, __ptep) \ +({ \ + pte_t __pte; \ + struct vm_area_struct *___vma = __vma; \ + unsigned long ___address = __address; \ + __pte = ptep_clear_flush(___vma, ___address, __ptep); \ + mmu_notifier_invalidate_page(___vma->vm_mm, ___address); \ + __pte; \ +}) + +#define ptep_clear_flush_young_notify(__vma, __address, __ptep) \ +({ \ + int __young; \ + struct vm_area_struct *___vma = __vma; \ + unsigned long ___address = __address; \ + __young = ptep_clear_flush_young(___vma, ___address, __ptep); \ + __young |= mmu_notifier_clear_flush_young(___vma->vm_mm, \ + ___address); \ + __young; \ +}) + +#else /* CONFIG_MMU_NOTIFIER */ + +static inline void mmu_notifier_release(struct mm_struct *mm) +{ +} + +static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address) +{ + return 0; +} + +static inline void mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address) +{ +} + +static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline void mmu_notifier_mm_init(struct mm_struct *mm) +{ +} + +static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) +{ +} + +#define ptep_clear_flush_young_notify ptep_clear_flush_young +#define ptep_clear_flush_notify ptep_clear_flush + +#endif /* CONFIG_MMU_NOTIFIER */ + +#endif /* _LINUX_MMU_NOTIFIER_H */ diff -r 082f312bc682 -r 9cf86778092b kernel/fork.c --- a/kernel/fork.c Thu Jun 05 17:30:17 2008 +0200 +++ b/kernel/fork.c Thu Jun 05 17:31:21 2008 +0200 @@ -54,6 +54,7 @@ #include #include #include +#include #include #include @@ -386,6 +387,7 @@ static struct mm_struct * mm_init(struct if (likely(!mm_alloc_pgd(mm))) { mm->def_flags = 0; + mmu_notifier_mm_init(mm); return mm; } @@ -418,6 +420,7 @@ void __mmdrop(struct mm_struct *mm) BUG_ON(mm == &init_mm); mm_free_pgd(mm); destroy_context(mm); + mmu_notifier_mm_destroy(mm); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); diff -r 082f312bc682 -r 9cf86778092b mm/Kconfig --- a/mm/Kconfig Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/Kconfig Thu Jun 05 17:31:21 2008 +0200 @@ -205,3 +205,6 @@ config VIRT_TO_BUS config VIRT_TO_BUS def_bool y depends on !ARCH_NO_VIRT_TO_BUS + +config MMU_NOTIFIER + bool diff -r 082f312bc682 -r 9cf86778092b mm/Makefile --- a/mm/Makefile Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/Makefile Thu Jun 05 17:31:21 2008 +0200 @@ -33,4 +33,5 @@ obj-$(CONFIG_SMP) += allocpercpu.o obj-$(CONFIG_SMP) += allocpercpu.o obj-$(CONFIG_QUICKLIST) += quicklist.o obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o +obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o diff -r 082f312bc682 -r 9cf86778092b mm/filemap_xip.c --- a/mm/filemap_xip.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/filemap_xip.c Thu Jun 05 17:31:21 2008 +0200 @@ -188,7 +188,7 @@ __xip_unmap (struct address_space * mapp if (pte) { /* Nuke the page table entry. */ flush_cache_page(vma, address, pte_pfn(*pte)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); page_remove_rmap(page, vma); dec_mm_counter(mm, file_rss); BUG_ON(pte_dirty(pteval)); diff -r 082f312bc682 -r 9cf86778092b mm/fremap.c --- a/mm/fremap.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/fremap.c Thu Jun 05 17:31:21 2008 +0200 @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -214,7 +215,9 @@ asmlinkage long sys_remap_file_pages(uns spin_unlock(&mapping->i_mmap_lock); } + mmu_notifier_invalidate_range_start(mm, start, start + size); err = populate_range(mm, vma, start, size, pgoff); + mmu_notifier_invalidate_range_end(mm, start, start + size); if (!err && !(flags & MAP_NONBLOCK)) { if (unlikely(has_write_lock)) { downgrade_write(&mm->mmap_sem); diff -r 082f312bc682 -r 9cf86778092b mm/hugetlb.c --- a/mm/hugetlb.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/hugetlb.c Thu Jun 05 17:31:21 2008 +0200 @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -823,6 +824,7 @@ void __unmap_hugepage_range(struct vm_ar BUG_ON(start & ~HPAGE_MASK); BUG_ON(end & ~HPAGE_MASK); + mmu_notifier_invalidate_range_start(mm, start, end); spin_lock(&mm->page_table_lock); for (address = start; address < end; address += HPAGE_SIZE) { ptep = huge_pte_offset(mm, address); @@ -843,6 +845,7 @@ void __unmap_hugepage_range(struct vm_ar } spin_unlock(&mm->page_table_lock); flush_tlb_range(vma, start, end); + mmu_notifier_invalidate_range_end(mm, start, end); list_for_each_entry_safe(page, tmp, &page_list, lru) { list_del(&page->lru); put_page(page); diff -r 082f312bc682 -r 9cf86778092b mm/memory.c --- a/mm/memory.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/memory.c Thu Jun 05 17:31:21 2008 +0200 @@ -51,6 +51,7 @@ #include #include #include +#include #include #include @@ -649,6 +650,7 @@ int copy_page_range(struct mm_struct *ds unsigned long next; unsigned long addr = vma->vm_start; unsigned long end = vma->vm_end; + int ret; /* * Don't copy ptes where a page fault will fill them correctly. @@ -664,17 +666,33 @@ int copy_page_range(struct mm_struct *ds if (is_vm_hugetlb_page(vma)) return copy_hugetlb_page_range(dst_mm, src_mm, vma); + /* + * We need to invalidate the secondary MMU mappings only when + * there could be a permission downgrade on the ptes of the + * parent mm. And a permission downgrade will only happen if + * is_cow_mapping() returns true. + */ + if (is_cow_mapping(vma->vm_flags)) + mmu_notifier_invalidate_range_start(src_mm, addr, end); + + ret = 0; dst_pgd = pgd_offset(dst_mm, addr); src_pgd = pgd_offset(src_mm, addr); do { next = pgd_addr_end(addr, end); if (pgd_none_or_clear_bad(src_pgd)) continue; - if (copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd, - vma, addr, next)) - return -ENOMEM; + if (unlikely(copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd, + vma, addr, next))) { + ret = -ENOMEM; + break; + } } while (dst_pgd++, src_pgd++, addr = next, addr != end); - return 0; + + if (is_cow_mapping(vma->vm_flags)) + mmu_notifier_invalidate_range_end(src_mm, + vma->vm_start, end); + return ret; } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -878,7 +896,9 @@ unsigned long unmap_vmas(struct mmu_gath unsigned long start = start_addr; spinlock_t *i_mmap_lock = details? details->i_mmap_lock: NULL; int fullmm = (*tlbp)->fullmm; + struct mm_struct *mm = vma->vm_mm; + mmu_notifier_invalidate_range_start(mm, start_addr, end_addr); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) { unsigned long end; @@ -929,6 +949,7 @@ unsigned long unmap_vmas(struct mmu_gath } } out: + mmu_notifier_invalidate_range_end(mm, start_addr, end_addr); return start; /* which is now the end (or restart) address */ } @@ -1561,10 +1582,11 @@ int apply_to_page_range(struct mm_struct { pgd_t *pgd; unsigned long next; - unsigned long end = addr + size; + unsigned long start = addr, end = addr + size; int err; BUG_ON(addr >= end); + mmu_notifier_invalidate_range_start(mm, start, end); pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); @@ -1572,6 +1594,7 @@ int apply_to_page_range(struct mm_struct if (err) break; } while (pgd++, addr = next, addr != end); + mmu_notifier_invalidate_range_end(mm, start, end); return err; } EXPORT_SYMBOL_GPL(apply_to_page_range); @@ -1773,7 +1796,7 @@ gotten: * seen in the presence of one thread doing SMC and another * thread doing COW. */ - ptep_clear_flush(vma, address, page_table); + ptep_clear_flush_notify(vma, address, page_table); set_pte_at(mm, address, page_table, entry); update_mmu_cache(vma, address, entry); lru_cache_add_active(new_page); diff -r 082f312bc682 -r 9cf86778092b mm/mmap.c --- a/mm/mmap.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/mmap.c Thu Jun 05 17:31:21 2008 +0200 @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -2048,6 +2049,7 @@ void exit_mmap(struct mm_struct *mm) /* mm's last user has gone, and its about to be pulled down */ arch_exit_mmap(mm); + mmu_notifier_release(mm); lru_add_drain(); flush_cache_mm(mm); diff -r 082f312bc682 -r 9cf86778092b mm/mmu_notifier.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/mm/mmu_notifier.c Thu Jun 05 17:31:21 2008 +0200 @@ -0,0 +1,276 @@ +/* + * linux/mm/mmu_notifier.c + * + * Copyright (C) 2008 Qumranet, Inc. + * Copyright (C) 2008 SGI + * Christoph Lameter + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include +#include +#include +#include +#include +#include + +/* + * This function can't run concurrently against mmu_notifier_register + * because mm->mm_users > 0 during mmu_notifier_register and exit_mmap + * runs with mm_users == 0. Other tasks may still invoke mmu notifiers + * in parallel despite there being no task using this mm any more, + * through the vmas outside of the exit_mmap context, such as with + * vmtruncate. This serializes against mmu_notifier_unregister with + * the mmu_notifier_mm->lock in addition to RCU and it serializes + * against the other mmu notifiers with RCU. struct mmu_notifier_mm + * can't go away from under us as exit_mmap holds an mm_count pin + * itself. + */ +void __mmu_notifier_release(struct mm_struct *mm) +{ + struct mmu_notifier *mn; + + spin_lock(&mm->mmu_notifier_mm->lock); + while (unlikely(!hlist_empty(&mm->mmu_notifier_mm->list))) { + mn = hlist_entry(mm->mmu_notifier_mm->list.first, + struct mmu_notifier, + hlist); + /* + * We arrived before mmu_notifier_unregister so + * mmu_notifier_unregister will do nothing other than + * to wait ->release to finish and + * mmu_notifier_unregister to return. + */ + hlist_del_init_rcu(&mn->hlist); + /* + * RCU here will block mmu_notifier_unregister until + * ->release returns. + */ + rcu_read_lock(); + spin_unlock(&mm->mmu_notifier_mm->lock); + /* + * if ->release runs before mmu_notifier_unregister it + * must be handled as it's the only way for the driver + * to flush all existing sptes and stop the driver + * from establishing any more sptes before all the + * pages in the mm are freed. + */ + if (mn->ops->release) + mn->ops->release(mn, mm); + rcu_read_unlock(); + spin_lock(&mm->mmu_notifier_mm->lock); + } + spin_unlock(&mm->mmu_notifier_mm->lock); + + /* + * synchronize_rcu here prevents mmu_notifier_release to + * return to exit_mmap (which would proceed freeing all pages + * in the mm) until the ->release method returns, if it was + * invoked by mmu_notifier_unregister. + * + * The mmu_notifier_mm can't go away from under us because one + * mm_count is hold by exit_mmap. + */ + synchronize_rcu(); +} + +/* + * If no young bitflag is supported by the hardware, ->clear_flush_young can + * unmap the address and return 1 or 0 depending if the mapping previously + * existed or not. + */ +int __mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + int young = 0; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->clear_flush_young) + young |= mn->ops->clear_flush_young(mn, mm, address); + } + rcu_read_unlock(); + + return young; +} + +void __mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_page) + mn->ops->invalidate_page(mn, mm, address); + } + rcu_read_unlock(); +} + +void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_range_start) + mn->ops->invalidate_range_start(mn, mm, start, end); + } + rcu_read_unlock(); +} + +void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_range_end) + mn->ops->invalidate_range_end(mn, mm, start, end); + } + rcu_read_unlock(); +} + +static int do_mmu_notifier_register(struct mmu_notifier *mn, + struct mm_struct *mm, + int take_mmap_sem) +{ + struct mmu_notifier_mm * mmu_notifier_mm; + int ret; + + BUG_ON(atomic_read(&mm->mm_users) <= 0); + + ret = -ENOMEM; + mmu_notifier_mm = kmalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL); + if (unlikely(!mmu_notifier_mm)) + goto out; + + if (take_mmap_sem) + down_write(&mm->mmap_sem); + ret = mm_take_all_locks(mm); + if (unlikely(ret)) + goto out_cleanup; + + if (!mm_has_notifiers(mm)) { + INIT_HLIST_HEAD(&mmu_notifier_mm->list); + spin_lock_init(&mmu_notifier_mm->lock); + mm->mmu_notifier_mm = mmu_notifier_mm; + mmu_notifier_mm = NULL; + } + atomic_inc(&mm->mm_count); + + /* + * Serialize the update against mmu_notifier_unregister. A + * side note: mmu_notifier_release can't run concurrently with + * us because we hold the mm_users pin (either implicitly as + * current->mm or explicitly with get_task_mm() or similar). + * We can't race against any other mmu notifier method either + * thanks to mm_take_all_locks(). + */ + spin_lock(&mm->mmu_notifier_mm->lock); + hlist_add_head(&mn->hlist, &mm->mmu_notifier_mm->list); + spin_unlock(&mm->mmu_notifier_mm->lock); + + mm_drop_all_locks(mm); +out_cleanup: + if (take_mmap_sem) + up_write(&mm->mmap_sem); + /* kfree() does nothing if mmu_notifier_mm is NULL */ + kfree(mmu_notifier_mm); +out: + BUG_ON(atomic_read(&mm->mm_users) <= 0); + return ret; +} + +/* + * Must not hold mmap_sem nor any other VM related lock when calling + * this registration function. Must also ensure mm_users can't go down + * to zero while this runs to avoid races with mmu_notifier_release, + * so mm has to be current->mm or the mm should be pinned safely such + * as with get_task_mm(). If the mm is not current->mm, the mm_users + * pin should be released by calling mmput after mmu_notifier_register + * returns. mmu_notifier_unregister must be always called to + * unregister the notifier. mm_count is automatically pinned to allow + * mmu_notifier_unregister to safely run at any time later, before or + * after exit_mmap. ->release will always be called before exit_mmap + * frees the pages. + */ +int mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) +{ + return do_mmu_notifier_register(mn, mm, 1); +} +EXPORT_SYMBOL_GPL(mmu_notifier_register); + +/* + * Same as mmu_notifier_register but here the caller must hold the + * mmap_sem in write mode. + */ +int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) +{ + return do_mmu_notifier_register(mn, mm, 0); +} +EXPORT_SYMBOL_GPL(__mmu_notifier_register); + +/* this is called after the last mmu_notifier_unregister() returned */ +void __mmu_notifier_mm_destroy(struct mm_struct *mm) +{ + BUG_ON(!hlist_empty(&mm->mmu_notifier_mm->list)); + kfree(mm->mmu_notifier_mm); + mm->mmu_notifier_mm = LIST_POISON1; /* debug */ +} + +/* + * This releases the mm_count pin automatically and frees the mm + * structure if it was the last user of it. It serializes against + * running mmu notifiers with RCU and against mmu_notifier_unregister + * with the unregister lock + RCU. All sptes must be dropped before + * calling mmu_notifier_unregister. ->release or any other notifier + * method may be invoked concurrently with mmu_notifier_unregister, + * and only after mmu_notifier_unregister returned we're guaranteed + * that ->release or any other method can't run anymore. + */ +void mmu_notifier_unregister(struct mmu_notifier *mn, struct mm_struct *mm) +{ + BUG_ON(atomic_read(&mm->mm_count) <= 0); + + spin_lock(&mm->mmu_notifier_mm->lock); + if (!hlist_unhashed(&mn->hlist)) { + hlist_del_rcu(&mn->hlist); + + /* + * RCU here will force exit_mmap to wait ->release to finish + * before freeing the pages. + */ + rcu_read_lock(); + spin_unlock(&mm->mmu_notifier_mm->lock); + /* + * exit_mmap will block in mmu_notifier_release to + * guarantee ->release is called before freeing the + * pages. + */ + if (mn->ops->release) + mn->ops->release(mn, mm); + rcu_read_unlock(); + } else + spin_unlock(&mm->mmu_notifier_mm->lock); + + /* + * Wait any running method to finish, of course including + * ->release if it was run by mmu_notifier_relase instead of us. + */ + synchronize_rcu(); + + BUG_ON(atomic_read(&mm->mm_count) <= 0); + + mmdrop(mm); +} +EXPORT_SYMBOL_GPL(mmu_notifier_unregister); diff -r 082f312bc682 -r 9cf86778092b mm/mprotect.c --- a/mm/mprotect.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/mprotect.c Thu Jun 05 17:31:21 2008 +0200 @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -207,10 +208,12 @@ success: dirty_accountable = 1; } + mmu_notifier_invalidate_range_start(mm, start, end); if (is_vm_hugetlb_page(vma)) hugetlb_change_protection(vma, start, end, vma->vm_page_prot); else change_protection(vma, start, end, vma->vm_page_prot, dirty_accountable); + mmu_notifier_invalidate_range_end(mm, start, end); vm_stat_account(mm, oldflags, vma->vm_file, -nrpages); vm_stat_account(mm, newflags, vma->vm_file, nrpages); return 0; diff -r 082f312bc682 -r 9cf86778092b mm/mremap.c --- a/mm/mremap.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/mremap.c Thu Jun 05 17:31:21 2008 +0200 @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -74,7 +75,11 @@ static void move_ptes(struct vm_area_str struct mm_struct *mm = vma->vm_mm; pte_t *old_pte, *new_pte, pte; spinlock_t *old_ptl, *new_ptl; + unsigned long old_start; + old_start = old_addr; + mmu_notifier_invalidate_range_start(vma->vm_mm, + old_start, old_end); if (vma->vm_file) { /* * Subtle point from Rajesh Venkatasubramanian: before @@ -116,6 +121,7 @@ static void move_ptes(struct vm_area_str pte_unmap_unlock(old_pte - 1, old_ptl); if (mapping) spin_unlock(&mapping->i_mmap_lock); + mmu_notifier_invalidate_range_end(vma->vm_mm, old_start, old_end); } #define LATENCY_LIMIT (64 * PAGE_SIZE) diff -r 082f312bc682 -r 9cf86778092b mm/rmap.c --- a/mm/rmap.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/rmap.c Thu Jun 05 17:31:21 2008 +0200 @@ -49,6 +49,7 @@ #include #include #include +#include #include @@ -287,7 +288,7 @@ static int page_referenced_one(struct pa if (vma->vm_flags & VM_LOCKED) { referenced++; *mapcount = 1; /* break early from loop */ - } else if (ptep_clear_flush_young(vma, address, pte)) + } else if (ptep_clear_flush_young_notify(vma, address, pte)) referenced++; /* Pretend the page is referenced if the task has the @@ -457,7 +458,7 @@ static int page_mkclean_one(struct page pte_t entry; flush_cache_page(vma, address, pte_pfn(*pte)); - entry = ptep_clear_flush(vma, address, pte); + entry = ptep_clear_flush_notify(vma, address, pte); entry = pte_wrprotect(entry); entry = pte_mkclean(entry); set_pte_at(mm, address, pte, entry); @@ -717,14 +718,14 @@ static int try_to_unmap_one(struct page * skipped over this mm) then we should reactivate it. */ if (!migration && ((vma->vm_flags & VM_LOCKED) || - (ptep_clear_flush_young(vma, address, pte)))) { + (ptep_clear_flush_young_notify(vma, address, pte)))) { ret = SWAP_FAIL; goto out_unmap; } /* Nuke the page table entry. */ flush_cache_page(vma, address, page_to_pfn(page)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); /* Move the dirty bit to the physical page now the pte is gone. */ if (pte_dirty(pteval)) @@ -849,12 +850,12 @@ static void try_to_unmap_cluster(unsigne page = vm_normal_page(vma, address, *pte); BUG_ON(!page || PageAnon(page)); - if (ptep_clear_flush_young(vma, address, pte)) + if (ptep_clear_flush_young_notify(vma, address, pte)) continue; /* Nuke the page table entry. */ flush_cache_page(vma, address, pte_pfn(*pte)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); /* If nonlinear, store the file page offset in the pte. */ if (page->index != linear_page_index(vma, address)) From andrea at qumranet.com Thu Jun 5 08:36:07 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 05 Jun 2008 17:36:07 +0200 Subject: [ofa-general] [PATCH 0 of 3] mmu notifier v18 Message-ID: This splits the list.h changes in patch number 1, mm_take_all_locks patch 2, and mmu-notifier-core becomes patch 3 as requested. mm_take/drop_all_locks have been cleaned up as requested to move the bitflips in proper methods outside the main vma loop. From rdreier at cisco.com Thu Jun 5 09:35:44 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 05 Jun 2008 09:35:44 -0700 Subject: [ofa-general][PATCH 8/12 v1] mlx4: Dynamic port configuration In-Reply-To: <4847EE3F.6090000@mellanox.co.il> (Yevgeny Petrilin's message of "Thu, 05 Jun 2008 16:46:39 +0300") References: <480F5026.9070400@mellanox.co.il> <4847EE3F.6090000@mellanox.co.il> Message-ID: > The purpose of this sysfs file is to show the port configuration of the device, > so I would refer to this as a single value. > We could present the configuration as a single, comma separated string. > Is it a reasonable solution, or should we have a sysfs file for each port? The rule of one value per file is strict. So yes, one sysfs file per port. From rdreier at cisco.com Thu Jun 5 09:36:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 05 Jun 2008 09:36:21 -0700 Subject: [ofa-general][PATCH 8/12 v1] mlx4: Dynamic port configuration In-Reply-To: <4847F219.5060305@mellanox.co.il> (Yevgeny Petrilin's message of "Thu, 05 Jun 2008 17:03:05 +0300") References: <480F5026.9070400@mellanox.co.il> <4847F219.5060305@mellanox.co.il> Message-ID: > If the port type changes, we unregister and then register the device, so I don't think > that the temporary wrong data in caps.port_type has any influence. Are there any other places that look at caps.port_type? Because if so there is a window where they will see wrong data. - R. From rdreier at cisco.com Thu Jun 5 09:38:36 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 05 Jun 2008 09:38:36 -0700 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <20080605054035.GA8059@ornl.gov> (Dave Dillow's message of "Thu, 5 Jun 2008 01:40:35 -0400") References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> <20080605054035.GA8059@ornl.gov> Message-ID: > That said, given that SRP's been using sysfs since it went in, is there > a reason to move to configfs other than it's the new preferred way to do > it? Given the desire to not break ABI's -- and IIRC sysfs was declared to > be under that unbrella -- wouldn't we have to at least carry both > interfaces for a while, assuming we can even get rid of the sysfs one? Yes, we'd definitely be carrying both interfaces for at least a year. Looking further into this, I'm not sure it makes much sense either. Another problem with configfs is that the lifetime of the object is controlled by userspace. So if we lose a connection to a target, the object will persist in configfs until userspace notices. - R. From andrea at qumranet.com Thu Jun 5 09:44:21 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 5 Jun 2008 18:44:21 +0200 Subject: [ofa-general] Re: [PATCH 2 of 3] mm_take_all_locks In-Reply-To: References: <082f312bc6821733b1c3.1212680169@duo.random> Message-ID: <20080605164421.GG15502@duo.random> On Thu, Jun 05, 2008 at 09:15:41AM -0700, Linus Torvalds wrote: > > Just a small comment fix. > > On Thu, 5 Jun 2008, Andrea Arcangeli wrote: > > + /* > > + * AS_MM_ALL_LOCKS can't change from under us because > > + * we hold the global_mm_spinlock. > > There's no global_mm_spinlock, you mean the 'mm_all_locks_mutex'. > > (There was at least one other case where you had that comment issue). From: Andrea Arcangeli Indeed, I meant mm_all_locks_mutex, this will fix it, or if you prefer a resubmit of the 2/3 let me know. Thanks! Signed-off-by: Andrea Arcangeli --- diff -r 082f312bc682 mm/mmap.c --- a/mm/mmap.c Thu Jun 05 17:30:17 2008 +0200 +++ b/mm/mmap.c Thu Jun 05 18:40:23 2008 +0200 @@ -2263,7 +2263,7 @@ static void vm_lock_anon_vma(struct anon if (!test_bit(0, (unsigned long *) &anon_vma->head.next)) { /* * The LSB of head.next can't change from under us - * because we hold the global_mm_spinlock. + * because we hold the mm_all_locks_mutex. */ spin_lock(&anon_vma->lock); /* @@ -2286,11 +2286,11 @@ static void vm_lock_mapping(struct addre if (!test_bit(AS_MM_ALL_LOCKS, &mapping->flags)) { /* * AS_MM_ALL_LOCKS can't change from under us because - * we hold the global_mm_spinlock. + * we hold the mm_all_locks_mutex. * * Operations on ->flags have to be atomic because * even if AS_MM_ALL_LOCKS is stable thanks to the - * global_mm_spinlock, there may be other cpus + * mm_all_locks_mutex, there may be other cpus * changing other bitflags in parallel to us. */ if (test_and_set_bit(AS_MM_ALL_LOCKS, &mapping->flags)) @@ -2362,7 +2362,7 @@ static void vm_unlock_anon_vma(struct an if (test_bit(0, (unsigned long *) &anon_vma->head.next)) { /* * The LSB of head.next can't change to 0 from under - * us because we hold the global_mm_spinlock. + * us because we hold the mm_all_locks_mutex. * * We must however clear the bitflag before unlocking * the vma so the users using the anon_vma->head will @@ -2384,7 +2384,7 @@ static void vm_unlock_mapping(struct add if (test_bit(AS_MM_ALL_LOCKS, &mapping->flags)) { /* * AS_MM_ALL_LOCKS can't change to 0 from under us - * because we hold the global_mm_spinlock. + * because we hold the mm_all_locks_mutex. */ spin_unlock(&mapping->i_mmap_lock); if (!test_and_clear_bit(AS_MM_ALL_LOCKS, From torvalds at linux-foundation.org Thu Jun 5 09:15:41 2008 From: torvalds at linux-foundation.org (Linus Torvalds) Date: Thu, 5 Jun 2008 09:15:41 -0700 (PDT) Subject: [ofa-general] Re: [PATCH 2 of 3] mm_take_all_locks In-Reply-To: <082f312bc6821733b1c3.1212680169@duo.random> References: <082f312bc6821733b1c3.1212680169@duo.random> Message-ID: Just a small comment fix. On Thu, 5 Jun 2008, Andrea Arcangeli wrote: > + /* > + * AS_MM_ALL_LOCKS can't change from under us because > + * we hold the global_mm_spinlock. There's no global_mm_spinlock, you mean the 'mm_all_locks_mutex'. (There was at least one other case where you had that comment issue). Linus From andrea at qumranet.com Thu Jun 5 09:51:04 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 5 Jun 2008 18:51:04 +0200 Subject: [ofa-general] Re: [PATCH 001/001] mmu-notifier-core v17 In-Reply-To: <20080516190752.GK11333@linux.vnet.ibm.com> References: <20080509193230.GH7710@duo.random> <20080516190752.GK11333@linux.vnet.ibm.com> Message-ID: <20080605165104.GI15502@duo.random> On Fri, May 16, 2008 at 12:07:52PM -0700, Paul E. McKenney wrote: > The hlist_del_init_rcu() primitive looks good. > > The rest of the RCU code looks fine assuming that "mn->ops->release()" > either does call_rcu() to defer actual removal, or that the actual > removal is deferred until after mmu_notifier_release() returns. Yes, actual removal is deferred until after mmu_notifier_release() returns. > Acked-by: Paul E. McKenney Thanks for the review Paul! I should also have added your precious Acked-by to the 1/3 and 3/3 but the important is the ack by email ;) From ydvwl at bowltracker.com Thu Jun 5 10:00:47 2008 From: ydvwl at bowltracker.com (Erik Godwin) Date: Thu, 5 Jun 2008 19:00:47 +0200 Subject: [ofa-general] Blue sexy pill - $0.69 Message-ID: <01c8c73e$767f1980$f34abf4e@ydvwl> Blue sexy pill - $0.69Visit us -------------- next part -------------- An HTML attachment was scrubbed... URL: From shemminger at vyatta.com Thu Jun 5 10:01:03 2008 From: shemminger at vyatta.com (Stephen Hemminger) Date: Thu, 5 Jun 2008 10:01:03 -0700 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> <20080605054035.GA8059@ornl.gov> Message-ID: <20080605100103.38ce7c4b@extreme> On Thu, 05 Jun 2008 09:38:36 -0700 Roland Dreier wrote: > > That said, given that SRP's been using sysfs since it went in, is there > > a reason to move to configfs other than it's the new preferred way to do > > it? Given the desire to not break ABI's -- and IIRC sysfs was declared to > > be under that unbrella -- wouldn't we have to at least carry both > > interfaces for a while, assuming we can even get rid of the sysfs one? > > Yes, we'd definitely be carrying both interfaces for at least a year. > > Looking further into this, I'm not sure it makes much sense either. > Another problem with configfs is that the lifetime of the object is > controlled by userspace. So if we lose a connection to a target, > the object will persist in configfs until userspace notices. > > - R. There is nothing stopping adding a well designed alternate interface. Either netlink or ioctl's are okay. As long as it is 32/64 bit clean. From orenk at dev.mellanox.co.il Thu Jun 5 10:23:43 2008 From: orenk at dev.mellanox.co.il (Oren Kladnitsky) Date: Thu, 05 Jun 2008 20:23:43 +0300 Subject: [ofa-general] Re: [PATCH] ibutils/ibis/ibis.c: Update for change to osm_mad_pool_init API In-Reply-To: <1212501425.31387.46.camel@hrosenstock-ws.xsigo.com> References: <1212501425.31387.46.camel@hrosenstock-ws.xsigo.com> Message-ID: <4848211F.7050107@dev.mellanox.co.il> Hal Rosenstock wrote: > ibutils/ibis/ibis.c: Update for change to osm_mad_pool_init API > > Signed-off-by: Hal Rosenstock > > diff --git a/ibis/src/ibis.c b/ibis/src/ibis.c > index 0684b32..2864f64 100644 > --- a/ibis/src/ibis.c > +++ b/ibis/src/ibis.c > @@ -120,9 +120,7 @@ ibis_init( > > IbisObj.p_opt = p_opt; > > - status = osm_mad_pool_init( > - &(IbisObj.mad_pool), > - &(IbisObj.log) ); > + status = osm_mad_pool_init( &(IbisObj.mad_pool) ); > if( status != IB_SUCCESS ) > { > osm_log( &IbisObj.log, OSM_LOG_ERROR, > > > done. From kaber at trash.net Thu Jun 5 10:32:59 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 05 Jun 2008 19:32:59 +0200 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <20080605100103.38ce7c4b@extreme> References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> <20080605054035.GA8059@ornl.gov> <20080605100103.38ce7c4b@extreme> Message-ID: <4848234B.1080200@trash.net> Stephen Hemminger wrote: > On Thu, 05 Jun 2008 09:38:36 -0700 > Roland Dreier wrote: > >> > That said, given that SRP's been using sysfs since it went in, is there >> > a reason to move to configfs other than it's the new preferred way to do >> > it? Given the desire to not break ABI's -- and IIRC sysfs was declared to >> > be under that unbrella -- wouldn't we have to at least carry both >> > interfaces for a while, assuming we can even get rid of the sysfs one? >> >> Yes, we'd definitely be carrying both interfaces for at least a year. >> >> Looking further into this, I'm not sure it makes much sense either. >> Another problem with configfs is that the lifetime of the object is >> controlled by userspace. So if we lose a connection to a target, >> the object will persist in configfs until userspace notices. >> >> - R. > > There is nothing stopping adding a well designed alternate interface. > Either netlink or ioctl's are okay. As long as it is 32/64 bit clean. From a quick look it seems it should use rtnl_link instead of adding yet another private sysfs interface. From dillowda at ornl.gov Thu Jun 5 10:54:46 2008 From: dillowda at ornl.gov (David Dillow) Date: Thu, 05 Jun 2008 13:54:46 -0400 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> <20080605054035.GA8059@ornl.gov> Message-ID: <1212688486.1173.5.camel@lap75545.ornl.gov> On Thu, 2008-06-05 at 09:38 -0700, Roland Dreier wrote: > > That said, given that SRP's been using sysfs since it went in, is there > > a reason to move to configfs other than it's the new preferred way to do > > it? Given the desire to not break ABI's -- and IIRC sysfs was declared to > > be under that unbrella -- wouldn't we have to at least carry both > > interfaces for a while, assuming we can even get rid of the sysfs one? > > Yes, we'd definitely be carrying both interfaces for at least a year. > > Looking further into this, I'm not sure it makes much sense either. > Another problem with configfs is that the lifetime of the object is > controlled by userspace. So if we lose a connection to a target, > the object will persist in configfs until userspace notices. Yep. Though I'd like to be able to have a persistent connection and retry capability in the kernel, so that I don't need to have srp_daemon, and the persistence would be fine in that case. I've not fully convinced myself that doing a persistent connection in the kernel is a good idea, but a deep background task is trying to do the OFED patches to the initiator in a way that is acceptable for inclusion in mainline, and they would easily fall out of that work. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From hrosenstock at xsigo.com Thu Jun 5 11:14:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 11:14:38 -0700 Subject: [ofa-general] [PATCH] libibmad/fields.c: _set_field64 sets field in network rather than host order Message-ID: <1212689678.6127.150.camel@hrosenstock-ws.xsigo.com> libibmad/fields.c: _set_field64 sets field in network rather than host order Fix issue introduced by commit 608cddb34fda942b37d0f2bb2e7889e22ac046f8 Signed-off-by: Hal Rosenstock diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c index c93a9d4..6942e85 100644 --- a/libibmad/src/fields.c +++ b/libibmad/src/fields.c @@ -366,7 +366,10 @@ ib_field_t ib_mad_f [] = { void _set_field64(void *buf, int base_offs, ib_field_t *f, uint64_t val) { - memcpy((char *)buf + base_offs + f->bitoffs / 8, &val, sizeof(uint64_t)); + uint64_t nval; + + nval = htonll(val); + memcpy((char *)buf + base_offs + f->bitoffs / 8, &nval, sizeof(uint64_t)); } uint64_t From hrosenstock at xsigo.com Thu Jun 5 11:14:46 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 11:14:46 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/README: Cosmetic commentary changes Message-ID: <1212689686.6127.151.camel@hrosenstock-ws.xsigo.com> ibsim/README: Cosmetic commentary changes Signed-off-by: Hal Rosenstock diff --git a/README b/README index b7615aa..7159543 100644 --- a/README +++ b/README @@ -33,19 +33,19 @@ SM/SA/PerfMgr to ibsim. Any libibumad based application may work with ibsim. Kernel support and userspace application recompilation are not required. -ibsim may works locally via unix sockets or remotely via inet sockets -(not implemented yet). +ibsim works locally via unix sockets or remotely via inet sockets +(latter not implemented yet). -As fabric topology description ibsim uses text file in the format +For fabric topology description, ibsim uses a text file in the format compatible with ibnetdiscover command output (for examples see -net-examples/) and it can be generated using real cluster snapshot. +net-examples/) and it can be generated using a real cluster snapshot. -ibsim has simple console command interface and can simulate random +ibsim has a simple console command interface and can simulate random packets drops and link up/down events. It is possible to run batch commands from file via pipe or named fifo. -Building and using ibsim. +Building and using ibsim ------------------------ 1. cd to unpacked simulator directory. From hrosenstock at xsigo.com Thu Jun 5 12:51:49 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 12:51:49 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found Message-ID: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found ERR: cannot open port_guids.list: No such file or directory Signed-off-by: Hal Rosenstock diff --git a/infiniband-diags/src/mcm_rereg_test.c b/infiniband-diags/src/mcm_rereg_test.c index 4f3ccc1..2bdf9dc 100644 --- a/infiniband-diags/src/mcm_rereg_test.c +++ b/infiniband-diags/src/mcm_rereg_test.c @@ -397,6 +397,11 @@ static int rereg_and_test_port(char *guid_file, int port, int agent, ib_portid_t } f = fopen(guid_file, "r"); + if (!f) { + err("cannot open %s: %s\n", guid_file, strerror(errno)); + return -1; + } + while (fgets(line, sizeof(line), f)) { guid = strtoull(line, NULL, 0); guid = htonll(guid); From hrosenstock at xsigo.com Thu Jun 5 12:51:53 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 12:51:53 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/ibsendtrap.c: Support CA and port num Message-ID: <1212695514.6127.162.camel@hrosenstock-ws.xsigo.com> infiniband-diags/ibsendtrap.c: Support CA and port num Also, fix usage Signed-off-by: Hal Rosenstock diff --git a/infiniband-diags/src/ibsendtrap.c b/infiniband-diags/src/ibsendtrap.c index c85fed9..66620de 100644 --- a/infiniband-diags/src/ibsendtrap.c +++ b/infiniband-diags/src/ibsendtrap.c @@ -48,8 +48,6 @@ #include "ibdiag_common.h" char *argv0 = ""; -char *sa_hca_name = NULL; -uint32_t sa_port_num = 0; static int send_144_node_desc_update(void) { @@ -99,12 +97,10 @@ trap_def_t traps[2] = { static void usage(void) { - int i = 0; + int i; + fprintf(stderr, "Usage: %s [-hV]" - "[-C -P ] \n", argv0); - fprintf(stderr, " Queries node records by default\n"); - fprintf(stderr, " -C specify the SA query HCA\n"); - fprintf(stderr, " -P specify the SA query port\n"); + " [-C ] [-P ] []\n", argv0); fprintf(stderr, " -V print version\n"); fprintf(stderr, " can be one of the following\n"); for (i = 0; traps[i].trap_name; i++) { @@ -118,7 +114,8 @@ static void usage(void) int send_trap(char *trap_name) { - int i = 0; + int i; + for (i = 0; traps[i].trap_name; i++) { if (strcmp(traps[i].trap_name, trap_name) == 0) { return (traps[i].send_func()); @@ -133,6 +130,8 @@ int main(int argc, char **argv) int mgmt_classes[2] = { IB_SMI_CLASS, IB_SMI_DIRECT_CLASS }; int ch = 0; char *trap_name = NULL; + char *ca = NULL; + int ca_port = 0; static char const str_opts[] = "hVP:C:"; static const struct option long_opts[] = { @@ -151,10 +150,10 @@ int main(int argc, char **argv) fprintf(stderr, "%s %s\n", argv0, get_build_version()); exit(-1); case 'C': - sa_hca_name = optarg; + ca = optarg; break; case 'P': - sa_port_num = strtoul(optarg, NULL, 0); + ca_port = strtoul(optarg, NULL, 0); break; case 'h': default: @@ -171,7 +170,7 @@ int main(int argc, char **argv) } madrpc_show_errors(1); - madrpc_init(NULL, 0, mgmt_classes, 2); + madrpc_init(ca, ca_port, mgmt_classes, 2); return (send_trap(trap_name)); } From rdreier at cisco.com Thu Jun 5 13:47:39 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 05 Jun 2008 13:47:39 -0700 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <4848234B.1080200@trash.net> (Patrick McHardy's message of "Thu, 05 Jun 2008 19:32:59 +0200") References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> <20080605054035.GA8059@ornl.gov> <20080605100103.38ce7c4b@extreme> <4848234B.1080200@trash.net> Message-ID: > > There is nothing stopping adding a well designed alternate interface. > > Either netlink or ioctl's are okay. As long as it is 32/64 bit clean. > From a quick look it seems it should use rtnl_link instead > of adding yet another private sysfs interface. This seems crazy -- why do we want to push things towards the complexity of rtnl_link instead of a simple text-based interface that can be trivial used with 'echo' from the shell? What advantages does that give? - R. From twbowman at gmail.com Thu Jun 5 14:05:19 2008 From: twbowman at gmail.com (Todd Bowman) Date: Thu, 5 Jun 2008 15:05:19 -0600 Subject: [ofa-general] IPoIB arp relation to the SM Message-ID: All, I am trying to have a better understanding of how the SM and IPoIB interact with respect to arp. If there is a document that describes this please point me in the right direction. In the past, (OFED 1.0 and pre OFED 1.0) I observed the the following: 1) Using hosts A,B A> ping B (succeeded) A> opensm stop A> arp -d B A> ping B (Failed) 2) Using opensm-3.1 A> ping B (succeeded) A> opensm stop A> arp -d B A> ping B (succeeded) A> ping C (failed) In scenario 2, what allows the arp query for B to be resolved after the SM is stopped and arp entry cleared? Is this due to connected mode, in that an RD connection is already established? Or is there an underlying cache that feeds the ARP cache? Thanks, Todd -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Thu Jun 5 14:18:22 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 05 Jun 2008 14:18:22 -0700 Subject: [ofa-general] IPoIB arp relation to the SM In-Reply-To: References: Message-ID: <1212700702.6127.181.camel@hrosenstock-ws.xsigo.com> Todd, On Thu, 2008-06-05 at 15:05 -0600, Todd Bowman wrote: > All, > > > I am trying to have a better understanding of how the SM and IPoIB > interact with respect to arp. If there is a document that describes > this please point me in the right direction. > > In the past, (OFED 1.0 and pre OFED 1.0) I observed the the following: > > 1) Using hosts A,B > > A> ping B (succeeded) > A> opensm stop > A> arp -d B > A> ping B (Failed) > > > 2) Using opensm-3.1 > > A> ping B (succeeded) > A> opensm stop > A> arp -d B > A> ping B (succeeded) > A> ping C (failed) > > > > In scenario 2, what allows the arp query for B to be resolved after > the SM is stopped and arp entry cleared? > Is this due to connected mode, in that an RD connection is already > established? > Or is there an underlying cache that feeds the ARP cache? There are two things here (assuming nodes A and B have already joined the IPoIB broadcast group prior to these scenarios). The end nodes first need to get PathRecords from the SA in order to be able to unicast prior to being able to ARP. If it is using broadcast, that information can also be obtained via MCMemberRecord from the SA but not sure that is used. ARP requests are broadcast and I think that ARP replies can be unicast or broadcast depending on the type. The point here is that the ARP/ping can work if the IPoIB broadcast group is complete enough for the nodes in question and neither end node needs any PathRecord information for the destination. I don't think there has been any SM changes which affect this but I do think there have been changes/optimizations in the end nodes (based on kernel or OFED version) as to when the PathRecords information is saved/needed but don't recall all the details. Maybe others on the list can elaborate on this. Hope this helps. -- Hal > Thanks, > Todd > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From kaber at trash.net Thu Jun 5 15:54:49 2008 From: kaber at trash.net (Patrick McHardy) Date: Fri, 06 Jun 2008 00:54:49 +0200 Subject: [ofa-general] Re: [PATCH v3 08/13] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: References: <20080529095126.9943.84692.stgit@localhost.localdomain> <20080529095754.9943.27936.stgit@localhost.localdomain> <20080529103003.010c4a08@extreme> <20080529174805.GA10903@kroah.com> <1212518703.2217.22.camel@lap75545.ornl.gov> <20080605054035.GA8059@ornl.gov> <20080605100103.38ce7c4b@extreme> <4848234B.1080200@trash.net> Message-ID: <48486EB9.4010706@trash.net> Roland Dreier wrote: > > > There is nothing stopping adding a well designed alternate interface. > > > Either netlink or ioctl's are okay. As long as it is 32/64 bit clean. > > > From a quick look it seems it should use rtnl_link instead > > of adding yet another private sysfs interface. > > This seems crazy -- why do we want to push things towards the complexity > of rtnl_link instead of a simple text-based interface that can be > trivial used with 'echo' from the shell? Its not complicated at all. Ability to use "echo" seems like a pretty poor argument to me. > What advantages does that give? - atomic configuration changes - atomic configuration dumps - configuration change notifications - same interface as for all other network-related configurations - easily extensible - not yet another private driver interface - not text based, which makes it a lot more convenient for using in programs From weiny2 at llnl.gov Thu Jun 5 19:12:51 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 5 Jun 2008 19:12:51 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. Message-ID: <20080605191251.6882c3d6.weiny2@llnl.gov> I noticed over the last couple days that ibnetdiscover would print the incorrect speed on the port I would run ibnetdiscover from. For example: Switch 24 "S-0008f10400411f56" # "SW1 wopr ISR9024D (MLX4 FW)" base port 0 lid 11 lmc 0 ... [13] "H-0002c90200219e64"[1](2c90200219e65) # "wopri" lid 32 4xSDR ^^^ (Note from the switch side of things it thinks the speed is SDR.) ... Ca 2 "H-0002c90200219e64" # "wopri" [1](2c90200219e65) "S-0008f10400411f56"[13] # lid 32 lmc 0 "SW1 wopr ISR9024D (MLX4 FW)" lid 11 4xDDR ^^^ (but here DDR.) It turns out that when you first discover a switch the port object created gets the PortInfo of Port "0". This patch queries again for the PortInfo of the port we "came in on". Ira >From 5bc66af276c7baabd4d66be9df0379271cb625b4 Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Thu, 5 Jun 2008 19:03:44 -0700 Subject: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. Previously the portinfo data for that port object would be the PortInfo of port "0" of the switch. Specifically this would cause the speed of the link to be printed incorrectly. Signed-off-by: Ira K. Weiny --- infiniband-diags/src/ibnetdiscover.c | 33 +++++++++++++++++++-------------- 1 files changed, 19 insertions(+), 14 deletions(-) diff --git a/infiniband-diags/src/ibnetdiscover.c b/infiniband-diags/src/ibnetdiscover.c index b8d4e92..20da1ea 100644 --- a/infiniband-diags/src/ibnetdiscover.c +++ b/infiniband-diags/src/ibnetdiscover.c @@ -129,6 +129,18 @@ node_type_str2(Node *node) return "??"; } +void +decode_port_info(void *pi, Port *port) +{ + mad_decode_field(pi, IB_PORT_LID_F, &port->lid); + mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); + mad_decode_field(pi, IB_PORT_STATE_F, &port->state); + mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); + mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); + mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); +} + + int get_port(Port *port, int portnum, ib_portid_t *portid) { @@ -139,13 +151,7 @@ get_port(Port *port, int portnum, ib_portid_t *portid) if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout)) return -1; - - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); + decode_port_info(pi, port); DEBUG("portid %s portnum %d: lid %d state %d physstate %d %s %s", portid2str(portid), portnum, port->lid, port->state, port->physstate, get_linkwidth_str(port->linkwidth), get_linkspeed_str(port->linkspeed)); @@ -181,13 +187,7 @@ get_node(Node *node, Port *port, ib_portid_t *portid) if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, 0, timeout)) return -1; - - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); + decode_port_info(pi, port); if (node->type != SWITCH_NODE) return 0; @@ -195,6 +195,11 @@ get_node(Node *node, Port *port, ib_portid_t *portid) node->smalid = port->lid; node->smalmc = port->lmc; + /* after we have the sma information find out the real PortInfo for this port */ + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->localport, timeout)) + return -1; + decode_port_info(pi, port); + if (!smp_query(si, portid, IB_ATTR_SWITCH_INFO, 0, timeout)) node->smaenhsp0 = 0; /* assume base SP0 */ else -- 1.5.1 From hrosenstock at xsigo.com Fri Jun 6 04:13:41 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 06 Jun 2008 04:13:41 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. In-Reply-To: <20080605191251.6882c3d6.weiny2@llnl.gov> References: <20080605191251.6882c3d6.weiny2@llnl.gov> Message-ID: <1212750821.6127.217.camel@hrosenstock-ws.xsigo.com> Hi Ira, On Thu, 2008-06-05 at 19:12 -0700, Ira Weiny wrote: > I noticed over the last couple days that ibnetdiscover would print the > incorrect speed on the port I would run ibnetdiscover from. For example: > > Switch 24 "S-0008f10400411f56" # "SW1 wopr ISR9024D (MLX4 FW)" base port 0 lid 11 lmc 0 > ... > [13] "H-0002c90200219e64"[1](2c90200219e65) # "wopri" lid 32 4xSDR > > ^^^ > (Note from the switch side of things it thinks the speed is SDR.) > ... > Ca 2 "H-0002c90200219e64" # "wopri" > [1](2c90200219e65) "S-0008f10400411f56"[13] # lid 32 lmc 0 "SW1 wopr ISR9024D (MLX4 FW)" lid 11 4xDDR > > ^^^ > (but here DDR.) Yes, that is clearly wrong. Good find. > It turns out that when you first discover a switch the port object created gets > the PortInfo of Port "0". Right; this trick works for peer CA and router ports but not switch ports where the actual port number is needed. > This patch queries again for the PortInfo of the port we "came in on". > > Ira > > > From 5bc66af276c7baabd4d66be9df0379271cb625b4 Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Thu, 5 Jun 2008 19:03:44 -0700 > Subject: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. > > Previously the portinfo data for that port object would be the PortInfo of > port "0" of the switch. Specifically this would cause the speed of the link > to be printed incorrectly. Did you try back to back CAs too ? Nit: this looks like 2 patches to me: 1. Factor out redundant code by adding decode_port_info routine 2. Fix link speed of local CA in switch display > Signed-off-by: Ira K. Weiny > --- > infiniband-diags/src/ibnetdiscover.c | 33 +++++++++++++++++++-------------- > 1 files changed, 19 insertions(+), 14 deletions(-) > > diff --git a/infiniband-diags/src/ibnetdiscover.c b/infiniband-diags/src/ibnetdiscover.c > index b8d4e92..20da1ea 100644 > --- a/infiniband-diags/src/ibnetdiscover.c > +++ b/infiniband-diags/src/ibnetdiscover.c > @@ -129,6 +129,18 @@ node_type_str2(Node *node) > return "??"; > } > > +void > +decode_port_info(void *pi, Port *port) > +{ > + mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > + mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > + mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > + mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > + mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > + mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > +} > + > + > int > get_port(Port *port, int portnum, ib_portid_t *portid) > { > @@ -139,13 +151,7 @@ get_port(Port *port, int portnum, ib_portid_t *portid) > > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout)) > return -1; > - > - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > + decode_port_info(pi, port); > > DEBUG("portid %s portnum %d: lid %d state %d physstate %d %s %s", > portid2str(portid), portnum, port->lid, port->state, port->physstate, get_linkwidth_str(port->linkwidth), get_linkspeed_str(port->linkspeed)); > @@ -181,13 +187,7 @@ get_node(Node *node, Port *port, ib_portid_t *portid) > > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, 0, timeout)) > return -1; > - > - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > + decode_port_info(pi, port); > > if (node->type != SWITCH_NODE) > return 0; > @@ -195,6 +195,11 @@ get_node(Node *node, Port *port, ib_portid_t *portid) > node->smalid = port->lid; > node->smalmc = port->lmc; > > + /* after we have the sma information find out the real PortInfo for this port */ > + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->localport, timeout)) > + return -1; > + decode_port_info(pi, port); > + Seems like this may only be needed when peer is switch but maybe it's not worth saving those queries (I didn't look to see if the peer is known here). -- Hal > if (!smp_query(si, portid, IB_ATTR_SWITCH_INFO, 0, timeout)) > node->smaenhsp0 = 0; /* assume base SP0 */ > else From sashak at voltaire.com Fri Jun 6 04:19:03 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:19:03 +0300 Subject: [ofa-general] Re: [PATCH][MINOR] opensm/osm_sa_mcmember_record.c: Minor logic change in __get_new_mlid In-Reply-To: <1212579420.6127.57.camel@hrosenstock-ws.xsigo.com> References: <1212579420.6127.57.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606111903.GC7795@sashak.voltaire.com> On 04:37 Wed 04 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: Minor logic change in __get_new_mlid > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:20:23 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:20:23 +0300 Subject: [ofa-general] Re: [PATCH][MINOR] opensm/osm_ucast_ftree.c: Eliminate unnecessary check in __osm_ftree_sw_tbl_element_create In-Reply-To: <1212579436.6127.58.camel@hrosenstock-ws.xsigo.com> References: <1212579436.6127.58.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606112023.GD7795@sashak.voltaire.com> On 04:37 Wed 04 Jun , Hal Rosenstock wrote: > opensm/osm_ucast_ftree.c: Eliminate unnecessary check in > __osm_ftree_sw_tbl_element_create > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:23:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:23:29 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_pkey.c: Eliminate potential NULL pointer dereference In-Reply-To: <1212579477.6127.60.camel@hrosenstock-ws.xsigo.com> References: <1212579477.6127.60.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606112329.GE7795@sashak.voltaire.com> On 04:37 Wed 04 Jun , Hal Rosenstock wrote: > opensm/osm_pkey.c: Eliminate potential NULL pointer dereference > Also, a comment reformat > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:32:00 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:32:00 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_port.c: Eliminate potential NULL pointer dereferences In-Reply-To: <1212583467.6127.64.camel@hrosenstock-ws.xsigo.com> References: <1212583467.6127.64.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606113200.GF7795@sashak.voltaire.com> On 05:44 Wed 04 Jun , Hal Rosenstock wrote: > opensm/osm_port.c: Eliminate potential NULL pointer dereferences > Also, comment format change > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:41:37 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:41:37 +0300 Subject: [ofa-general] Re: [PATCHv2][TRIVIAL] ibsim/sim_mad.c: Cosmetic change to some debug messages In-Reply-To: <1212670305.6127.135.camel@hrosenstock-ws.xsigo.com> References: <1212670305.6127.135.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606114137.GG7795@sashak.voltaire.com> On 05:51 Thu 05 Jun , Hal Rosenstock wrote: > ibsim/sim_mad.c: Cosmetic change to some debug messages > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:41:52 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:41:52 +0300 Subject: [ofa-general] Re: [PATCHv2][TRIVIAL] ibsim/ibsim.c: Cosmetic change to some debug messages In-Reply-To: <1212670307.6127.136.camel@hrosenstock-ws.xsigo.com> References: <1212670307.6127.136.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606114152.GH7795@sashak.voltaire.com> On 05:51 Thu 05 Jun , Hal Rosenstock wrote: > ibsim/ibsim.c: Cosmetic change to some debug messages > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:44:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:44:21 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/README: Cosmetic commentary changes In-Reply-To: <1212689686.6127.151.camel@hrosenstock-ws.xsigo.com> References: <1212689686.6127.151.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606114421.GI7795@sashak.voltaire.com> On 11:14 Thu 05 Jun , Hal Rosenstock wrote: > ibsim/README: Cosmetic commentary changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. But see below. > diff --git a/README b/README > index b7615aa..7159543 100644 > --- a/README > +++ b/README > @@ -33,19 +33,19 @@ SM/SA/PerfMgr to ibsim. > Any libibumad based application may work with ibsim. Kernel support > and userspace application recompilation are not required. > > -ibsim may works locally via unix sockets or remotely via inet sockets > -(not implemented yet). > +ibsim works locally via unix sockets or remotely via inet sockets > +(latter not implemented yet). It is implemented already. Sasha From hrosenstock at xsigo.com Fri Jun 6 04:47:46 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 06 Jun 2008 04:47:46 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/README: Cosmetic commentary changes In-Reply-To: <20080606114421.GI7795@sashak.voltaire.com> References: <1212689686.6127.151.camel@hrosenstock-ws.xsigo.com> <20080606114421.GI7795@sashak.voltaire.com> Message-ID: <1212752866.6127.229.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-06 at 14:44 +0300, Sasha Khapyorsky wrote: > > -ibsim may works locally via unix sockets or remotely via inet sockets > > -(not implemented yet). > > +ibsim works locally via unix sockets or remotely via inet sockets > > +(latter not implemented yet). > > It is implemented already. Then remove that parenthetical comment. I was merely updating for grammar. -- Hal > Sasha From sashak at voltaire.com Fri Jun 6 04:47:35 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:47:35 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/README: Cosmetic commentary changes In-Reply-To: <1212752866.6127.229.camel@hrosenstock-ws.xsigo.com> References: <1212689686.6127.151.camel@hrosenstock-ws.xsigo.com> <20080606114421.GI7795@sashak.voltaire.com> <1212752866.6127.229.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606114735.GJ7795@sashak.voltaire.com> On 04:47 Fri 06 Jun , Hal Rosenstock wrote: > On Fri, 2008-06-06 at 14:44 +0300, Sasha Khapyorsky wrote: > > > -ibsim may works locally via unix sockets or remotely via inet sockets > > > -(not implemented yet). > > > +ibsim works locally via unix sockets or remotely via inet sockets > > > +(latter not implemented yet). > > > > It is implemented already. > > Then remove that parenthetical comment. Already done. Sasha From sashak at voltaire.com Fri Jun 6 04:50:31 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:50:31 +0300 Subject: [ofa-general] Re: [PATCH] libibmad/fields.c: _set_field64 sets field in network rather than host order In-Reply-To: <1212689678.6127.150.camel@hrosenstock-ws.xsigo.com> References: <1212689678.6127.150.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606115031.GK7795@sashak.voltaire.com> On 11:14 Thu 05 Jun , Hal Rosenstock wrote: > libibmad/fields.c: _set_field64 sets field in network rather than host > order > > Fix issue introduced by commit 608cddb34fda942b37d0f2bb2e7889e22ac046f8 > > Signed-off-by: Hal Rosenstock Nice finding. Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 6 04:53:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 14:53:50 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606115350.GL7795@sashak.voltaire.com> On 12:51 Thu 05 Jun , Hal Rosenstock wrote: > infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found > > ERR: cannot open port_guids.list: No such file or directory > > Signed-off-by: Hal Rosenstock Applied. Thanks. Actually there is a better tool under ibsim/tests - mcast_storm. Sasha From sashak at voltaire.com Fri Jun 6 05:03:56 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Jun 2008 15:03:56 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/ibsendtrap.c: Support CA and port num In-Reply-To: <1212695514.6127.162.camel@hrosenstock-ws.xsigo.com> References: <1212695514.6127.162.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606120356.GM7795@sashak.voltaire.com> On 12:51 Thu 05 Jun , Hal Rosenstock wrote: > infiniband-diags/ibsendtrap.c: Support CA and port num > Also, fix usage > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From fenkes at de.ibm.com Fri Jun 6 09:35:43 2008 From: fenkes at de.ibm.com (Joachim Fenkes) Date: Fri, 6 Jun 2008 18:35:43 +0200 Subject: [ofa-general] IB/ehca: Reject send WRs only for RESET, INIT and RTR state Message-ID: <200806061835.43802.fenkes@de.ibm.com> Signed-off-by: Joachim Fenkes --- drivers/infiniband/hw/ehca/ehca_reqs.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_reqs.c b/drivers/infiniband/hw/ehca/ehca_reqs.c index bbe0436..f093b00 100644 --- a/drivers/infiniband/hw/ehca/ehca_reqs.c +++ b/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -421,8 +421,10 @@ int ehca_post_send(struct ib_qp *qp, int ret = 0; unsigned long flags; - if (unlikely(my_qp->state != IB_QPS_RTS)) { - ehca_err(qp->device, "QP not in RTS state qpn=%x", qp->qp_num); + /* Reject WR if QP is in RESET, INIT or RTR state */ + if (unlikely(my_qp->state < IB_QPS_RTS)) { + ehca_err(qp->device, "Invalid QP state qp_state=%d qpn=%x", + my_qp->state, qp->qp_num); return -EINVAL; } -- 1.5.5 From ralph.campbell at qlogic.com Fri Jun 6 09:49:41 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Fri, 06 Jun 2008 09:49:41 -0700 Subject: [ofa-general] [PATCH] IB/ipath - fix SM trap forwarding Message-ID: <20080606164941.8239.10224.stgit@eng-46.mv.qlogic.com> SM/SMA traps received by the ipath driver should be forwarded to the SM if it is running on the host. The ib_ipath driver was incorrectly replying with bad method. Signed-off-by: Ralph Campbell --- drivers/infiniband/hw/ipath/ipath_mad.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/hw/ipath/ipath_mad.c b/drivers/infiniband/hw/ipath/ipath_mad.c index 1ff46ae..5f9315d 100644 --- a/drivers/infiniband/hw/ipath/ipath_mad.c +++ b/drivers/infiniband/hw/ipath/ipath_mad.c @@ -1492,6 +1492,10 @@ static int process_subn(struct ib_device *ibdev, int mad_flags, goto bail; } + case IB_MGMT_METHOD_TRAP: + case IB_MGMT_METHOD_REPORT: + case IB_MGMT_METHOD_REPORT_RESP: + case IB_MGMT_METHOD_TRAP_REPRESS: case IB_MGMT_METHOD_GET_RESP: /* * The ib_mad module will call us to process responses From PHF at zurich.ibm.com Fri Jun 6 09:59:27 2008 From: PHF at zurich.ibm.com (Philip Frey1) Date: Fri, 6 Jun 2008 18:59:27 +0200 Subject: [ofa-general] missed cq event Message-ID: Hello, what is the correct way to get completions from the CQ without missing any? I am currently using two separate CQs for the SQ and for the RQ. In pseudocode I do about the following to wait for a completion: ibv_get_cq_event(channel, &dst_cq, &ctx); //blocking wait for CQ event <-- EVENT --> ibv_req_notify_cq(dst_cq, 0); // request event for next completion ibv_poll_cq(dest_cq, 1, wc); // get the WC from the CQ ibv_ack_cq_events(dst_cq, 1); // ack the event It might now happen on the RQ, that two consecutive inbound sends consume receive WRs and therefore generate two WCs on the RQ. If the second is generated before I get the first one, I will miss it and wait infinitely long for it since I already got the event but that event does not tell me how many WCs are pending on the respective CQ. One way to handle this issue would be to do an ibv_poll_cq() before waiting for the event in case there is something on the CQ for which I missed the event. Is there a more elegant way to get all events pending on the CQ or find out how many completions are pending? I could also do a ibv_poll_cq() and request say 10 completions and see how many I got but that is kind of a hack... Many thanks for your advice, Philip -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Fri Jun 6 10:25:31 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Jun 2008 12:25:31 -0500 Subject: [ofa-general] missed cq event In-Reply-To: References: Message-ID: <4849730B.5060008@opengridcomputing.com> Philip Frey1 wrote: > > Hello, > > what is the correct way to get completions from the CQ without missing > any? > I am currently using two separate CQs for the SQ and for the RQ. > In pseudocode I do about the following to wait for a completion: > > ibv_get_cq_event(channel, &dst_cq, &ctx); //blocking wait for > CQ event > <-- EVENT --> > ibv_req_notify_cq(dst_cq, 0); // request event > for next completion > ibv_poll_cq(dest_cq, 1, wc); // get the WC from > the CQ > ibv_ack_cq_events(dst_cq, 1); // ack the event > > It might now happen on the RQ, that two consecutive inbound sends consume > receive WRs and therefore generate two WCs on the RQ. If the second is > generated > before I get the first one, I will miss it and wait infinitely long > for it since I already got the event > but that event does not tell me how many WCs are pending on the > respective CQ. > > One way to handle this issue would be to do an ibv_poll_cq() before > waiting for the event > in case there is something on the CQ for which I missed the event. > > Is there a more elegant way to get all events pending on the CQ or > find out how many > completions are pending? I could also do a ibv_poll_cq() and request > say 10 completions > and see how many I got but that is kind of a hack... you need to poll the cq until it is empty instead of just polling once. get_cq_event() req_notify_cq() do { ne = poll_cq(); } while (ne != 0); ack_cq_event() Steve. From weiny2 at llnl.gov Fri Jun 6 10:33:06 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 6 Jun 2008 10:33:06 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. In-Reply-To: <1212750821.6127.217.camel@hrosenstock-ws.xsigo.com> References: <20080605191251.6882c3d6.weiny2@llnl.gov> <1212750821.6127.217.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080606103306.163d6797.weiny2@llnl.gov> Hey Hal, comments inline... On Fri, 06 Jun 2008 04:13:41 -0700 Hal Rosenstock wrote: > Hi Ira, > > On Thu, 2008-06-05 at 19:12 -0700, Ira Weiny wrote: > > I noticed over the last couple days that ibnetdiscover would print the > > incorrect speed on the port I would run ibnetdiscover from. For example: > > > > Switch 24 "S-0008f10400411f56" # "SW1 wopr ISR9024D (MLX4 FW)" base port 0 lid 11 lmc 0 > > ... > > [13] "H-0002c90200219e64"[1](2c90200219e65) # "wopri" lid 32 4xSDR > > > > ^^^ > > (Note from the switch side of things it thinks the speed is SDR.) > > ... > > Ca 2 "H-0002c90200219e64" # "wopri" > > [1](2c90200219e65) "S-0008f10400411f56"[13] # lid 32 lmc 0 "SW1 wopr ISR9024D (MLX4 FW)" lid 11 4xDDR > > > > ^^^ > > (but here DDR.) > > Yes, that is clearly wrong. Good find. > > > It turns out that when you first discover a switch the port object created gets > > the PortInfo of Port "0". > > Right; this trick works for peer CA and router ports but not switch > ports where the actual port number is needed. Ah! I guess that is the reason the PortInfo is querried here instead of just in "get_port" later? I was somewhat curious why things were done this way. That makes a lot more sense. > > > This patch queries again for the PortInfo of the port we "came in on". > > > > Ira > > > > > > From 5bc66af276c7baabd4d66be9df0379271cb625b4 Mon Sep 17 00:00:00 2001 > > From: Ira K. Weiny > > Date: Thu, 5 Jun 2008 19:03:44 -0700 > > Subject: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. > > > > Previously the portinfo data for that port object would be the PortInfo of > > port "0" of the switch. Specifically this would cause the speed of the link > > to be printed incorrectly. > > Did you try back to back CAs too ? No. See below regarding ca/routers. > > Nit: this looks like 2 patches to me: > 1. Factor out redundant code by adding decode_port_info routine > 2. Fix link speed of local CA in switch display Well technically yes but I think #1 really is a result of the fix because now the code is repeated 3 times instead of 2. I could split it, if you like, but it seemed like the fix required some code clean up to be "clean". (Although I admit that the patch itself is more complex.) > > > Signed-off-by: Ira K. Weiny > > --- > > infiniband-diags/src/ibnetdiscover.c | 33 +++++++++++++++++++-------------- > > 1 files changed, 19 insertions(+), 14 deletions(-) > > > > diff --git a/infiniband-diags/src/ibnetdiscover.c b/infiniband-diags/src/ibnetdiscover.c > > index b8d4e92..20da1ea 100644 > > --- a/infiniband-diags/src/ibnetdiscover.c > > +++ b/infiniband-diags/src/ibnetdiscover.c > > @@ -129,6 +129,18 @@ node_type_str2(Node *node) > > return "??"; > > } > > > > +void > > +decode_port_info(void *pi, Port *port) > > +{ > > + mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > > + mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > > + mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > > + mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > > + mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > > + mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > > +} > > + > > + > > int > > get_port(Port *port, int portnum, ib_portid_t *portid) > > { > > @@ -139,13 +151,7 @@ get_port(Port *port, int portnum, ib_portid_t *portid) > > > > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout)) > > return -1; > > - > > - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > > - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > > - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > > - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > > - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > > - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > > + decode_port_info(pi, port); > > > > DEBUG("portid %s portnum %d: lid %d state %d physstate %d %s %s", > > portid2str(portid), portnum, port->lid, port->state, port->physstate, get_linkwidth_str(port->linkwidth), get_linkspeed_str(port->linkspeed)); > > @@ -181,13 +187,7 @@ get_node(Node *node, Port *port, ib_portid_t *portid) > > > > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, 0, timeout)) > > return -1; > > - > > - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > > - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > > - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > > - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > > - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > > - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > > + decode_port_info(pi, port); > > > > if (node->type != SWITCH_NODE) > > return 0; > > @@ -195,6 +195,11 @@ get_node(Node *node, Port *port, ib_portid_t *portid) > > node->smalid = port->lid; > > node->smalmc = port->lmc; > > > > + /* after we have the sma information find out the real PortInfo for this port */ > > + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->localport, timeout)) > > + return -1; > > + decode_port_info(pi, port); > > + > > Seems like this may only be needed when peer is switch but maybe it's > not worth saving those queries (I didn't look to see if the peer is > known here). This is past a check for not Switch. Here is a more complete chunk: if (node->type != SWITCH_NODE) return 0; node->smalid = port->lid; node->smalmc = port->lmc; /* after we have the sma information find out the real PortInfo for this port */ if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->localport, timeout)) return -1; decode_port_info(pi, port); So I think for CA's/routers, back to back it should work the same. Also the additional query will only be issued if needed. Ira > > -- Hal > > > if (!smp_query(si, portid, IB_ATTR_SWITCH_INFO, 0, timeout)) > > node->smaenhsp0 = 0; /* assume base SP0 */ > > else > From swise at opengridcomputing.com Fri Jun 6 10:34:17 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Jun 2008 12:34:17 -0500 Subject: [ofa-general] recv with rdma immediate completion questions Message-ID: <48497519.3020106@opengridcomputing.com> Why is there a work completion opcode IB_WC_RECV_RDMA_WITH_IMM -and- a work completion flag IB_WC_WITH_IMM indicating the immediate data is present in the WC? Isn't this redundant? Steve. From rdreier at cisco.com Fri Jun 6 10:38:02 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 10:38:02 -0700 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: <48497519.3020106@opengridcomputing.com> (Steve Wise's message of "Fri, 06 Jun 2008 12:34:17 -0500") References: <48497519.3020106@opengridcomputing.com> Message-ID: > Why is there a work completion opcode IB_WC_RECV_RDMA_WITH_IMM -and- a > work completion flag IB_WC_WITH_IMM indicating the immediate data is > present in the WC? Off the top of my head ... because RDMA operations usually generate no completion on the target side, so you need some opcode for when one comes in with immediate data. I'd have to check what the IB verbs spec says about this case (obviously iWARP doesn't have to worry about immediate data) From Zjensen at doyon.com Fri Jun 6 08:52:06 2008 From: Zjensen at doyon.com (fleming byrd) Date: Fri, 06 Jun 2008 15:52:06 +0000 Subject: [ofa-general] Brand name timepieces at half price Message-ID: <000501c8c7fc$05828705$14a909ba@rmhqha> Look rich and successful without having to spend thousands. Nothing impresses business colleagues and friends more than a diamond studded Rolex, or a classic Patek Philippe strapped arond your wrist. King Replicas houses more than 20+ world famous brands. Prices for the originals range in the tends of thousands, but pick up a buy for just $49 up. Patek, Rolex, Breitling, Cartier and many more! Sale ends this week - so visit now for the best bargains. Visit - www.merrqts.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsleepctr at stvincent.org Fri Jun 6 08:52:43 2008 From: tsleepctr at stvincent.org (elvis faith) Date: Fri, 06 Jun 2008 15:52:43 +0000 Subject: [ofa-general] Purchase high fashion at low cost Message-ID: <000801c8c7fc$04b08603$159a3895@vpppyc> Look rich and successful without having to spend thousands. Nothing impresses business colleagues and friends more than a diamond studded Rolex, or a classic Patek Philippe strapped arond your wrist. King Replicas houses more than 20+ world famous brands. Prices for the originals range in the tends of thousands, but pick up a buy for just $49 up. Patek, Rolex, Breitling, Cartier and many more! Sale ends this week - so visit now for the best bargains. Visit - www.merrqts.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralph.campbell at qlogic.com Fri Jun 6 10:41:26 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Fri, 06 Jun 2008 10:41:26 -0700 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: <48497519.3020106@opengridcomputing.com> References: <48497519.3020106@opengridcomputing.com> Message-ID: <1212774086.22429.73.camel@chromite.mv.qlogic.com> On Fri, 2008-06-06 at 12:34 -0500, Steve Wise wrote: > Why is there a work completion opcode IB_WC_RECV_RDMA_WITH_IMM -and- a > work completion flag IB_WC_WITH_IMM indicating the immediate data is > present in the WC? > > Isn't this redundant? > > Steve. Not really. The CQE opcode tells the difference between a posted receive WR being consumed by a SEND verses a RDMA WRITE with immediate. From hrosenstock at xsigo.com Fri Jun 6 10:43:22 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 06 Jun 2008 10:43:22 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. In-Reply-To: <20080606103306.163d6797.weiny2@llnl.gov> References: <20080605191251.6882c3d6.weiny2@llnl.gov> <1212750821.6127.217.camel@hrosenstock-ws.xsigo.com> <20080606103306.163d6797.weiny2@llnl.gov> Message-ID: <1212774202.6127.233.camel@hrosenstock-ws.xsigo.com> Hi Ira, On Fri, 2008-06-06 at 10:33 -0700, Ira Weiny wrote: > Hey Hal, comments inline... > > On Fri, 06 Jun 2008 04:13:41 -0700 > Hal Rosenstock wrote: > > > Hi Ira, > > > > On Thu, 2008-06-05 at 19:12 -0700, Ira Weiny wrote: > > > I noticed over the last couple days that ibnetdiscover would print the > > > incorrect speed on the port I would run ibnetdiscover from. For example: > > > > > > Switch 24 "S-0008f10400411f56" # "SW1 wopr ISR9024D (MLX4 FW)" base port 0 lid 11 lmc 0 > > > ... > > > [13] "H-0002c90200219e64"[1](2c90200219e65) # "wopri" lid 32 4xSDR > > > > > > ^^^ > > > (Note from the switch side of things it thinks the speed is SDR.) > > > ... > > > Ca 2 "H-0002c90200219e64" # "wopri" > > > [1](2c90200219e65) "S-0008f10400411f56"[13] # lid 32 lmc 0 "SW1 wopr ISR9024D (MLX4 FW)" lid 11 4xDDR > > > > > > ^^^ > > > (but here DDR.) > > > > Yes, that is clearly wrong. Good find. > > > > > It turns out that when you first discover a switch the port object created gets > > > the PortInfo of Port "0". > > > > Right; this trick works for peer CA and router ports but not switch > > ports where the actual port number is needed. > > Ah! I guess that is the reason the PortInfo is querried here instead of just > in "get_port" later? I was somewhat curious why things were done this way. > That makes a lot more sense. > > > > > > This patch queries again for the PortInfo of the port we "came in on". > > > > > > Ira > > > > > > > > > From 5bc66af276c7baabd4d66be9df0379271cb625b4 Mon Sep 17 00:00:00 2001 > > > From: Ira K. Weiny > > > Date: Thu, 5 Jun 2008 19:03:44 -0700 > > > Subject: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. > > > > > > Previously the portinfo data for that port object would be the PortInfo of > > > port "0" of the switch. Specifically this would cause the speed of the link > > > to be printed incorrectly. > > > > Did you try back to back CAs too ? > > No. See below regarding ca/routers. > > > > > Nit: this looks like 2 patches to me: > > 1. Factor out redundant code by adding decode_port_info routine > > 2. Fix link speed of local CA in switch display > > Well technically yes but I think #1 really is a result of the fix because now > the code is repeated 3 times instead of 2. I could split it, if you like, but > it seemed like the fix required some code clean up to be "clean". (Although I > admit that the patch itself is more complex.) It's between you and Sasha; just a nit in separating the changes. > > > > > Signed-off-by: Ira K. Weiny > > > --- > > > infiniband-diags/src/ibnetdiscover.c | 33 +++++++++++++++++++-------------- > > > 1 files changed, 19 insertions(+), 14 deletions(-) > > > > > > diff --git a/infiniband-diags/src/ibnetdiscover.c b/infiniband-diags/src/ibnetdiscover.c > > > index b8d4e92..20da1ea 100644 > > > --- a/infiniband-diags/src/ibnetdiscover.c > > > +++ b/infiniband-diags/src/ibnetdiscover.c > > > @@ -129,6 +129,18 @@ node_type_str2(Node *node) > > > return "??"; > > > } > > > > > > +void > > > +decode_port_info(void *pi, Port *port) > > > +{ > > > + mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > > > + mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > > > + mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > > > + mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > > > + mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > > > + mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > > > +} > > > + > > > + > > > int > > > get_port(Port *port, int portnum, ib_portid_t *portid) > > > { > > > @@ -139,13 +151,7 @@ get_port(Port *port, int portnum, ib_portid_t *portid) > > > > > > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout)) > > > return -1; > > > - > > > - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > > > - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > > > - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > > > - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > > > - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > > > - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > > > + decode_port_info(pi, port); > > > > > > DEBUG("portid %s portnum %d: lid %d state %d physstate %d %s %s", > > > portid2str(portid), portnum, port->lid, port->state, port->physstate, get_linkwidth_str(port->linkwidth), get_linkspeed_str(port->linkspeed)); > > > @@ -181,13 +187,7 @@ get_node(Node *node, Port *port, ib_portid_t *portid) > > > > > > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, 0, timeout)) > > > return -1; > > > - > > > - mad_decode_field(pi, IB_PORT_LID_F, &port->lid); > > > - mad_decode_field(pi, IB_PORT_LMC_F, &port->lmc); > > > - mad_decode_field(pi, IB_PORT_STATE_F, &port->state); > > > - mad_decode_field(pi, IB_PORT_PHYS_STATE_F, &port->physstate); > > > - mad_decode_field(pi, IB_PORT_LINK_WIDTH_ACTIVE_F, &port->linkwidth); > > > - mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); > > > + decode_port_info(pi, port); > > > > > > if (node->type != SWITCH_NODE) > > > return 0; > > > @@ -195,6 +195,11 @@ get_node(Node *node, Port *port, ib_portid_t *portid) > > > node->smalid = port->lid; > > > node->smalmc = port->lmc; > > > > > > + /* after we have the sma information find out the real PortInfo for this port */ > > > + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->localport, timeout)) > > > + return -1; > > > + decode_port_info(pi, port); > > > + > > > > Seems like this may only be needed when peer is switch but maybe it's > > not worth saving those queries (I didn't look to see if the peer is > > known here). > > This is past a check for not Switch. Here is a more complete chunk: > > if (node->type != SWITCH_NODE) > return 0; > > node->smalid = port->lid; > node->smalmc = port->lmc; > > /* after we have the sma information find out the real PortInfo for this port */ > if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->localport, timeout)) > return -1; > decode_port_info(pi, port); > > > So I think for CA's/routers, back to back it should work the same. Also > the additional query will only be issued if needed. Looks good to me. -- Hal > Ira > > > > > -- Hal > > > > > if (!smp_query(si, portid, IB_ATTR_SWITCH_INFO, 0, timeout)) > > > node->smaenhsp0 = 0; /* assume base SP0 */ > > > else > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From swise at opengridcomputing.com Fri Jun 6 10:44:46 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Jun 2008 12:44:46 -0500 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: <1212774086.22429.73.camel@chromite.mv.qlogic.com> References: <48497519.3020106@opengridcomputing.com> <1212774086.22429.73.camel@chromite.mv.qlogic.com> Message-ID: <4849778E.40709@opengridcomputing.com> Ralph Campbell wrote: > On Fri, 2008-06-06 at 12:34 -0500, Steve Wise wrote: > >> Why is there a work completion opcode IB_WC_RECV_RDMA_WITH_IMM -and- a >> work completion flag IB_WC_WITH_IMM indicating the immediate data is >> present in the WC? >> >> Isn't this redundant? >> >> Steve. >> > > Not really. The CQE opcode tells the difference between a > posted receive WR being consumed by a SEND verses a > RDMA WRITE with immediate. > So would the flag indicating the immediate data word in the wc is valid. From rdreier at cisco.com Fri Jun 6 10:46:20 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 10:46:20 -0700 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: <4849778E.40709@opengridcomputing.com> (Steve Wise's message of "Fri, 06 Jun 2008 12:44:46 -0500") References: <48497519.3020106@opengridcomputing.com> <1212774086.22429.73.camel@chromite.mv.qlogic.com> <4849778E.40709@opengridcomputing.com> Message-ID: > > Not really. The CQE opcode tells the difference between a > > posted receive WR being consumed by a SEND verses a > > RDMA WRITE with immediate. > > > So would the flag indicating the immediate data word in the wc is valid. But how do you tell the difference between an incoming send with immediate and an incoming RDMA write with immediate? From swise at opengridcomputing.com Fri Jun 6 10:46:44 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Jun 2008 12:46:44 -0500 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: References: <48497519.3020106@opengridcomputing.com> Message-ID: <48497804.4010702@opengridcomputing.com> Roland Dreier wrote: > > Why is there a work completion opcode IB_WC_RECV_RDMA_WITH_IMM -and- a > > work completion flag IB_WC_WITH_IMM indicating the immediate data is > > present in the WC? > > Off the top of my head ... because RDMA operations usually generate no > completion on the target side, so you need some opcode for when one > comes in with immediate data. > > I'd have to check what the IB verbs spec says about this case (obviously > iWARP doesn't have to worry about immediate data) > What I was thinking was that: completion with IB_WC_RECV + IB_WC_WITH_IMM is the same information as completion wtih IB_WC_RECV_RDMA_WITH_IMM + IB_WC_WITH_IMMM. And I was wondering if there was a good reason for this, and whether we need both an opcode -and- a flag for recv completions that are due to an ingress SEND_WITH_INV... Seems like you only either need the opcode or the flag, but not both... Steve. From ralph.campbell at qlogic.com Fri Jun 6 10:51:41 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Fri, 06 Jun 2008 10:51:41 -0700 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: <48497804.4010702@opengridcomputing.com> References: <48497519.3020106@opengridcomputing.com> <48497804.4010702@opengridcomputing.com> Message-ID: <1212774701.22429.77.camel@chromite.mv.qlogic.com> On Fri, 2008-06-06 at 12:46 -0500, Steve Wise wrote: > Roland Dreier wrote: > > > Why is there a work completion opcode IB_WC_RECV_RDMA_WITH_IMM -and- a > > > work completion flag IB_WC_WITH_IMM indicating the immediate data is > > > present in the WC? > > > > Off the top of my head ... because RDMA operations usually generate no > > completion on the target side, so you need some opcode for when one > > comes in with immediate data. > > > > I'd have to check what the IB verbs spec says about this case (obviously > > iWARP doesn't have to worry about immediate data) > > > What I was thinking was that: > > completion with IB_WC_RECV + IB_WC_WITH_IMM is the same information as > completion wtih IB_WC_RECV_RDMA_WITH_IMM + IB_WC_WITH_IMMM. > > And I was wondering if there was a good reason for this, and whether we > need both an opcode -and- a flag for recv completions that are due to an > ingress SEND_WITH_INV... > > Seems like you only either need the opcode or the flag, but not both... > > Steve. The cases are: opcode == IB_WC_RECV, flags 0 or IB_WC_WITH_IMM, opcode == IB_WC_RECV_RDMA_WITH_IMM, flags IB_WC_WITH_IMM. There is no IB_WC_RECV_RDMA opcode because it doesn't create a CQE. From swise at opengridcomputing.com Fri Jun 6 10:52:49 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Jun 2008 12:52:49 -0500 Subject: [ofa-general] recv with rdma immediate completion questions In-Reply-To: References: <48497519.3020106@opengridcomputing.com> <1212774086.22429.73.camel@chromite.mv.qlogic.com> <4849778E.40709@opengridcomputing.com> Message-ID: <48497971.90405@opengridcomputing.com> Roland Dreier wrote: > > > Not really. The CQE opcode tells the difference between a > > > posted receive WR being consumed by a SEND verses a > > > RDMA WRITE with immediate. > > > > > So would the flag indicating the immediate data word in the wc is valid. > > But how do you tell the difference between an incoming send with > immediate and an incoming RDMA write with immediate? > Ah. There's a send-with-immediate? :) I see. Nevermind. From rdreier at cisco.com Fri Jun 6 11:22:32 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 11:22:32 -0700 Subject: [ofa-general] Re: IB/ehca: Reject send WRs only for RESET, INIT and RTR state In-Reply-To: <200806061835.43802.fenkes@de.ibm.com> (Joachim Fenkes's message of "Fri, 6 Jun 2008 18:35:43 +0200") References: <200806061835.43802.fenkes@de.ibm.com> Message-ID: thanks, applied. From rdreier at cisco.com Fri Jun 6 11:23:39 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 11:23:39 -0700 Subject: [ofa-general] [PATCH] IB/ipath - fix SM trap forwarding In-Reply-To: <20080606164941.8239.10224.stgit@eng-46.mv.qlogic.com> (Ralph Campbell's message of "Fri, 06 Jun 2008 09:49:41 -0700") References: <20080606164941.8239.10224.stgit@eng-46.mv.qlogic.com> Message-ID: thanks, applied. From rdreier at cisco.com Fri Jun 6 11:26:11 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 11:26:11 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This will get some small fixes for various low-level driver issues: Chien Tung (1): MAINTAINERS: Update NetEffect (iw_nes) entry Joachim Fenkes (1): IB/ehca: Reject send WRs only for RESET, INIT and RTR state Ralph Campbell (2): IB/ipath: Fix device capability flags IB/ipath: Fix SM trap forwarding Roland Dreier (1): IB/ipath: Avoid test_bit() on u64 SDMA status value MAINTAINERS | 4 ++-- drivers/infiniband/hw/ehca/ehca_reqs.c | 6 ++++-- drivers/infiniband/hw/ipath/ipath_kernel.h | 5 +++++ drivers/infiniband/hw/ipath/ipath_mad.c | 4 ++++ drivers/infiniband/hw/ipath/ipath_sdma.c | 12 ++++-------- drivers/infiniband/hw/ipath/ipath_verbs.c | 3 ++- 6 files changed, 21 insertions(+), 13 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index 0a6d2ca..bd3badb 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2866,8 +2866,8 @@ S: Maintained NETEFFECT IWARP RNIC DRIVER (IW_NES) P: Faisal Latif M: flatif at neteffect.com -P: Nishi Gupta -M: ngupta at neteffect.com +P: Chien Tung +M: ctung at neteffect.com P: Glenn Streiff M: gstreiff at neteffect.com L: general at lists.openfabrics.org diff --git a/drivers/infiniband/hw/ehca/ehca_reqs.c b/drivers/infiniband/hw/ehca/ehca_reqs.c index bbe0436..f093b00 100644 --- a/drivers/infiniband/hw/ehca/ehca_reqs.c +++ b/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -421,8 +421,10 @@ int ehca_post_send(struct ib_qp *qp, int ret = 0; unsigned long flags; - if (unlikely(my_qp->state != IB_QPS_RTS)) { - ehca_err(qp->device, "QP not in RTS state qpn=%x", qp->qp_num); + /* Reject WR if QP is in RESET, INIT or RTR state */ + if (unlikely(my_qp->state < IB_QPS_RTS)) { + ehca_err(qp->device, "Invalid QP state qp_state=%d qpn=%x", + my_qp->state, qp->qp_num); return -EINVAL; } diff --git a/drivers/infiniband/hw/ipath/ipath_kernel.h b/drivers/infiniband/hw/ipath/ipath_kernel.h index 59a8b25..0bd8bcb 100644 --- a/drivers/infiniband/hw/ipath/ipath_kernel.h +++ b/drivers/infiniband/hw/ipath/ipath_kernel.h @@ -232,6 +232,11 @@ struct ipath_sdma_desc { #define IPATH_SDMA_TXREQ_S_ABORTED 2 #define IPATH_SDMA_TXREQ_S_SHUTDOWN 3 +#define IPATH_SDMA_STATUS_SCORE_BOARD_DRAIN_IN_PROG (1ull << 63) +#define IPATH_SDMA_STATUS_ABORT_IN_PROG (1ull << 62) +#define IPATH_SDMA_STATUS_INTERNAL_SDMA_ENABLE (1ull << 61) +#define IPATH_SDMA_STATUS_SCB_EMPTY (1ull << 30) + /* max dwords in small buffer packet */ #define IPATH_SMALLBUF_DWORDS (dd->ipath_piosize2k >> 2) diff --git a/drivers/infiniband/hw/ipath/ipath_mad.c b/drivers/infiniband/hw/ipath/ipath_mad.c index 1ff46ae..5f9315d 100644 --- a/drivers/infiniband/hw/ipath/ipath_mad.c +++ b/drivers/infiniband/hw/ipath/ipath_mad.c @@ -1492,6 +1492,10 @@ static int process_subn(struct ib_device *ibdev, int mad_flags, goto bail; } + case IB_MGMT_METHOD_TRAP: + case IB_MGMT_METHOD_REPORT: + case IB_MGMT_METHOD_REPORT_RESP: + case IB_MGMT_METHOD_TRAP_REPRESS: case IB_MGMT_METHOD_GET_RESP: /* * The ib_mad module will call us to process responses diff --git a/drivers/infiniband/hw/ipath/ipath_sdma.c b/drivers/infiniband/hw/ipath/ipath_sdma.c index 0a8c1b8..eaba032 100644 --- a/drivers/infiniband/hw/ipath/ipath_sdma.c +++ b/drivers/infiniband/hw/ipath/ipath_sdma.c @@ -263,14 +263,10 @@ static void sdma_abort_task(unsigned long opaque) hwstatus = ipath_read_kreg64(dd, dd->ipath_kregs->kr_senddmastatus); - if (/* ScoreBoardDrainInProg */ - test_bit(63, &hwstatus) || - /* AbortInProg */ - test_bit(62, &hwstatus) || - /* InternalSDmaEnable */ - test_bit(61, &hwstatus) || - /* ScbEmpty */ - !test_bit(30, &hwstatus)) { + if ((hwstatus & (IPATH_SDMA_STATUS_SCORE_BOARD_DRAIN_IN_PROG | + IPATH_SDMA_STATUS_ABORT_IN_PROG | + IPATH_SDMA_STATUS_INTERNAL_SDMA_ENABLE)) || + !(hwstatus & IPATH_SDMA_STATUS_SCB_EMPTY)) { if (dd->ipath_sdma_reset_wait > 0) { /* not done shutting down sdma */ --dd->ipath_sdma_reset_wait; diff --git a/drivers/infiniband/hw/ipath/ipath_verbs.c b/drivers/infiniband/hw/ipath/ipath_verbs.c index e0ec540..7779165 100644 --- a/drivers/infiniband/hw/ipath/ipath_verbs.c +++ b/drivers/infiniband/hw/ipath/ipath_verbs.c @@ -1494,7 +1494,8 @@ static int ipath_query_device(struct ib_device *ibdev, props->device_cap_flags = IB_DEVICE_BAD_PKEY_CNTR | IB_DEVICE_BAD_QKEY_CNTR | IB_DEVICE_SHUTDOWN_PORT | - IB_DEVICE_SYS_IMAGE_GUID; + IB_DEVICE_SYS_IMAGE_GUID | IB_DEVICE_RC_RNR_NAK_GEN | + IB_DEVICE_PORT_ACTIVE_EVENT | IB_DEVICE_SRQ_RESIZE; props->page_size_cap = PAGE_SIZE; props->vendor_id = dev->dd->ipath_vendorid; props->vendor_part_id = dev->dd->ipath_deviceid; From rdreier at cisco.com Fri Jun 6 11:53:54 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 11:53:54 -0700 Subject: [ofa-general] Re: [PATCH 2.6.27] RDMA/Core: MEM_MGT_EXTENSIONS support In-Reply-To: <20080602175400.3510.77341.stgit@dell3.ogc.int> (Steve Wise's message of "Mon, 02 Jun 2008 12:54:01 -0500") References: <20080602174831.3510.63409.stgit@dell3.ogc.int> <20080602175400.3510.77341.stgit@dell3.ogc.int> Message-ID: OK, I got this applied... thanks for the very detailed changelog entry, that helps a lot. I fixed up a few things here and there, including changing a few names to what I thought were better ones -- however I am fine changing them back (or changing names further) if anyone has suggestions. I also merged in Eli's patch to add IB_WC_WITH_INVALIDATE, and my patch to add a ib_wc.ex.invalidate_rkey field. Full details below. I also pushed this out in my for-2.6.27 branch; further improvements/cleanups/fixes/documentation clarification etc of course welcome -- please send as patches on top of this one. - R. --- [PATCH] RDMA/Core: Add memory management extensions support This patch adds support for the IB "base memory management extension" (BMME) and the equivalent iWARP operations (which the iWARP verbs mandates all devices must implement). The new operations are: - Allocate an ib_mr for use in fast register work requests. - Allocate/free a physical buffer lists for use in fast register work requests. This allows device drivers to allocate this memory as needed for use in posting send requests (eg via dma_alloc_coherent). - New send queue work requests: * send with remote invalidate * fast register memory region * local invalidate memory region * RDMA read with invalidate local memory region (iWARP only) Consumer interface details: - A new device capability flag IB_DEVICE_MEM_MGT_EXTENSIONS is added to indicate device support for these features. - New send work request opcodes IB_WR_FAST_REG_MR, IB_WR_LOCAL_INV, IB_WR_RDMA_READ_WITH_INV are added. - A new consumer API function, ib_alloc_mr() is added to allocate fast register memory regions. - New consumer API functions, ib_alloc_fast_reg_page_list() and ib_free_fast_reg_page_list() are added to allocate and free device-specific memory for fast registration page lists. - A new consumer API function, ib_update_fast_reg_key(), is added to allow the key portion of the R_Key and L_Key of a fast registration MR to be updated. Consumers call this if desired before posting a IB_WR_FAST_REG_MR work request. Consumers can use this as follows: - MR is allocated with ib_alloc_mr(). - Page list memory is allocated with ib_alloc_fast_reg_page_list(). - MR R_Key/L_Key "key" field is updated with ib_update_fast_reg_key(). - MR made VALID and bound to a specific page list via ib_post_send(IB_WR_FAST_REG_MR) - MR made INVALID via ib_post_send(IB_WR_LOCAL_INV), ib_post_send(IB_WR_RDMA_READ_WITH_INV) or an incoming send with invalidate operation. - MR is deallocated with ib_dereg_mr() - page lists dealloced via ib_free_fast_reg_page_list(). Applications can allocate a fast register MR once, and then can repeatedly bind the MR to different physical block lists (PBLs) via posting work requests to a send queue (SQ). For each outstanding MR-to-PBL binding in the SQ pipe, a fast_reg_page_list needs to be allocated (the fast_reg_page_list is owned by the low-level driver from the consumer posting a work request until the request completes). Thus pipelining can be achieved while still allowing device-specific page_list processing. The 32-bit fast register memory key/STag is composed of a 24-bit index and an 8-bit key. The application can change the key each time it fast registers thus allowing more control over the peer's use of the key/STag (ie it can effectively be changed each time the rkey is rebound to a page list). Signed-off-by: Steve Wise Signed-off-by: Roland Dreier --- drivers/infiniband/core/uverbs_cmd.c | 2 +- drivers/infiniband/core/verbs.c | 46 +++++++++++++++++ drivers/infiniband/hw/ehca/ehca_reqs.c | 2 +- drivers/infiniband/hw/ipath/ipath_cq.c | 2 +- drivers/infiniband/hw/ipath/ipath_rc.c | 4 +- drivers/infiniband/hw/ipath/ipath_ruc.c | 4 +- drivers/infiniband/hw/ipath/ipath_uc.c | 8 ++-- drivers/infiniband/hw/ipath/ipath_ud.c | 8 ++-- drivers/infiniband/hw/mlx4/cq.c | 12 ++-- drivers/infiniband/hw/mthca/mthca_cq.c | 4 +- include/rdma/ib_verbs.h | 84 ++++++++++++++++++++++++++++++- 11 files changed, 151 insertions(+), 25 deletions(-) diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c index 112b37c..6a7f905 100644 --- a/drivers/infiniband/core/uverbs_cmd.c +++ b/drivers/infiniband/core/uverbs_cmd.c @@ -917,7 +917,7 @@ ssize_t ib_uverbs_poll_cq(struct ib_uverbs_file *file, resp->wc[i].opcode = wc[i].opcode; resp->wc[i].vendor_err = wc[i].vendor_err; resp->wc[i].byte_len = wc[i].byte_len; - resp->wc[i].imm_data = (__u32 __force) wc[i].imm_data; + resp->wc[i].imm_data = (__u32 __force) wc[i].ex.imm_data; resp->wc[i].qp_num = wc[i].qp->qp_num; resp->wc[i].src_qp = wc[i].src_qp; resp->wc[i].wc_flags = wc[i].wc_flags; diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 9f399d3..e0fbe59 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -753,6 +753,52 @@ int ib_dereg_mr(struct ib_mr *mr) } EXPORT_SYMBOL(ib_dereg_mr); +struct ib_mr *ib_alloc_fast_reg_mr(struct ib_pd *pd, int max_page_list_len) +{ + struct ib_mr *mr; + + if (!pd->device->alloc_fast_reg_mr) + return ERR_PTR(-ENOSYS); + + mr = pd->device->alloc_fast_reg_mr(pd, max_page_list_len); + + if (!IS_ERR(mr)) { + mr->device = pd->device; + mr->pd = pd; + mr->uobject = NULL; + atomic_inc(&pd->usecnt); + atomic_set(&mr->usecnt, 0); + } + + return mr; +} +EXPORT_SYMBOL(ib_alloc_fast_reg_mr); + +struct ib_fast_reg_page_list *ib_alloc_fast_reg_page_list(struct ib_device *device, + int max_page_list_len) +{ + struct ib_fast_reg_page_list *page_list; + + if (!device->alloc_fast_reg_page_list) + return ERR_PTR(-ENOSYS); + + page_list = device->alloc_fast_reg_page_list(device, max_page_list_len); + + if (!IS_ERR(page_list)) { + page_list->device = device; + page_list->max_page_list_len = max_page_list_len; + } + + return page_list; +} +EXPORT_SYMBOL(ib_alloc_fast_reg_page_list); + +void ib_free_fast_reg_page_list(struct ib_fast_reg_page_list *page_list) +{ + page_list->device->free_fast_reg_page_list(page_list); +} +EXPORT_SYMBOL(ib_free_fast_reg_page_list); + /* Memory windows */ struct ib_mw *ib_alloc_mw(struct ib_pd *pd) diff --git a/drivers/infiniband/hw/ehca/ehca_reqs.c b/drivers/infiniband/hw/ehca/ehca_reqs.c index f093b00..b799b27 100644 --- a/drivers/infiniband/hw/ehca/ehca_reqs.c +++ b/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -681,7 +681,7 @@ poll_cq_one_read_cqe: wc->dlid_path_bits = cqe->dlid; wc->src_qp = cqe->remote_qp_number; wc->wc_flags = cqe->w_completion_flags; - wc->imm_data = cpu_to_be32(cqe->immediate_data); + wc->ex.imm_data = cpu_to_be32(cqe->immediate_data); wc->sl = cqe->service_level; poll_cq_one_exit0: diff --git a/drivers/infiniband/hw/ipath/ipath_cq.c b/drivers/infiniband/hw/ipath/ipath_cq.c index a03bd28..59680c6 100644 --- a/drivers/infiniband/hw/ipath/ipath_cq.c +++ b/drivers/infiniband/hw/ipath/ipath_cq.c @@ -82,7 +82,7 @@ void ipath_cq_enter(struct ipath_cq *cq, struct ib_wc *entry, int solicited) wc->uqueue[head].opcode = entry->opcode; wc->uqueue[head].vendor_err = entry->vendor_err; wc->uqueue[head].byte_len = entry->byte_len; - wc->uqueue[head].imm_data = (__u32 __force)entry->imm_data; + wc->uqueue[head].imm_data = (__u32 __force) entry->ex.imm_data; wc->uqueue[head].qp_num = entry->qp->qp_num; wc->uqueue[head].src_qp = entry->src_qp; wc->uqueue[head].wc_flags = entry->wc_flags; diff --git a/drivers/infiniband/hw/ipath/ipath_rc.c b/drivers/infiniband/hw/ipath/ipath_rc.c index 108df66..9771052 100644 --- a/drivers/infiniband/hw/ipath/ipath_rc.c +++ b/drivers/infiniband/hw/ipath/ipath_rc.c @@ -1703,11 +1703,11 @@ void ipath_rc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr, case OP(SEND_LAST_WITH_IMMEDIATE): send_last_imm: if (header_in_data) { - wc.imm_data = *(__be32 *) data; + wc.ex.imm_data = *(__be32 *) data; data += sizeof(__be32); } else { /* Immediate data comes after BTH */ - wc.imm_data = ohdr->u.imm_data; + wc.ex.imm_data = ohdr->u.imm_data; } hdrsize += 4; wc.wc_flags = IB_WC_WITH_IMM; diff --git a/drivers/infiniband/hw/ipath/ipath_ruc.c b/drivers/infiniband/hw/ipath/ipath_ruc.c index a4b5521..af051f7 100644 --- a/drivers/infiniband/hw/ipath/ipath_ruc.c +++ b/drivers/infiniband/hw/ipath/ipath_ruc.c @@ -331,7 +331,7 @@ again: switch (wqe->wr.opcode) { case IB_WR_SEND_WITH_IMM: wc.wc_flags = IB_WC_WITH_IMM; - wc.imm_data = wqe->wr.ex.imm_data; + wc.ex.imm_data = wqe->wr.ex.imm_data; /* FALLTHROUGH */ case IB_WR_SEND: if (!ipath_get_rwqe(qp, 0)) @@ -342,7 +342,7 @@ again: if (unlikely(!(qp->qp_access_flags & IB_ACCESS_REMOTE_WRITE))) goto inv_err; wc.wc_flags = IB_WC_WITH_IMM; - wc.imm_data = wqe->wr.ex.imm_data; + wc.ex.imm_data = wqe->wr.ex.imm_data; if (!ipath_get_rwqe(qp, 1)) goto rnr_nak; /* FALLTHROUGH */ diff --git a/drivers/infiniband/hw/ipath/ipath_uc.c b/drivers/infiniband/hw/ipath/ipath_uc.c index 0596ec1..82cc588 100644 --- a/drivers/infiniband/hw/ipath/ipath_uc.c +++ b/drivers/infiniband/hw/ipath/ipath_uc.c @@ -379,11 +379,11 @@ void ipath_uc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr, case OP(SEND_LAST_WITH_IMMEDIATE): send_last_imm: if (header_in_data) { - wc.imm_data = *(__be32 *) data; + wc.ex.imm_data = *(__be32 *) data; data += sizeof(__be32); } else { /* Immediate data comes after BTH */ - wc.imm_data = ohdr->u.imm_data; + wc.ex.imm_data = ohdr->u.imm_data; } hdrsize += 4; wc.wc_flags = IB_WC_WITH_IMM; @@ -483,11 +483,11 @@ void ipath_uc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr, case OP(RDMA_WRITE_LAST_WITH_IMMEDIATE): rdma_last_imm: if (header_in_data) { - wc.imm_data = *(__be32 *) data; + wc.ex.imm_data = *(__be32 *) data; data += sizeof(__be32); } else { /* Immediate data comes after BTH */ - wc.imm_data = ohdr->u.imm_data; + wc.ex.imm_data = ohdr->u.imm_data; } hdrsize += 4; wc.wc_flags = IB_WC_WITH_IMM; diff --git a/drivers/infiniband/hw/ipath/ipath_ud.c b/drivers/infiniband/hw/ipath/ipath_ud.c index 77ca8ca..36aa242 100644 --- a/drivers/infiniband/hw/ipath/ipath_ud.c +++ b/drivers/infiniband/hw/ipath/ipath_ud.c @@ -96,7 +96,7 @@ static void ipath_ud_loopback(struct ipath_qp *sqp, struct ipath_swqe *swqe) if (swqe->wr.opcode == IB_WR_SEND_WITH_IMM) { wc.wc_flags = IB_WC_WITH_IMM; - wc.imm_data = swqe->wr.ex.imm_data; + wc.ex.imm_data = swqe->wr.ex.imm_data; } /* @@ -492,14 +492,14 @@ void ipath_ud_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr, if (qp->ibqp.qp_num > 1 && opcode == IB_OPCODE_UD_SEND_ONLY_WITH_IMMEDIATE) { if (header_in_data) { - wc.imm_data = *(__be32 *) data; + wc.ex.imm_data = *(__be32 *) data; data += sizeof(__be32); } else - wc.imm_data = ohdr->u.ud.imm_data; + wc.ex.imm_data = ohdr->u.ud.imm_data; wc.wc_flags = IB_WC_WITH_IMM; hdrsize += sizeof(u32); } else if (opcode == IB_OPCODE_UD_SEND_ONLY) { - wc.imm_data = 0; + wc.ex.imm_data = 0; wc.wc_flags = 0; } else { dev->n_pkt_drops++; diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c index 4521319..299f208 100644 --- a/drivers/infiniband/hw/mlx4/cq.c +++ b/drivers/infiniband/hw/mlx4/cq.c @@ -663,18 +663,18 @@ repoll: switch (cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) { case MLX4_RECV_OPCODE_RDMA_WRITE_IMM: - wc->opcode = IB_WC_RECV_RDMA_WITH_IMM; - wc->wc_flags = IB_WC_WITH_IMM; - wc->imm_data = cqe->immed_rss_invalid; + wc->opcode = IB_WC_RECV_RDMA_WITH_IMM; + wc->wc_flags = IB_WC_WITH_IMM; + wc->ex.imm_data = cqe->immed_rss_invalid; break; case MLX4_RECV_OPCODE_SEND: wc->opcode = IB_WC_RECV; wc->wc_flags = 0; break; case MLX4_RECV_OPCODE_SEND_IMM: - wc->opcode = IB_WC_RECV; - wc->wc_flags = IB_WC_WITH_IMM; - wc->imm_data = cqe->immed_rss_invalid; + wc->opcode = IB_WC_RECV; + wc->wc_flags = IB_WC_WITH_IMM; + wc->ex.imm_data = cqe->immed_rss_invalid; break; } diff --git a/drivers/infiniband/hw/mthca/mthca_cq.c b/drivers/infiniband/hw/mthca/mthca_cq.c index f788fce..d9f4735 100644 --- a/drivers/infiniband/hw/mthca/mthca_cq.c +++ b/drivers/infiniband/hw/mthca/mthca_cq.c @@ -620,13 +620,13 @@ static inline int mthca_poll_one(struct mthca_dev *dev, case IB_OPCODE_SEND_LAST_WITH_IMMEDIATE: case IB_OPCODE_SEND_ONLY_WITH_IMMEDIATE: entry->wc_flags = IB_WC_WITH_IMM; - entry->imm_data = cqe->imm_etype_pkey_eec; + entry->ex.imm_data = cqe->imm_etype_pkey_eec; entry->opcode = IB_WC_RECV; break; case IB_OPCODE_RDMA_WRITE_LAST_WITH_IMMEDIATE: case IB_OPCODE_RDMA_WRITE_ONLY_WITH_IMMEDIATE: entry->wc_flags = IB_WC_WITH_IMM; - entry->imm_data = cqe->imm_etype_pkey_eec; + entry->ex.imm_data = cqe->imm_etype_pkey_eec; entry->opcode = IB_WC_RECV_RDMA_WITH_IMM; break; default: diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index ce733ca..6d246e4 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -104,6 +104,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_SEND_W_INV = (1<<21), + IB_DEVICE_MEM_MGT_EXTENSIONS = (1<<22), }; enum ib_atomic_cap { @@ -149,6 +150,7 @@ struct ib_device_attr { int max_srq; int max_srq_wr; int max_srq_sge; + unsigned int max_fast_reg_page_list_len; u16 max_pkeys; u8 local_ca_ack_delay; }; @@ -412,6 +414,8 @@ enum ib_wc_opcode { IB_WC_FETCH_ADD, IB_WC_BIND_MW, IB_WC_LSO, + IB_WC_FAST_REG_MR, + IB_WC_LOCAL_INV, /* * Set value of IB_WC_RECV so consumers can test if a completion is a * receive by testing (opcode & IB_WC_RECV). @@ -422,7 +426,8 @@ enum ib_wc_opcode { enum ib_wc_flags { IB_WC_GRH = 1, - IB_WC_WITH_IMM = (1<<1) + IB_WC_WITH_IMM = (1<<1), + IB_WC_WITH_INVALIDATE = (1<<2), }; struct ib_wc { @@ -432,7 +437,10 @@ struct ib_wc { u32 vendor_err; u32 byte_len; struct ib_qp *qp; - __be32 imm_data; + union { + __be32 imm_data; + u32 invalidate_rkey; + } ex; u32 src_qp; int wc_flags; u16 pkey_index; @@ -626,6 +634,9 @@ enum ib_wr_opcode { IB_WR_ATOMIC_FETCH_AND_ADD, IB_WR_LSO, IB_WR_SEND_WITH_INV, + IB_WR_FAST_REG_MR, + IB_WR_LOCAL_INV, + IB_WR_RDMA_READ_WITH_INV, }; enum ib_send_flags { @@ -642,6 +653,12 @@ struct ib_sge { u32 lkey; }; +struct ib_fast_reg_page_list { + struct ib_device *device; + u64 *page_list; + unsigned int max_page_list_len; +}; + struct ib_send_wr { struct ib_send_wr *next; u64 wr_id; @@ -674,6 +691,16 @@ struct ib_send_wr { u16 pkey_index; /* valid for GSI only */ u8 port_num; /* valid for DR SMPs on switch only */ } ud; + struct { + u64 iova_start; + struct ib_fast_reg_page_list *page_list; + unsigned int page_shift; + unsigned int page_list_len; + unsigned int first_byte_offset; + u32 length; + int access_flags; + u32 rkey; + } fast_reg; } wr; }; @@ -1012,6 +1039,11 @@ struct ib_device { int (*query_mr)(struct ib_mr *mr, struct ib_mr_attr *mr_attr); int (*dereg_mr)(struct ib_mr *mr); + struct ib_mr * (*alloc_fast_reg_mr)(struct ib_pd *pd, + int max_page_list_len); + struct ib_fast_reg_page_list * (*alloc_fast_reg_page_list)(struct ib_device *device, + int page_list_len); + void (*free_fast_reg_page_list)(struct ib_fast_reg_page_list *page_list); int (*rereg_phys_mr)(struct ib_mr *mr, int mr_rereg_mask, struct ib_pd *pd, @@ -1806,6 +1838,54 @@ int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr); int ib_dereg_mr(struct ib_mr *mr); /** + * ib_alloc_fast_reg_mr - Allocates memory region usable with the + * IB_WR_FAST_REG_MR send work request. + * @pd: The protection domain associated with the region. + * @max_page_list_len: requested max physical buffer list length to be + * used with fast register work requests for this MR. + */ +struct ib_mr *ib_alloc_fast_reg_mr(struct ib_pd *pd, int max_page_list_len); + +/** + * ib_alloc_fast_reg_page_list - Allocates a page list array + * @device - ib device pointer. + * @page_list_len - size of the page list array to be allocated. + * + * This allocates and returns a struct ib_fast_reg_page_list * and a + * page_list array that is at least page_list_len in size. The actual + * size is returned in max_page_list_len. The caller is responsible + * for initializing the contents of the page_list array before posting + * a send work request with the IB_WC_FAST_REG_MR opcode. + * + * The page_list array entries must be translated using one of the + * ib_dma_*() functions just like the addresses passed to + * ib_map_phys_fmr(). Once the ib_post_send() is issued, the struct + * ib_fast_reg_page_list must not be modified by the caller until the + * IB_WC_FAST_REG_MR work request completes. + */ +struct ib_fast_reg_page_list *ib_alloc_fast_reg_page_list( + struct ib_device *device, int page_list_len); + +/** + * ib_free_fast_reg_page_list - Deallocates a previously allocated + * page list array. + * @page_list - struct ib_fast_reg_page_list pointer to be deallocated. + */ +void ib_free_fast_reg_page_list(struct ib_fast_reg_page_list *page_list); + +/** + * ib_update_fast_reg_key - updates the key portion of the fast_reg MR + * R_Key and L_Key. + * @mr - struct ib_mr pointer to be updated. + * @newkey - new key to be used. + */ +static inline void ib_update_fast_reg_key(struct ib_mr *mr, u8 newkey) +{ + mr->lkey = (mr->lkey & 0xffffff00) | newkey; + mr->rkey = (mr->rkey & 0xffffff00) | newkey; +} + +/** * ib_alloc_mw - Allocates a memory window. * @pd: The protection domain associated with the memory window. */ -- 1.5.5.1 From rdreier at cisco.com Fri Jun 6 12:01:08 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 12:01:08 -0700 Subject: [ofa-general] Re: mthca MR attrs userspace change In-Reply-To: <20080603204407.GA7840@osc.edu> (Pete Wyckoff's message of "Tue, 3 Jun 2008 16:44:07 -0400") References: <20080527180004.GA15444@osc.edu> <20080603204407.GA7840@osc.edu> Message-ID: > Nor am I. It never showed up in updates-testing or the real > updates, a week after you built it. I can't find any developer docs > on how this is supposed to work. Maybe someone on the list knows > what button to push? OK, I think I figured it out... I need to do a "make update" to get it submitted to the Fedora "bodhi" tool, where it gets into testing eventually (a manual process that may take some time). The request is at https://admin.fedoraproject.org/updates/F9/pending/libmthca-1.0.5-1.fc9 for the moment. - R. From dotanba at gmail.com Fri Jun 6 13:20:33 2008 From: dotanba at gmail.com (Dotan Barak) Date: Fri, 06 Jun 2008 22:20:33 +0200 Subject: [ofa-general] IB/ehca: Reject send WRs only for RESET, INIT and RTR state In-Reply-To: <200806061835.43802.fenkes@de.ibm.com> References: <200806061835.43802.fenkes@de.ibm.com> Message-ID: <48499C11.7030504@gmail.com> Hi. I checked the code in the ehca driver and noticed that post RR to a QP is being accepted in any state (including the RESET state). thanks Dotan Joachim Fenkes wrote: > Signed-off-by: Joachim Fenkes > --- > drivers/infiniband/hw/ehca/ehca_reqs.c | 6 ++++-- > 1 files changed, 4 insertions(+), 2 deletions(-) > > diff --git a/drivers/infiniband/hw/ehca/ehca_reqs.c b/drivers/infiniband/hw/ehca/ehca_reqs.c > index bbe0436..f093b00 100644 > --- a/drivers/infiniband/hw/ehca/ehca_reqs.c > +++ b/drivers/infiniband/hw/ehca/ehca_reqs.c > @@ -421,8 +421,10 @@ int ehca_post_send(struct ib_qp *qp, > int ret = 0; > unsigned long flags; > > - if (unlikely(my_qp->state != IB_QPS_RTS)) { > - ehca_err(qp->device, "QP not in RTS state qpn=%x", qp->qp_num); > + /* Reject WR if QP is in RESET, INIT or RTR state */ > + if (unlikely(my_qp->state < IB_QPS_RTS)) { > + ehca_err(qp->device, "Invalid QP state qp_state=%d qpn=%x", > + my_qp->state, qp->qp_num); > return -EINVAL; > } > > From moonwort at redmondesign.com Fri Jun 6 13:00:04 2008 From: moonwort at redmondesign.com (Trent Comer) Date: Fri, 06 Jun 2008 22:00:04 +0200 Subject: [ofa-general] ~After Effects CS 3 Pro~ Message-ID: <000601c8c80f$2328cc00$0100007f@erxnvge> ~ Adobe CS3 Master Collection for PC or MAC includes: ~ InDesign CS3 ~ Photoshop CS3 ~ Illustrator CS3 ~ Acrobat 8 Professional ~ Flash CS3 Professional ~ Dreamweaver CS3 ~ Fireworks CS3 ~ Contribute CS3 ~ After Effects CS3 Professional ~ Premiere Pro CS3 ~ Encore DVD CS3 ~ Soundbooth CS3 ~ gethotsoft. com in Internet browser ~ System Requirements ~ For PC: ~ Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core ~ Duo (or compatible) processor; SSE2-enabled processor required for AMD systems ~ Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) ~ 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ~ 38GB of available hard-disk space (additional free space required during installation) ~ Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ~ Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) ~ 1,280x1,024 monitor resolution with 32-bit color adapter ~ DVD-ROM drive ~ For MAC: ~ PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) ~ Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server ~ 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ~ 36GB of available hard-disk space (additional free space required during installation) ~ Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ~ Core Audio compatible sound card ~ 1,280x1,024 monitor resolution with 32-bit color adapter ~ DVD-ROM drive~ DVD+-R burner required for DVD creation Watermelon Gets Premium Price in Japan Homeland Security officials will take a second look at the case of a Canadian seized in New York as a terrorism suspect and sent to Syria, where he was allegedly tortured. House Democrats want the Justice Department to conduct a criminal probe into the 2002 case. From rdreier at cisco.com Fri Jun 6 14:33:54 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 14:33:54 -0700 Subject: [ofa-general][PATCH 3/12 v1] mlx4: Qp range reservation In-Reply-To: <480F4E10.9080203@mellanox.co.il> (Yevgeny Petrilin's message of "Wed, 23 Apr 2008 17:56:16 +0300") References: <480F4E10.9080203@mellanox.co.il> Message-ID: > +static unsigned long find_aligned_range(unsigned long *bitmap, > + u32 start, u32 nbits, > + int len, int align) Seems that a lot of this code can be replaced by a call to bitmap_find_free_region()? - R. From rdreier at cisco.com Fri Jun 6 14:40:27 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 14:40:27 -0700 Subject: [ofa-general][PATCH 4/12 v2] mlx4: Pre reserved Qp regions In-Reply-To: <480F4E98.7010803@mellanox.co.il> (Yevgeny Petrilin's message of "Wed, 23 Apr 2008 17:58:32 +0300") References: <480F4E98.7010803@mellanox.co.il> Message-ID: > +int mlx4_bitmap_init_with_effective_max(struct mlx4_bitmap *bitmap, > + u32 num, u32 mask, u32 reserved, > + u32 effective_max) I don't really understand what effective_max is buying you here... it seems that the caller could just reduce the real max and get the same effect without tracking two different values? It's hard to say without seeing the ethernet code, but would it make more sense to add support to the allocator to allocate from multiple regions? > +static int num_mac = 1; > +module_param_named(num_mac, num_mac, int, 0444); > +MODULE_PARM_DESC(num_mac, "Maximum number of MACs per ETH port " > + "(1-127, default 1)"); > + > +static int num_vlan; > +module_param_named(num_vlan, num_vlan, int, 0444); > +MODULE_PARM_DESC(num_vlan, "Maximum number of VLANs per ETH port " > + "(0-126, default 0)"); Jack is pushing to make these types of module parameters specified using log2 of the value instead of the absolute value -- seems you guys should agree on the interface you want to use. From jschoeny at hcinet.net Fri Jun 6 18:38:10 2008 From: jschoeny at hcinet.net (Yahoo/Msn Lottery) Date: Fri, 6 Jun 2008 21:38:10 -0400 Subject: [ofa-general] Winning Notification Message-ID: <20080607013810.TAIG4203.invictus@[192.168.1.30]> Yahoo/Msn Lottery Incoperation, Baley House, Har Road Sutton, Greater London SM1 4te,United Kingdom. TO WINNERS IN OUR PROGRAM This is to inform you that you have won a prize money of Five hundred thousand, Great Britain Pound Sterlings(£500,000.00) for the month of JUNE 2008 Lottery promotion which is organized by YAHOO/MSN LOTTERY INC &WINDOWS LIVE. YAHOO/MSN & MICROSOFT WINDOWS, collects all the email addresses of the people that are active online, among the millions that subscribed to INTERNET we only select five people every Month as our winners through electronic ballotingSystem without the winner applying,we congratulate you for being one of the people selected. You are to contact the events manager on or before your date of Claim, Winners Shall be paid in accordance with his/her Settlement Centre. Yahoo/Msn Lottery Prize must be claimed no later than 10 days from date of Draw Notification afterthe Draw date in which Prize has won. Any prize not claimed within this period will be forfeited. These are your identification numbers: Batch number........................YM 09102XN Reff number.........................YM35447XN Winning number......................YM09788 These numbers fall within your Location file, you are requested to contact the events manager, send your winning identification numbers to him,to enable him verify your claims.How ever you will have to fill and submit this form to the events manager for verification & direction on how you can claim your winning fund. 1. Full name............. 2. Contact Address................ 3. Age................... 4. Telephone Number............... 5. Marital Status........ 6. Sex............................ 7. Next Of Kin........... 8. Zip Code....................... 9. Occupation............ 10. Company....................... 11.State:................ 12.Country........................ 13.Nationality........... 14.Your Reference and Batch number are at the top of this mail: CONTACT EVENTS MANAGER) Name:Dr.Christopher Mcdonald Email :claims_macdonalds at yahoo.co.uk Thank you and Accept my hearty congratulations once again! Yours faithfully, Mrs.Teresa Billington (Yahoo/Msn Lottery Games/Lottery Coordinator). From dimmerosv at a-beyer-plakate.de Fri Jun 6 20:49:56 2008 From: dimmerosv at a-beyer-plakate.de (Dalton Huggins) Date: Sat, 7 Jun 2008 00:49:56 -0300 Subject: [ofa-general] Those young people taking the position you deserve because you lack a Degree? yr vmzm ahiciq Message-ID: <01c8c838$677c9200$6a0792c8@dimmerosv> A Gen srj uine Un ypb ivers gop ity Deg fqc ree in 4-6 we dvu eks! Have you ever thought that the only thing stopping you from a great j si ob and better pay was a few let na ters behind your name? Well now you can get them! BA___BSc___MA___MSc___MBA___PhD Within 4-6 weeks! No St prg udy Re fm qui gbn red! 100% Ver yl ifia xlh ble! These are real, ge yhr nuine degrees that include Bac xz helors, Ma iu sters, MBA and Do avg ctorate Deg km rees. They are fully ve rh rifia yxj ble and ce mk rtif nh ied transcripts are available. Just ca xnz ll the number below. You will thank me later… 1-5 jq 20-2 nzp 03-0 ggk 21 fbx 3Leave a message 24 hours a day, 7 days a week including Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Fri Jun 6 22:21:18 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Jun 2008 22:21:18 -0700 Subject: [ofa-general] [PATCH] IB/umem: Avoid sign problems when demoting npages to integer Message-ID: On a 64-bit architecture, if ib_umem_get() is called with a size value that is so big that npages is negative when cast to int, then the length of the page list passed to get_user_pages(), namely min_t(int, npages, PAGE_SIZE / sizeof (struct page *)) will be negative, and get_user_pages() will immediately return 0 (at least since 900cf086, "Be more robust about bad arguments in get_user_pages()"). This leads to an infinite loop in ib_umem_get(), since the code boils down to: while (npages) { ret = get_user_pages(...); npages -= ret; } Fix this by taking the minimum as unsigned longs, so that the value of npages is never truncated. The impact of this bug isn't too severe, since the value of npages is checked against RLIMIT_MEMLOCK, so a process would need to have an astronomical limit or have CAP_IPC_LOCK to be able to trigger this, and such a process could already cause lots of mischief. But it does let buggy userspace code cause a kernel lock-up; for example I hit this with code that passes a negative value into a memory registartion function where it is promoted to a huge u64 value. Cc: Signed-off-by: Roland Dreier --- One I hit while debugging my own buggy userspace code. Definitely should go into 2.6.26 and I think -stable too. drivers/infiniband/core/umem.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index fe78f7d..a1768db 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -150,7 +150,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, ret = 0; while (npages) { ret = get_user_pages(current, current->mm, cur_base, - min_t(int, npages, + min_t(unsigned long, npages, PAGE_SIZE / sizeof (struct page *)), 1, !umem->writable, page_list, vma_list); -- 1.5.5.1 From hrosenstock at xsigo.com Sat Jun 7 06:03:17 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sat, 07 Jun 2008 06:03:17 -0700 Subject: [ofa-general] [RFC] OpenSM and IPv6 Scalability Proposal Message-ID: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> There have been several discussions on SM issues with IPv6 solicited node multicast (SNM) scalability. There was a thread entitled "IPv6 and IPoIB scalability issue" (http://lists.openfabrics.org/pipermail/general/2006-November/029621.html) and a couple of subsequent threads on a workaround. It is proposed here to remove the workaround and replace it with a complete solution for this issue. PROBLEM STATEMENT The primary issue is that IPv6 SNM traded off using separate multicast groups (rather than broadcast) for performing neighbor discovery (ND). SM's that utilize a simple scheme of a 1:1 mapping of multicast group (MGID) to multicast LID (MLID) consume too many MLIDs in large clusters. CURRENT DESIGN There is a current workaround in OpenSM for this is an option called consolidate_ipv6_snm_req. This workaround attempts to compress the IPv6 SNM groups to 1 MLID. Limitations of this workaround have been discussed on this list previously. Underlying this, the current OpenSM design assumes a 1:1 mapping of multicast group to MLID. It currently utilizes a "quick" map, which is a red/black tree, supporting up through 64 bit keys. Unfortunately, multicast group (MGID) is a 128 bit key. PROPOSED APPROACH The approach is in 2 steps: 1. Change the current underlying multicast tree from being MLID based to MGID based. This involves using fleximap rather than qmap. The downside of this is that MLID lookups will be slower as now they are not as "direct" as the MLID will no longer be the key in the map. Rather than searching by MLID key, the tree will need to be scanned entry by entry for MLID matches. It's unclear how much this will slow down MLID searches but it is thought that none of these searches are time critical (and shouldn't cause any existing timeouts to "pop"). 2. Add in support for overloading MLIDs. On the configuration side, a number of additional options would be added to consolidate_ipv6_snm_req. These include the number of MLIDs to compress down to (default 16), a multicast group (MGID) base address and (full MGID) mask. this would default to 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY where Z is the scope, XXXX is the P_Key, and YYYYYY is the last 24 bits of the port guid ( the YYYYYY bits would be masked out by default). This is what the current workaround uses for collapsing the multicast groups. The criteria for overloading MLIDs includes any group parameters that need to be in common (e.g. rate. MTU, perhaps PKey (see below), etc.). Aside from changing the underlying implementation of MLID searches, multicast group deletion wll need another check when there are no ports left in a group. If that group is on a compressed MLID (this part of the check is an optimization), then the multicast group tree needs to be checked to ensure there are no other groups sharing that MLID. IBA 1.2.1 v1 p.151 4.1.3 Local Identifiers item 10) states: "When a multicast LID is overloaded, the multicast groups sharing the same MLID must have the same P_Key. This simplification is required to allow switches and routers that implement optional P_Key enforcement for multicast operations." This is part of the C4-5 compliance. OPEN ISSUE As PKey is part of the MGID, does this need to be addressed (and if so) how ? More on the above as I get further. If the approach above seems reasonable, I will work on such a set of patches. Comments ? Thoughts ? -- Hal From hrosenstock at xsigo.com Sat Jun 7 06:03:28 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sat, 07 Jun 2008 06:03:28 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/osm_sa_mcmember_record.c: Comment reformatting Message-ID: <1212843808.6127.244.camel@hrosenstock-ws.xsigo.com> OpenSM/osm_sa_mcmember_record.c: Comment reformatting Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 3cfd5f7..1693507 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -111,9 +111,9 @@ static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, } /********************************************************************* -Copy certain fields between two mcmember records -used during the process of join request to copy data from the mgrp to the -port record. + Copy certain fields between two mcmember records + used during the process of join request to copy data from the mgrp + to the port record. **********************************************************************/ static inline void __copy_from_create_mc_rec(IN ib_member_rec_t * const dest, @@ -130,9 +130,9 @@ __copy_from_create_mc_rec(IN ib_member_rec_t * const dest, } /********************************************************************* -Return an mlid to the pool of free mlids. -But this implementation is not a pool - it simply scans through -the MGRP database for unused mlids... + Return mlid to the pool of free mlids. + But this implementation is not a pool - it simply scans through + the MGRP database for unused mlids... *********************************************************************/ static void __free_mlid(IN osm_sa_t * sa, IN uint16_t mlid) { @@ -141,9 +141,9 @@ static void __free_mlid(IN osm_sa_t * sa, IN uint16_t mlid) } /********************************************************************* -Get a new unused mlid by scanning all the used ones in the subnet. -TODO: Implement a more scalable - O(1) solution based on pool of -available mlids. + Get a new unused mlid by scanning all the used ones in the subnet. + TODO: Implement a more scalable - O(1) solution based on pool of + available mlids. **********************************************************************/ static ib_net16_t __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) @@ -166,7 +166,7 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) goto Exit; } - /* If MCGroups table empty, first return the min mlid */ + /* If MCGroups table is empty, first return the min mlid */ p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { mlid = IB_LID_MCAST_START_HO; @@ -236,11 +236,11 @@ Exit: } /********************************************************************* -This procedure is only invoked to cleanup an INTERMEDIATE mgrp. -If there is only one port on the mgrp it means that the current -request was the only member and the group is not really needed. So we -silently drop it. Since it was an intermediate group no need to -re-route it. + This procedure is only invoked to cleanup an INTERMEDIATE mgrp. + If there is only one port on the mgrp it means that the current + request was the only member and the group is not really needed. So + we silently drop it. Since it was an intermediate group no need to + re-route it. **********************************************************************/ static void __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) @@ -258,8 +258,8 @@ __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) } /********************************************************************* -Add a port to the group. Calculating its PROXY_JOIN by the Port and -requester gids. + Add a port to the group. Calculating its PROXY_JOIN by the Port and + requester gids. **********************************************************************/ static ib_api_status_t __add_new_mgrp_port(IN osm_sa_t * sa, @@ -329,7 +329,7 @@ __check_create_comp_mask(ib_net64_t comp_mask, } /********************************************************************** -Generate the response MAD + Generate the response MAD **********************************************************************/ static void __osm_mcmr_rcv_respond(IN osm_sa_t * sa, @@ -368,9 +368,9 @@ Exit: } /********************************************************************* -In joining an existing group, or when querying the mc groups, -we make sure the following components provided match: MTU and RATE -HACK: Currently we ignore the PKT_LIFETIME field. + In joining an existing group, or when querying the mc groups, + we make sure the following components provided match: MTU and RATE + HACK: Currently we ignore the PKT_LIFETIME field. **********************************************************************/ static boolean_t __validate_more_comp_fields(osm_log_t * p_log, @@ -460,8 +460,8 @@ __validate_more_comp_fields(osm_log_t * p_log, } /********************************************************************* -In joining an existing group, we make sure the following components -are physically realizable: MTU and RATE + In joining an existing group, we make sure the following components + are physically realizable: MTU and RATE **********************************************************************/ static boolean_t __validate_port_caps(osm_log_t * const p_log, @@ -796,12 +796,14 @@ __mgrp_request_is_realizable(IN osm_sa_t * sa, /* * End of o15-0.2.3 specifies: * .... - * The entity may also supply the other components such as HopLimit, MTU, - * etc. during group creation time. If these components are not provided - * during group creation time, SA will provide them for the group. The values - * chosen are vendor-dependent and beyond the scope of the specification. + * The entity may also supply the other components such as HopLimit, + * MTU, etc. during group creation time. If these components are not + * provided during group creation time, SA will provide them for the + * group. The values chosen are vendor-dependent and beyond the scope + * of the specification. * - * so we might also need to assign RATE/MTU if they are not comp masked in. + * so we might also need to assign RATE/MTU if they are not comp + * masked in. */ port_mtu = p_physp ? ib_port_info_get_mtu_cap(&p_physp->port_info) : 0; From Targeted at lists.openfabrics.org Sat Jun 7 07:05:32 2008 From: Targeted at lists.openfabrics.org (Targeted at lists.openfabrics.org) Date: 07 Jun 2008 07:05:32 -0700 Subject: [ofa-general] How to get free quality visitors to your website? Message-ID: <20080607070532.A07D3718C4682D08@from.header.has.no.domain> No Matter what you are selling - Hit-Booster will send targeted visitors to your website! Within 15 minutes you will have your own website traffic generator that will bring in an ever increasing amount of hits to your websites! Automatically This software is perfect for bringing real traffic to your site... even if... it's an affiliate link where you have no control over the website content! For Full Details please read the attached .html file Unsubscribe: on attached html file click on contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From Targeted at lists.openfabrics.org Sat Jun 7 07:06:09 2008 From: Targeted at lists.openfabrics.org (Targeted at lists.openfabrics.org) Date: 07 Jun 2008 07:06:09 -0700 Subject: [ofa-general] How to get free quality visitors to your website? Message-ID: <20080607070608.AF3EBD6934C5A8D0@from.header.has.no.domain> No Matter what you are selling - Hit-Booster will send targeted visitors to your website! Within 15 minutes you will have your own website traffic generator that will bring in an ever increasing amount of hits to your websites! Automatically This software is perfect for bringing real traffic to your site... even if... it's an affiliate link where you have no control over the website content! For Full Details please read the attached .html file Unsubscribe: on attached html file click on contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Sat Jun 7 10:30:46 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sat, 07 Jun 2008 10:30:46 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM: Remove some vestigial comments Message-ID: <1212859846.14047.29.camel@hrosenstock-ws.xsigo.com> OpenSM: Remove some vestigial comments Signed-off-by: Hal Rosenstock diff --git a/opensm/include/complib/cl_atomic.h b/opensm/include/complib/cl_atomic.h index ceacd8e..92620ee 100644 --- a/opensm/include/complib/cl_atomic.h +++ b/opensm/include/complib/cl_atomic.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of atomic manipulation functions. - * - * Environment: - * All - * - * $Revision: 1.4 $ */ #ifndef _CL_ATOMIC_H_ diff --git a/opensm/include/complib/cl_atomic_osd.h b/opensm/include/complib/cl_atomic_osd.h index 114c01a..ac14f8a 100644 --- a/opensm/include/complib/cl_atomic_osd.h +++ b/opensm/include/complib/cl_atomic_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Implementation specific header files for atomic operations. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #ifndef _CL_ATOMIC_OSD_H_ diff --git a/opensm/include/complib/cl_byteswap.h b/opensm/include/complib/cl_byteswap.h index 6a1c9bf..bef24b0 100644 --- a/opensm/include/complib/cl_byteswap.h +++ b/opensm/include/complib/cl_byteswap.h @@ -37,11 +37,6 @@ * Abstract: * provides byteswapping utilities. Basic functions are obtained from * platform specific implementations from byteswap_osd.h. - * - * Environment: - * All - * - * $Revision: 1.5 $ */ #ifndef _CL_BYTESWAP_H_ diff --git a/opensm/include/complib/cl_byteswap_osd.h b/opensm/include/complib/cl_byteswap_osd.h index 37e821d..72ff40e 100644 --- a/opensm/include/complib/cl_byteswap_osd.h +++ b/opensm/include/complib/cl_byteswap_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Provides common macros for dealing with byte swapping issues. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #ifndef _CL_BYTESWAP_OSD_H_ diff --git a/opensm/include/complib/cl_comppool.h b/opensm/include/complib/cl_comppool.h index c891cc1..a9c93f2 100644 --- a/opensm/include/complib/cl_comppool.h +++ b/opensm/include/complib/cl_comppool.h @@ -39,11 +39,6 @@ * The composite pool managers a pool of composite objects. A composite object is an object * that is made of multiple sub objects. * The pool can grow to meet demand, limited only by system memory. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_COMP_POOL_H_ diff --git a/opensm/include/complib/cl_debug.h b/opensm/include/complib/cl_debug.h index fbca7bd..05e9769 100644 --- a/opensm/include/complib/cl_debug.h +++ b/opensm/include/complib/cl_debug.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of functions for reporting debug output. - * - * Environment: - * All - * - * $Revision: 1.4 $ */ #ifndef _CL_DEBUG_H_ diff --git a/opensm/include/complib/cl_debug_osd.h b/opensm/include/complib/cl_debug_osd.h index 943a360..31d6972 100644 --- a/opensm/include/complib/cl_debug_osd.h +++ b/opensm/include/complib/cl_debug_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Debug Macros. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #ifndef _CL_DEBUG_OSD_H_ diff --git a/opensm/include/complib/cl_dispatcher.h b/opensm/include/complib/cl_dispatcher.h index 70ba4d4..81aa06f 100644 --- a/opensm/include/complib/cl_dispatcher.h +++ b/opensm/include/complib/cl_dispatcher.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of dispatcher abstraction. - * - * Environment: - * All - * - * $Revision: 1.4 $ */ #ifndef _CL_DISPATCHER_H_ diff --git a/opensm/include/complib/cl_event.h b/opensm/include/complib/cl_event.h index ae9c123..10805fb 100644 --- a/opensm/include/complib/cl_event.h +++ b/opensm/include/complib/cl_event.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of event abstraction. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_EVENT_H_ diff --git a/opensm/include/complib/cl_event_osd.h b/opensm/include/complib/cl_event_osd.h index 92f3da3..541ced0 100644 --- a/opensm/include/complib/cl_event_osd.h +++ b/opensm/include/complib/cl_event_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of event object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.3 $ */ #ifndef _CL_EVENT_OSD_H_ diff --git a/opensm/include/complib/cl_event_wheel.h b/opensm/include/complib/cl_event_wheel.h index ba466c6..ac02242 100644 --- a/opensm/include/complib/cl_event_wheel.h +++ b/opensm/include/complib/cl_event_wheel.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of event wheel abstraction. - * - * Environment: - * All - * - * $Revision: 1.4 $ */ #ifndef _CL_EVENT_WHEEL_H_ diff --git a/opensm/include/complib/cl_fleximap.h b/opensm/include/complib/cl_fleximap.h index 423d99a..c9912f1 100644 --- a/opensm/include/complib/cl_fleximap.h +++ b/opensm/include/complib/cl_fleximap.h @@ -37,11 +37,6 @@ * Abstract: * Declaration of flexi map, a binary tree where the caller always provides * all necessary storage. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_FLEXIMAP_H_ diff --git a/opensm/include/complib/cl_list.h b/opensm/include/complib/cl_list.h index 353cb70..9c6310a 100644 --- a/opensm/include/complib/cl_list.h +++ b/opensm/include/complib/cl_list.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of list. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_LIST_H_ diff --git a/opensm/include/complib/cl_log.h b/opensm/include/complib/cl_log.h index d790ed3..45c7ec3 100644 --- a/opensm/include/complib/cl_log.h +++ b/opensm/include/complib/cl_log.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of logging mechanisms. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_LOG_H_ diff --git a/opensm/include/complib/cl_map.h b/opensm/include/complib/cl_map.h index a1a48f9..5bf779c 100644 --- a/opensm/include/complib/cl_map.h +++ b/opensm/include/complib/cl_map.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of map, a binary tree. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_MAP_H_ diff --git a/opensm/include/complib/cl_math.h b/opensm/include/complib/cl_math.h index 74f63da..47489bd 100644 --- a/opensm/include/complib/cl_math.h +++ b/opensm/include/complib/cl_math.h @@ -36,11 +36,6 @@ /* * Abstract: * Defines standard math related macros and functions. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_MATH_H_ diff --git a/opensm/include/complib/cl_packoff.h b/opensm/include/complib/cl_packoff.h index 7b5ab9b..52ee381 100644 --- a/opensm/include/complib/cl_packoff.h +++ b/opensm/include/complib/cl_packoff.h @@ -38,11 +38,6 @@ * Turns off byte packing, which is necessary for passing information from * system to system over a network to ensure no padding by the compiler has * taken place. - * - * Environment: - * Linux platforms (User mode) - * - * $Revision: 1.3 $ */ #ifdef PACK_SUFFIX diff --git a/opensm/include/complib/cl_packon.h b/opensm/include/complib/cl_packon.h index 239e5ee..ffc8e11 100644 --- a/opensm/include/complib/cl_packon.h +++ b/opensm/include/complib/cl_packon.h @@ -38,11 +38,6 @@ * Turns on byte packing, which is necessary for passing information from * system to system over a network to ensure no padding by the compiler has * taken place. - * - * Environment: - * Linux platforms (User mode) - * - * $Revision: 1.3 $ */ /****h* Component Library/Structure Packing diff --git a/opensm/include/complib/cl_passivelock.h b/opensm/include/complib/cl_passivelock.h index b6ba6c8..bafd339 100644 --- a/opensm/include/complib/cl_passivelock.h +++ b/opensm/include/complib/cl_passivelock.h @@ -39,11 +39,6 @@ * The passive lock allows multiple readers to access a resource * simultaneously, exclusive from a single thread allowed writing. * Several writer threads are allowed - but only one can write at a given time - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_PASSIVE_LOCK_H_ diff --git a/opensm/include/complib/cl_pool.h b/opensm/include/complib/cl_pool.h index 435db48..a9358fd 100644 --- a/opensm/include/complib/cl_pool.h +++ b/opensm/include/complib/cl_pool.h @@ -38,11 +38,6 @@ * Declaration of the pool. * The pool manages a pool of objects. * The pool can grow to meet demand, limited only by system memory. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_POOL_H_ diff --git a/opensm/include/complib/cl_ptr_vector.h b/opensm/include/complib/cl_ptr_vector.h index 36df42e..70e4f00 100644 --- a/opensm/include/complib/cl_ptr_vector.h +++ b/opensm/include/complib/cl_ptr_vector.h @@ -37,11 +37,6 @@ * Abstract: * This file contains pointer vector definitions. Pointer Vector provides * dynmically resizable array functionality. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_PTR_VECTOR_H_ diff --git a/opensm/include/complib/cl_qcomppool.h b/opensm/include/complib/cl_qcomppool.h index b58c977..9fc2570 100644 --- a/opensm/include/complib/cl_qcomppool.h +++ b/opensm/include/complib/cl_qcomppool.h @@ -39,11 +39,6 @@ * manages a pool of composite objects. A composite object is an object * that is made of multiple sub objects. * It can grow to meet demand, limited only by system memory. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_QUICK_COMPOSITE_POOL_H_ diff --git a/opensm/include/complib/cl_qlist.h b/opensm/include/complib/cl_qlist.h index a51b357..9804b85 100644 --- a/opensm/include/complib/cl_qlist.h +++ b/opensm/include/complib/cl_qlist.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of quick list. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_QUICK_LIST_H_ diff --git a/opensm/include/complib/cl_qmap.h b/opensm/include/complib/cl_qmap.h index caf6f0e..75b30d3 100644 --- a/opensm/include/complib/cl_qmap.h +++ b/opensm/include/complib/cl_qmap.h @@ -37,11 +37,6 @@ * Abstract: * Declaration of quick map, a binary tree where the caller always provides * all necessary storage. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_QMAP_H_ diff --git a/opensm/include/complib/cl_qpool.h b/opensm/include/complib/cl_qpool.h index 5a38206..ba4eacc 100644 --- a/opensm/include/complib/cl_qpool.h +++ b/opensm/include/complib/cl_qpool.h @@ -38,11 +38,6 @@ * Declaration of the quick pool. * The quick pool manages a pool of objects. * The pool can grow to meet demand, limited only by system memory. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_QUICK_POOL_H_ diff --git a/opensm/include/complib/cl_spinlock.h b/opensm/include/complib/cl_spinlock.h index 2e4cdf3..1a04b56 100644 --- a/opensm/include/complib/cl_spinlock.h +++ b/opensm/include/complib/cl_spinlock.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of spin lock object. - * - * Environment: - * All - * - * $Revision: 1.4 $ */ #ifndef _CL_SPINLOCK_H_ diff --git a/opensm/include/complib/cl_spinlock_osd.h b/opensm/include/complib/cl_spinlock_osd.h index bc236c5..beb6405 100644 --- a/opensm/include/complib/cl_spinlock_osd.h +++ b/opensm/include/complib/cl_spinlock_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of spin lock object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _CL_SPINLOCK_OSD_H_ diff --git a/opensm/include/complib/cl_thread.h b/opensm/include/complib/cl_thread.h index 38ffc93..eaa784f 100644 --- a/opensm/include/complib/cl_thread.h +++ b/opensm/include/complib/cl_thread.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of thread abstraction and thread related operations. - * - * Environment: - * All - * - * $Revision: 1.7 $ */ #ifndef _CL_THREAD_H_ diff --git a/opensm/include/complib/cl_thread_osd.h b/opensm/include/complib/cl_thread_osd.h index 12fbf1a..6be9a12 100644 --- a/opensm/include/complib/cl_thread_osd.h +++ b/opensm/include/complib/cl_thread_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of thread abstraction and thread related operations. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _CL_THREAD_OSD_H_ diff --git a/opensm/include/complib/cl_threadpool.h b/opensm/include/complib/cl_threadpool.h index bd4648a..e9b2c78 100644 --- a/opensm/include/complib/cl_threadpool.h +++ b/opensm/include/complib/cl_threadpool.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of thread pool. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_THREAD_POOL_H_ diff --git a/opensm/include/complib/cl_timer.h b/opensm/include/complib/cl_timer.h index 0457536..194e374 100644 --- a/opensm/include/complib/cl_timer.h +++ b/opensm/include/complib/cl_timer.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of timer abstraction. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_TIMER_H_ diff --git a/opensm/include/complib/cl_timer_osd.h b/opensm/include/complib/cl_timer_osd.h index 9d5b49a..ed36fea 100644 --- a/opensm/include/complib/cl_timer_osd.h +++ b/opensm/include/complib/cl_timer_osd.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of timer object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.3 $ */ #ifndef _CL_TIMER_OSD_H_ diff --git a/opensm/include/complib/cl_types.h b/opensm/include/complib/cl_types.h index 6969fdd..283075f 100644 --- a/opensm/include/complib/cl_types.h +++ b/opensm/include/complib/cl_types.h @@ -36,11 +36,6 @@ /* * Abstract: * Defines standard return codes, keywords, macros, and debug levels. - * - * Environment: - * All supported platforms - * - * $Revision: 1.4 $ */ #ifdef __WIN__ diff --git a/opensm/include/complib/cl_types_osd.h b/opensm/include/complib/cl_types_osd.h index fe7e829..d12aa4c 100644 --- a/opensm/include/complib/cl_types_osd.h +++ b/opensm/include/complib/cl_types_osd.h @@ -39,11 +39,6 @@ * exported sizes are int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t * int64_t, uint64_t. uintn_t is a polymorphic type, size is native size and * also size of the pointer. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #ifndef _CL_TYPES_OSD_H_ diff --git a/opensm/include/complib/cl_vector.h b/opensm/include/complib/cl_vector.h index e1e8ca7..8c7fa0a 100644 --- a/opensm/include/complib/cl_vector.h +++ b/opensm/include/complib/cl_vector.h @@ -38,11 +38,6 @@ * This file contains vector definitions. Vector provides dynmically * resizable array functionality. Objects in a Vector are not relocated * when the array is resized. - * - * Environment: - * All - * - * $Revision: 1.3 $ */ #ifndef _CL_VECTOR_H_ diff --git a/opensm/include/opensm/osm_attrib_req.h b/opensm/include/opensm/osm_attrib_req.h index be993a5..4cf1ce3 100644 --- a/opensm/include/opensm/osm_attrib_req.h +++ b/opensm/include/opensm/osm_attrib_req.h @@ -53,11 +53,6 @@ BEGIN_C_DECLS * encapsulates information needed by the generic request controller * to request an attribute from a node. * These objects are part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ /****h* OpenSM/Attribute Request * NAME diff --git a/opensm/include/opensm/osm_base.h b/opensm/include/opensm/osm_base.h index 43ec033..d07ad8c 100644 --- a/opensm/include/opensm/osm_base.h +++ b/opensm/include/opensm/osm_base.h @@ -38,11 +38,6 @@ * Basic OpenSM definitions and structures. * This object represents an OpenSM "base class". * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.15 $ */ #ifndef _OSM_BASE_H_ diff --git a/opensm/include/opensm/osm_console_io.h b/opensm/include/opensm/osm_console_io.h index a70ace8..f8f10a5 100644 --- a/opensm/include/opensm/osm_console_io.h +++ b/opensm/include/opensm/osm_console_io.h @@ -35,11 +35,6 @@ * Declaration of osm_console_t. * This object represents the OpenSM Console object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.0 $ */ #ifndef _OSM_CONSOLE_IO_H_ diff --git a/opensm/include/opensm/osm_db.h b/opensm/include/opensm/osm_db.h index eb05f45..a35bae8 100644 --- a/opensm/include/opensm/osm_db.h +++ b/opensm/include/opensm/osm_db.h @@ -39,8 +39,6 @@ /* * Abstract: * Declaration of the DB interface. - * - * $Revision: 1.4 $ */ #include diff --git a/opensm/include/opensm/osm_errors.h b/opensm/include/opensm/osm_errors.h index 2c7a9ba..aff4300 100644 --- a/opensm/include/opensm/osm_errors.h +++ b/opensm/include/opensm/osm_errors.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of error code ranges for the various OpenSM modules. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_ERRORS_H_ diff --git a/opensm/include/opensm/osm_fwd_tbl.h b/opensm/include/opensm/osm_fwd_tbl.h index 789d633..2bdd8b7 100644 --- a/opensm/include/opensm/osm_fwd_tbl.h +++ b/opensm/include/opensm/osm_fwd_tbl.h @@ -38,11 +38,6 @@ * Declaration of osm_fwd_tbl_t. * This object represents a unicast forwarding table. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_FWD_TBL_H_ diff --git a/opensm/include/opensm/osm_helper.h b/opensm/include/opensm/osm_helper.h index bd885d7..09dd5b8 100644 --- a/opensm/include/opensm/osm_helper.h +++ b/opensm/include/opensm/osm_helper.h @@ -55,11 +55,6 @@ BEGIN_C_DECLS /* * Abstract: * Declaration of helpful functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ /****f* OpenSM: Helper/ib_get_sa_method_str * NAME diff --git a/opensm/include/opensm/osm_inform.h b/opensm/include/opensm/osm_inform.h index 5da513e..67d1c61 100644 --- a/opensm/include/opensm/osm_inform.h +++ b/opensm/include/opensm/osm_inform.h @@ -39,13 +39,8 @@ * This object represents an IBA Inform Record. * This object is part of the OpenSM family of objects. * - * Environment: - * Linux User Mode - * * Author: * Eitan Zahavi, Mellanox - * - * $Revision: 1.5 $ */ #ifndef _OSM_INFR_H_ diff --git a/opensm/include/opensm/osm_lid_mgr.h b/opensm/include/opensm/osm_lid_mgr.h index 15b230f..7d6b84c 100644 --- a/opensm/include/opensm/osm_lid_mgr.h +++ b/opensm/include/opensm/osm_lid_mgr.h @@ -38,11 +38,6 @@ * Declaration of osm_lid_mgr_t. * This object represents the LID Manager object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_LID_MGR_H_ diff --git a/opensm/include/opensm/osm_lin_fwd_tbl.h b/opensm/include/opensm/osm_lin_fwd_tbl.h index c3db69a..6c64c09 100644 --- a/opensm/include/opensm/osm_lin_fwd_tbl.h +++ b/opensm/include/opensm/osm_lin_fwd_tbl.h @@ -38,11 +38,6 @@ * Declaration of osm_lin_fwd_tbl_t. * This object represents a linear forwarding table. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_LIN_FWD_TBL_H_ diff --git a/opensm/include/opensm/osm_log.h b/opensm/include/opensm/osm_log.h index 48fb15e..f90aa9b 100644 --- a/opensm/include/opensm/osm_log.h +++ b/opensm/include/opensm/osm_log.h @@ -38,11 +38,6 @@ * Declaration of osm_log_t. * This object represents the log file. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #ifndef _OSM_LOG_H_ diff --git a/opensm/include/opensm/osm_mad_pool.h b/opensm/include/opensm/osm_mad_pool.h index e3234f4..3f5df01 100644 --- a/opensm/include/opensm/osm_mad_pool.h +++ b/opensm/include/opensm/osm_mad_pool.h @@ -38,11 +38,6 @@ * Declaration of osm_mad_pool_t. * This object represents a pool of management datagram (MAD) objects. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #ifndef _OSM_MAD_POOL_H_ diff --git a/opensm/include/opensm/osm_madw.h b/opensm/include/opensm/osm_madw.h index f957a99..4664374 100644 --- a/opensm/include/opensm/osm_madw.h +++ b/opensm/include/opensm/osm_madw.h @@ -38,11 +38,6 @@ * Declaration of osm_mad_wrapper_t. * This object represents the context wrapper for OpenSM MAD processing. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #ifndef _OSM_MADW_H_ diff --git a/opensm/include/opensm/osm_mcast_tbl.h b/opensm/include/opensm/osm_mcast_tbl.h index 46a41ec..3aa5aa9 100644 --- a/opensm/include/opensm/osm_mcast_tbl.h +++ b/opensm/include/opensm/osm_mcast_tbl.h @@ -38,11 +38,6 @@ * Declaration of osm_mcast_tbl_t. * This object represents a multicast forwarding table. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #ifndef _OSM_MCAST_TBL_H_ diff --git a/opensm/include/opensm/osm_mcm_info.h b/opensm/include/opensm/osm_mcm_info.h index 202839d..2f5eb11 100644 --- a/opensm/include/opensm/osm_mcm_info.h +++ b/opensm/include/opensm/osm_mcm_info.h @@ -38,11 +38,6 @@ * Declaration of osm_mcm_info_t. * This object represents a Multicast Forwarding Information object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_MCM_INFO_H_ diff --git a/opensm/include/opensm/osm_mcm_port.h b/opensm/include/opensm/osm_mcm_port.h index da3ca15..e8c560e 100644 --- a/opensm/include/opensm/osm_mcm_port.h +++ b/opensm/include/opensm/osm_mcm_port.h @@ -38,11 +38,6 @@ * Declaration of osm_mcm_port_t. * This object represents the membership of a port in a multicast group. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_MCM_PORT_H_ diff --git a/opensm/include/opensm/osm_msgdef.h b/opensm/include/opensm/osm_msgdef.h index 0dee6b9..dbf3e53 100644 --- a/opensm/include/opensm/osm_msgdef.h +++ b/opensm/include/opensm/osm_msgdef.h @@ -36,11 +36,6 @@ /* * Abstract: * Declaration of Dispatcher message values. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #ifndef _OSM_MSGDEF_H_ diff --git a/opensm/include/opensm/osm_mtree.h b/opensm/include/opensm/osm_mtree.h index d1d719e..e9d663c 100644 --- a/opensm/include/opensm/osm_mtree.h +++ b/opensm/include/opensm/osm_mtree.h @@ -38,11 +38,6 @@ * Declaration of osm_mtree_t. * This object represents multicast spanning tree. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_MTREE_H_ diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index f442a45..ce827d1 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -38,11 +38,6 @@ * Declaration of osm_mgrp_t. * This object represents an IBA Multicast Group. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_MULTICAST_H_ diff --git a/opensm/include/opensm/osm_node.h b/opensm/include/opensm/osm_node.h index 56e4dbb..a679f65 100644 --- a/opensm/include/opensm/osm_node.h +++ b/opensm/include/opensm/osm_node.h @@ -38,11 +38,6 @@ * Declaration of osm_node_t. * This object represents an IBA node. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_NODE_H_ diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index a545a91..92e4238 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -38,11 +38,6 @@ * Declaration of osm_opensm_t. * This object represents the OpenSM super object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #ifndef _OSM_OPENSM_H_ diff --git a/opensm/include/opensm/osm_partition.h b/opensm/include/opensm/osm_partition.h index 326aeb6..6ddb8ae 100644 --- a/opensm/include/opensm/osm_partition.h +++ b/opensm/include/opensm/osm_partition.h @@ -38,11 +38,6 @@ * Declaration of osm_prtn_t. * This object represents an IBA Partition. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_PARTITION_H_ diff --git a/opensm/include/opensm/osm_path.h b/opensm/include/opensm/osm_path.h index e26036c..33c5667 100644 --- a/opensm/include/opensm/osm_path.h +++ b/opensm/include/opensm/osm_path.h @@ -53,11 +53,6 @@ BEGIN_C_DECLS * Abstract: * Declaration of path related objects. * These objects are part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ /****h* OpenSM/DR Path * NAME diff --git a/opensm/include/opensm/osm_pkey.h b/opensm/include/opensm/osm_pkey.h index c1cdcc6..54bfcd9 100644 --- a/opensm/include/opensm/osm_pkey.h +++ b/opensm/include/opensm/osm_pkey.h @@ -64,11 +64,6 @@ struct _osm_physp; /* * Abstract: * Declaration of pkey manipulation functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.1 $ */ /****s* OpenSM: osm_pkey_tbl_t diff --git a/opensm/include/opensm/osm_pkey_mgr.h b/opensm/include/opensm/osm_pkey_mgr.h index 0832350..91587bd 100644 --- a/opensm/include/opensm/osm_pkey_mgr.h +++ b/opensm/include/opensm/osm_pkey_mgr.h @@ -36,11 +36,6 @@ * Abstract: * Prototype for osm_pkey_mgr_process() function * This is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_PKEY_MGR_H_ diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index a9bf78f..3d501d8 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -38,11 +38,6 @@ * Declaration of port related objects. * These objects comprise an IBA port. * These objects are part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #ifndef _OSM_PORT_H_ diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index bbb59ef..d735388 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -38,11 +38,6 @@ * Declaration of Switch/osm_port_profile_t. * This object represents a port profile for an IBA switch. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #ifndef _OSM_PORT_PROFILE_H_ diff --git a/opensm/include/opensm/osm_qos_policy.h b/opensm/include/opensm/osm_qos_policy.h index f5815d8..d57a40a 100644 --- a/opensm/include/opensm/osm_qos_policy.h +++ b/opensm/include/opensm/osm_qos_policy.h @@ -37,9 +37,6 @@ * Abstract: * Declaration of OSM QoS Policy data types and functions. * - * Environment: - * Linux User Mode - * * Author: * Yevgeny Kliteynik, Mellanox */ diff --git a/opensm/include/opensm/osm_rand_fwd_tbl.h b/opensm/include/opensm/osm_rand_fwd_tbl.h index 8f46ef2..c7c4586 100644 --- a/opensm/include/opensm/osm_rand_fwd_tbl.h +++ b/opensm/include/opensm/osm_rand_fwd_tbl.h @@ -38,11 +38,6 @@ * Declaration of osm_switch_t. * This object represents an IBA switch. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_RAND_FWD_TBL_H_ diff --git a/opensm/include/opensm/osm_remote_sm.h b/opensm/include/opensm/osm_remote_sm.h index 4c5f67c..1bdc6cb 100644 --- a/opensm/include/opensm/osm_remote_sm.h +++ b/opensm/include/opensm/osm_remote_sm.h @@ -38,11 +38,6 @@ * Declaration of osm_sm_t, osm_remote_sm_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_REMOTE_SM_H_ diff --git a/opensm/include/opensm/osm_router.h b/opensm/include/opensm/osm_router.h index e7d2294..7ba6b3d 100644 --- a/opensm/include/opensm/osm_router.h +++ b/opensm/include/opensm/osm_router.h @@ -38,10 +38,6 @@ * Declaration of osm_router_t. * This object represents an IBA router. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * */ #ifndef _OSM_ROUTER_H_ diff --git a/opensm/include/opensm/osm_sa.h b/opensm/include/opensm/osm_sa.h index 370e4e0..eff991b 100644 --- a/opensm/include/opensm/osm_sa.h +++ b/opensm/include/opensm/osm_sa.h @@ -38,11 +38,6 @@ * Declaration of osm_sa_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #ifndef _OSM_SA_H_ diff --git a/opensm/include/opensm/osm_sa_mad_ctrl.h b/opensm/include/opensm/osm_sa_mad_ctrl.h index a51c0b6..1fde803 100644 --- a/opensm/include/opensm/osm_sa_mad_ctrl.h +++ b/opensm/include/opensm/osm_sa_mad_ctrl.h @@ -39,11 +39,6 @@ * This object represents a controller that receives the IBA SA * attributes from a node. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_SA_MAD_CTRL_H_ diff --git a/opensm/include/opensm/osm_service.h b/opensm/include/opensm/osm_service.h index 82b6e52..d1dc506 100644 --- a/opensm/include/opensm/osm_service.h +++ b/opensm/include/opensm/osm_service.h @@ -41,11 +41,6 @@ * Declaration of osm_service_rec_t. * This object represents an IBA Service Record. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #include diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h index f4339e4..eaabefe 100644 --- a/opensm/include/opensm/osm_sm.h +++ b/opensm/include/opensm/osm_sm.h @@ -38,11 +38,6 @@ * Declaration of osm_sm_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #ifndef _OSM_SM_H_ diff --git a/opensm/include/opensm/osm_sm_mad_ctrl.h b/opensm/include/opensm/osm_sm_mad_ctrl.h index ccacb93..adf68b9 100644 --- a/opensm/include/opensm/osm_sm_mad_ctrl.h +++ b/opensm/include/opensm/osm_sm_mad_ctrl.h @@ -39,11 +39,6 @@ * This object represents a controller that receives the IBA NodeInfo * attribute from a node. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_SM_MAD_CTRL_H_ diff --git a/opensm/include/opensm/osm_stats.h b/opensm/include/opensm/osm_stats.h index ecd752b..ad50d5a 100644 --- a/opensm/include/opensm/osm_stats.h +++ b/opensm/include/opensm/osm_stats.h @@ -38,11 +38,6 @@ * Declaration of osm_stats_t. * This object represents the OpenSM statistics object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_STATS_H_ diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 1b862c0..338b274 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -38,11 +38,6 @@ * Declaration of osm_subn_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.12 $ */ #ifndef _OSM_SUBNET_H_ diff --git a/opensm/include/opensm/osm_sweep_fail_ctrl.h b/opensm/include/opensm/osm_sweep_fail_ctrl.h index 28ae7a6..fccc5fa 100644 --- a/opensm/include/opensm/osm_sweep_fail_ctrl.h +++ b/opensm/include/opensm/osm_sweep_fail_ctrl.h @@ -39,11 +39,6 @@ * This object represents a controller that * handles transport failures during sweeps. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_SWEEP_FAIL_CTRL_H_ diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index 904a61d..f5455af 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -38,11 +38,6 @@ * Declaration of osm_switch_t. * This object represents an IBA switch. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #ifndef _OSM_SWITCH_H_ diff --git a/opensm/include/opensm/osm_ucast_mgr.h b/opensm/include/opensm/osm_ucast_mgr.h index 0317c93..478078d 100644 --- a/opensm/include/opensm/osm_ucast_mgr.h +++ b/opensm/include/opensm/osm_ucast_mgr.h @@ -38,11 +38,6 @@ * Declaration of osm_ucast_mgr_t. * This object represents the Unicast Manager object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_UCAST_MGR_H_ diff --git a/opensm/include/opensm/osm_vl15intf.h b/opensm/include/opensm/osm_vl15intf.h index 75dfd11..7453de8 100644 --- a/opensm/include/opensm/osm_vl15intf.h +++ b/opensm/include/opensm/osm_vl15intf.h @@ -38,11 +38,6 @@ * Declaration of osm_vl15_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #ifndef _OSM_VL15INTF_H_ diff --git a/opensm/include/vendor/osm_pkt_randomizer.h b/opensm/include/vendor/osm_pkt_randomizer.h index 0c831e0..5a8ef5f 100644 --- a/opensm/include/vendor/osm_pkt_randomizer.h +++ b/opensm/include/vendor/osm_pkt_randomizer.h @@ -38,11 +38,6 @@ * Declaration of osm_subn_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.12 $ */ #ifndef _OSM_PKT_RANDOMIZER_H_ diff --git a/opensm/include/vendor/osm_umadt.h b/opensm/include/vendor/osm_umadt.h index 809f478..129627d 100644 --- a/opensm/include/vendor/osm_umadt.h +++ b/opensm/include/vendor/osm_umadt.h @@ -38,11 +38,6 @@ * Declaration of osm_mad_wrapper_t. * This object represents the context wrapper for OpenSM MAD processing. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_UMADT_h_ diff --git a/opensm/include/vendor/osm_vendor.h b/opensm/include/vendor/osm_vendor.h index eb7665e..747c090 100644 --- a/opensm/include/vendor/osm_vendor.h +++ b/opensm/include/vendor/osm_vendor.h @@ -36,11 +36,6 @@ /* * Abstract: * Include file used by OpenSM to pull in the correct vendor file. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ /* diff --git a/opensm/include/vendor/osm_vendor_al.h b/opensm/include/vendor/osm_vendor_al.h index 1f80006..e7371c9 100644 --- a/opensm/include/vendor/osm_vendor_al.h +++ b/opensm/include/vendor/osm_vendor_al.h @@ -38,11 +38,6 @@ * Declaration of osm_mad_wrapper_t. * This object represents the context wrapper for OpenSM MAD processing. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_VENDOR_AL_H_ diff --git a/opensm/include/vendor/osm_vendor_api.h b/opensm/include/vendor/osm_vendor_api.h index 8e3011d..70eb6cc 100644 --- a/opensm/include/vendor/osm_vendor_api.h +++ b/opensm/include/vendor/osm_vendor_api.h @@ -37,11 +37,6 @@ * Abstract: * Specification of the OpenSM transport API. This API is OpenSM's view * of the Infiniband transport. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_VENDOR_API_H_ diff --git a/opensm/include/vendor/osm_vendor_mtl.h b/opensm/include/vendor/osm_vendor_mtl.h index e740b2e..df48260 100644 --- a/opensm/include/vendor/osm_vendor_mtl.h +++ b/opensm/include/vendor/osm_vendor_mtl.h @@ -37,11 +37,6 @@ * Abstract: * Definition of interface for the MTL Vendor * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.3 $ */ #ifndef _OSM_VENDOR_MTL_H_ diff --git a/opensm/include/vendor/osm_vendor_mtl_hca_guid.h b/opensm/include/vendor/osm_vendor_mtl_hca_guid.h index 59bfad5..1b3da88 100644 --- a/opensm/include/vendor/osm_vendor_mtl_hca_guid.h +++ b/opensm/include/vendor/osm_vendor_mtl_hca_guid.h @@ -37,11 +37,6 @@ * Abstract: * Provides interface over VAPI for obtaining the local ports guids or from guid * obtaining the HCA and port number. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.3 $ */ #ifndef _OSM_VENDOR_HCA_GUID_H_ diff --git a/opensm/include/vendor/osm_vendor_mtl_transaction_mgr.h b/opensm/include/vendor/osm_vendor_mtl_transaction_mgr.h index bdd0714..6ec5b86 100644 --- a/opensm/include/vendor/osm_vendor_mtl_transaction_mgr.h +++ b/opensm/include/vendor/osm_vendor_mtl_transaction_mgr.h @@ -37,11 +37,6 @@ * Abstract: * Definition of interface for the MTL Vendor * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_TRANSACTION_MGR_H_ diff --git a/opensm/include/vendor/osm_vendor_sa_api.h b/opensm/include/vendor/osm_vendor_sa_api.h index 3de7fe3..4a4eeaf 100644 --- a/opensm/include/vendor/osm_vendor_sa_api.h +++ b/opensm/include/vendor/osm_vendor_sa_api.h @@ -37,11 +37,6 @@ * Abstract: * Specification of the OpenSM SA Client API. This API uses the basic osm * vendor API to provide SA Client interface. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_VENDOR_SA_API_H_ diff --git a/opensm/include/vendor/osm_vendor_select.h b/opensm/include/vendor/osm_vendor_select.h index 5bd6ba6..8dcc05b 100644 --- a/opensm/include/vendor/osm_vendor_select.h +++ b/opensm/include/vendor/osm_vendor_select.h @@ -36,11 +36,6 @@ /* * Abstract: * Include file that defines which vendor files to compile. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #ifndef _OSM_VENDOR_SELECT_H_ diff --git a/opensm/include/vendor/osm_vendor_test.h b/opensm/include/vendor/osm_vendor_test.h index 26e6314..a1ae1eb 100644 --- a/opensm/include/vendor/osm_vendor_test.h +++ b/opensm/include/vendor/osm_vendor_test.h @@ -57,11 +57,6 @@ BEGIN_C_DECLS * This is the "Test" vendor which allows compilation and some * testing without a real vendor interface. * These objects are part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ /****h* OpenSM/Vendor Test * NAME diff --git a/opensm/include/vendor/osm_vendor_ts.h b/opensm/include/vendor/osm_vendor_ts.h index 2c46fb9..a43f4e4 100644 --- a/opensm/include/vendor/osm_vendor_ts.h +++ b/opensm/include/vendor/osm_vendor_ts.h @@ -37,11 +37,6 @@ * Abstract: * Definition of interface for the TS Vendor * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.3 $ */ #ifndef _OSM_VENDOR_TS_H_ diff --git a/opensm/include/vendor/osm_vendor_umadt.h b/opensm/include/vendor/osm_vendor_umadt.h index 68b7dfb..8cdb631 100644 --- a/opensm/include/vendor/osm_vendor_umadt.h +++ b/opensm/include/vendor/osm_vendor_umadt.h @@ -38,11 +38,6 @@ * Declaration of osm_mad_wrapper_t. * This object represents the context wrapper for OpenSM MAD processing. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #ifndef _OSM_VENDOR_UMADT_h_ diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 05b3dd5..040a0d7 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -36,12 +36,8 @@ /* * Abstract: * Command line interface for opensm. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.23 $ */ + #if HAVE_CONFIG_H # include #endif /* HAVE_CONFIG_H */ diff --git a/opensm/opensm/osm_console_io.c b/opensm/opensm/osm_console_io.c index 0e788d6..ee0b632 100644 --- a/opensm/opensm/osm_console_io.c +++ b/opensm/opensm/osm_console_io.c @@ -30,6 +30,7 @@ * SOFTWARE. * */ + /* * Abstract: * Provide a framework for the Console which decouples the connection @@ -37,11 +38,6 @@ * * Extensible - allows a variety of connection methods independent of * the console commands. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.0 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_db_files.c b/opensm/opensm/osm_db_files.c index ce61980..1d1aaab 100644 --- a/opensm/opensm/osm_db_files.c +++ b/opensm/opensm/osm_db_files.c @@ -35,9 +35,7 @@ /* * Abstract: - * Implemntation of the osm_db interface using simple text files - * - * $Revision: 1.4 $ + * Implementation of the osm_db interface using simple text files */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index f9b0b00..e2f5828 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -38,11 +38,6 @@ * Implementation of osm_drop_mgr_t. * This object represents the Drop Manager object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_fwd_tbl.c b/opensm/opensm/osm_fwd_tbl.c index 433656b..2ea74af 100644 --- a/opensm/opensm/osm_fwd_tbl.c +++ b/opensm/opensm/osm_fwd_tbl.c @@ -38,11 +38,6 @@ * Implementation of osm_fwd_tbl_t. * This object represents a unicast forwarding table. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c index 15cc2e9..1b20638 100644 --- a/opensm/opensm/osm_helper.c +++ b/opensm/opensm/osm_helper.c @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of opensm helper functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.19 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index 9553f7f..7782f02 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of inform record functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.18 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index ed6f7a5..1ad9ef0 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -74,11 +74,6 @@ * 3.2 if any change send the signal PENDING... * * 4. Store the guid2lid - * - * Environment: - * Linux User Mode - * - * $Revision: 1.15 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_lin_fwd_rcv.c b/opensm/opensm/osm_lin_fwd_rcv.c index 18be6a3..c0ec72d 100644 --- a/opensm/opensm/osm_lin_fwd_rcv.c +++ b/opensm/opensm/osm_lin_fwd_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_lft_rcv_t. * This object represents the NodeDescription Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_lin_fwd_tbl.c b/opensm/opensm/osm_lin_fwd_tbl.c index 438e3c3..5c969c7 100644 --- a/opensm/opensm/osm_lin_fwd_tbl.c +++ b/opensm/opensm/osm_lin_fwd_tbl.c @@ -38,11 +38,6 @@ * Implementation of osm_lin_fwd_tbl_t. * This object represents an linear forwarding table. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c index e4ba93e..3454024 100644 --- a/opensm/opensm/osm_link_mgr.c +++ b/opensm/opensm/osm_link_mgr.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_link_mgr_t. * This file implements the Link Manager object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.15 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_log.c b/opensm/opensm/osm_log.c index 727aabc..e10c5ea 100644 --- a/opensm/opensm/osm_log.c +++ b/opensm/opensm/osm_log.c @@ -38,11 +38,6 @@ * Implementation of osm_log_t. * This object represents the log file. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mad_pool.c b/opensm/opensm/osm_mad_pool.c index a7769d4..3341052 100644 --- a/opensm/opensm/osm_mad_pool.c +++ b/opensm/opensm/osm_mad_pool.c @@ -38,11 +38,6 @@ * Implementation of osm_mad_pool_t. * This object represents a pool of management datagram (MAD) objects. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mcast_fwd_rcv.c b/opensm/opensm/osm_mcast_fwd_rcv.c index 8c75254..b2f491f 100644 --- a/opensm/opensm/osm_mcast_fwd_rcv.c +++ b/opensm/opensm/osm_mcast_fwd_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_mft_rcv_t. * This object represents the Multicast Forwarding Table Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index a6185fe..75afbbb 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_mcast_mgr_t. * This file implements the Multicast Manager object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.9 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mcast_tbl.c b/opensm/opensm/osm_mcast_tbl.c index 8715869..92fbb63 100644 --- a/opensm/opensm/osm_mcast_tbl.c +++ b/opensm/opensm/osm_mcast_tbl.c @@ -38,11 +38,6 @@ * Implementation of osm_mcast_tbl_t. * This object represents an multicast forwarding table. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mcm_info.c b/opensm/opensm/osm_mcm_info.c index 5d5d3a9..0325a34 100644 --- a/opensm/opensm/osm_mcm_info.c +++ b/opensm/opensm/osm_mcm_info.c @@ -38,11 +38,6 @@ * Declaration of osm_mcm_info_t. * This object represents a Multicast Forwarding Information object. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mcm_port.c b/opensm/opensm/osm_mcm_port.c index 1dac7fe..d68cf65 100644 --- a/opensm/opensm/osm_mcm_port.c +++ b/opensm/opensm/osm_mcm_port.c @@ -38,11 +38,6 @@ * Implementation of osm_mcm_port_t. * This object represents the membership of a port in a multicast group. * This object is part of the OpenSM family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_mtree.c b/opensm/opensm/osm_mtree.c index 397a8ee..2a24c6d 100644 --- a/opensm/opensm/osm_mtree.c +++ b/opensm/opensm/osm_mtree.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_mtree_node_t. * This file implements the Multicast Tree object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index a07e5da..77e61ad 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of multicast functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_node.c b/opensm/opensm/osm_node.c index 0ebaf90..d99c656 100644 --- a/opensm/opensm/osm_node.c +++ b/opensm/opensm/osm_node.c @@ -38,11 +38,6 @@ * Implementation of osm_node_t. * This object represents an Infiniband Node. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_node_desc_rcv.c b/opensm/opensm/osm_node_desc_rcv.c index 4a22aab..d848513 100644 --- a/opensm/opensm/osm_node_desc_rcv.c +++ b/opensm/opensm/osm_node_desc_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_nd_rcv_t. * This object represents the NodeDescription Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_node_info_rcv.c b/opensm/opensm/osm_node_info_rcv.c index 6818e05..2bccf33 100644 --- a/opensm/opensm/osm_node_info_rcv.c +++ b/opensm/opensm/osm_node_info_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_ni_rcv_t. * This object represents the NodeInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.9 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index abe55b5..167caab 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -38,11 +38,6 @@ * Implementation of osm_opensm_t. * This object represents the opensm super object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_pkey.c b/opensm/opensm/osm_pkey.c index c3b8394..2b2ea36 100644 --- a/opensm/opensm/osm_pkey.c +++ b/opensm/opensm/osm_pkey.c @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of opensm pkey manipulation functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.1 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_pkey_mgr.c b/opensm/opensm/osm_pkey_mgr.c index 0e94df4..04fb18b 100644 --- a/opensm/opensm/osm_pkey_mgr.c +++ b/opensm/opensm/osm_pkey_mgr.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of the P_Key Manager (Partititon Manager). * This is part of the OpenSM. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_port.c b/opensm/opensm/osm_port.c index d66e6be..18f1ef2 100644 --- a/opensm/opensm/osm_port.c +++ b/opensm/opensm/osm_port.c @@ -38,11 +38,6 @@ * Implementation of osm_physp_t. * This object represents an Infiniband Port. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_port_info_rcv.c b/opensm/opensm/osm_port_info_rcv.c index 98d367b..c2ddfb0 100644 --- a/opensm/opensm/osm_port_info_rcv.c +++ b/opensm/opensm/osm_port_info_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_pi_rcv_t. * This object represents the PortInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_prtn.c b/opensm/opensm/osm_prtn.c index 187cff6..9234214 100644 --- a/opensm/opensm/osm_prtn.c +++ b/opensm/opensm/osm_prtn.c @@ -36,11 +36,6 @@ * Implementation of osm_prtn_t. * This object represents an IBA partition. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision$ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_prtn_config.c b/opensm/opensm/osm_prtn_config.c index 57467d4..9511608 100644 --- a/opensm/opensm/osm_prtn_config.c +++ b/opensm/opensm/osm_prtn_config.c @@ -34,10 +34,6 @@ /* * Abstract: * Implementation of opensm partition management configuration - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_qos.c b/opensm/opensm/osm_qos.c index ca9a9d3..15eaea1 100644 --- a/opensm/opensm/osm_qos.c +++ b/opensm/opensm/osm_qos.c @@ -34,10 +34,6 @@ /* * Abstract: * Implementation of OpenSM QoS infrastructure primitives - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index 6c81872..f1d4e54 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -37,9 +37,6 @@ * Abstract: * OSM QoS Policy functions. * - * Environment: - * Linux User Mode - * * Author: * Yevgeny Kliteynik, Mellanox */ diff --git a/opensm/opensm/osm_remote_sm.c b/opensm/opensm/osm_remote_sm.c index 4d44d53..96bfd78 100644 --- a/opensm/opensm/osm_remote_sm.c +++ b/opensm/opensm/osm_remote_sm.c @@ -38,11 +38,6 @@ * Implementation of osm_sm_t. * This object represents the remote SM object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 42a638a..5454144 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -38,11 +38,6 @@ * Implementation of osm_req_t. * This object represents the generic attribute requester. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_resp.c b/opensm/opensm/osm_resp.c index 09239de..d141812 100644 --- a/opensm/opensm/osm_resp.c +++ b/opensm/opensm/osm_resp.c @@ -38,11 +38,6 @@ * Implementation of osm_resp_t. * This object represents the generic attribute responder. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_router.c b/opensm/opensm/osm_router.c index 8b93fd6..cb5a236 100644 --- a/opensm/opensm/osm_router.c +++ b/opensm/opensm/osm_router.c @@ -38,10 +38,6 @@ * Implementation of osm_router_t. * This object represents an Infiniband router. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index 46c5bf7..adf4931 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -38,11 +38,6 @@ * Implementation of osm_sa_t. * This object represents the Subnet Administration object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.14 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_class_port_info.c b/opensm/opensm/osm_sa_class_port_info.c index 0839c1b..a43524f 100644 --- a/opensm/opensm/osm_sa_class_port_info.c +++ b/opensm/opensm/osm_sa_class_port_info.c @@ -38,11 +38,6 @@ * Implementation of osm_cpi_rcv_t. * This object represents the ClassPortInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_guidinfo_record.c b/opensm/opensm/osm_sa_guidinfo_record.c index 43f249a..f38c575 100644 --- a/opensm/opensm/osm_sa_guidinfo_record.c +++ b/opensm/opensm/osm_sa_guidinfo_record.c @@ -38,10 +38,6 @@ * Implementation of osm_gir_rcv_t. * This object represents the GUIDInfoRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_informinfo.c b/opensm/opensm/osm_sa_informinfo.c index f91d0ab..2e0c504 100644 --- a/opensm/opensm/osm_sa_informinfo.c +++ b/opensm/opensm/osm_sa_informinfo.c @@ -38,11 +38,6 @@ * Implementation of osm_infr_rcv_t. * This object represents the InformInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_lft_record.c b/opensm/opensm/osm_sa_lft_record.c index 82bb63c..1699824 100644 --- a/opensm/opensm/osm_sa_lft_record.c +++ b/opensm/opensm/osm_sa_lft_record.c @@ -38,11 +38,6 @@ * Implementation of osm_lftr_rcv_t. * This object represents the LinearForwardingTable Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_link_record.c b/opensm/opensm/osm_sa_link_record.c index 31abf43..6cc1836 100644 --- a/opensm/opensm/osm_sa_link_record.c +++ b/opensm/opensm/osm_sa_link_record.c @@ -38,11 +38,6 @@ * Implementation of osm_lr_rcv_t. * This object represents the LinkRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_mad_ctrl.c b/opensm/opensm/osm_sa_mad_ctrl.c index abd8d02..08026bc 100644 --- a/opensm/opensm/osm_sa_mad_ctrl.c +++ b/opensm/opensm/osm_sa_mad_ctrl.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_sa_mad_ctrl_t. * This object is part of the SA object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 3cfd5f7..e7822fe 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -38,11 +38,6 @@ * Implementation of osm_mcmr_recv_t. * This object represents the MCMemberRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.15 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_mft_record.c b/opensm/opensm/osm_sa_mft_record.c index da786a0..24b4c3b 100644 --- a/opensm/opensm/osm_sa_mft_record.c +++ b/opensm/opensm/osm_sa_mft_record.c @@ -38,10 +38,6 @@ * Implementation of osm_mftr_rcv_t. * This object represents the MulticastForwardingTable Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_multipath_record.c b/opensm/opensm/osm_sa_multipath_record.c index a800fbe..ad2e1a0 100644 --- a/opensm/opensm/osm_sa_multipath_record.c +++ b/opensm/opensm/osm_sa_multipath_record.c @@ -38,10 +38,6 @@ * Implementation of osm_mpr_rcv_t. * This object represents the MultiPath Record Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * */ #if defined (VENDOR_RMPP_SUPPORT) && defined (DUAL_SIDED_RMPP) diff --git a/opensm/opensm/osm_sa_node_record.c b/opensm/opensm/osm_sa_node_record.c index 1e9d7dc..a43bba7 100644 --- a/opensm/opensm/osm_sa_node_record.c +++ b/opensm/opensm/osm_sa_node_record.c @@ -38,11 +38,6 @@ * Implementation of osm_nr_rcv_t. * This object represents the NodeInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.8 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 9a9423c..65e6a74 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -38,11 +38,6 @@ * Implementation of osm_pr_rcv_t. * This object represents the PathRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.10 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_portinfo_record.c b/opensm/opensm/osm_sa_portinfo_record.c index ad9c9ae..8a5ed6d 100644 --- a/opensm/opensm/osm_sa_portinfo_record.c +++ b/opensm/opensm/osm_sa_portinfo_record.c @@ -38,11 +38,6 @@ * Implementation of osm_pir_rcv_t. * This object represents the PortInfoRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.10 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_service_record.c b/opensm/opensm/osm_sa_service_record.c index ce99db2..d68ad22 100644 --- a/opensm/opensm/osm_sa_service_record.c +++ b/opensm/opensm/osm_sa_service_record.c @@ -38,11 +38,6 @@ * Implementation of osm_sr_rcv_t. * This object represents the ServiceRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.9 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_slvl_record.c b/opensm/opensm/osm_sa_slvl_record.c index ae72623..0248594 100644 --- a/opensm/opensm/osm_sa_slvl_record.c +++ b/opensm/opensm/osm_sa_slvl_record.c @@ -38,11 +38,6 @@ * Implementation of osm_slvl_rec_rcv_t. * This object represents the SLtoVL Mapping Query Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_sminfo_record.c b/opensm/opensm/osm_sa_sminfo_record.c index 75c50ad..1860e55 100644 --- a/opensm/opensm/osm_sa_sminfo_record.c +++ b/opensm/opensm/osm_sa_sminfo_record.c @@ -38,11 +38,6 @@ * Implementation of osm_smir_rcv_t. * This object represents the SMInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_sw_info_record.c b/opensm/opensm/osm_sa_sw_info_record.c index bae4c75..9003a6c 100644 --- a/opensm/opensm/osm_sa_sw_info_record.c +++ b/opensm/opensm/osm_sa_sw_info_record.c @@ -38,10 +38,6 @@ * Implementation of osm_sir_rcv_t. * This object represents the SwitchInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sa_vlarb_record.c b/opensm/opensm/osm_sa_vlarb_record.c index 5948cc1..6308a70 100644 --- a/opensm/opensm/osm_sa_vlarb_record.c +++ b/opensm/opensm/osm_sa_vlarb_record.c @@ -38,11 +38,6 @@ * Implementation of osm_vlarb_rec_rcv_t. * This object represents the VLArbitrationRecord Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_service.c b/opensm/opensm/osm_service.c index 41aada2..d85be4c 100644 --- a/opensm/opensm/osm_service.c +++ b/opensm/opensm/osm_service.c @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of service record functions. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_slvl_map_rcv.c b/opensm/opensm/osm_slvl_map_rcv.c index c78951d..9899625 100644 --- a/opensm/opensm/osm_slvl_map_rcv.c +++ b/opensm/opensm/osm_slvl_map_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_slvl_rcv_t. * This object represents the SLtoVL Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.4 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index 32525ba..7393ead 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -38,11 +38,6 @@ * Implementation of osm_sm_t. * This object represents the SM Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.9 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sm_mad_ctrl.c b/opensm/opensm/osm_sm_mad_ctrl.c index 4b0c27f..aa91c21 100644 --- a/opensm/opensm/osm_sm_mad_ctrl.c +++ b/opensm/opensm/osm_sm_mad_ctrl.c @@ -38,11 +38,6 @@ * Implementation of osm_sm_mad_ctrl_t. * This object represents the SM MAD request controller object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sm_state_mgr.c b/opensm/opensm/osm_sm_state_mgr.c index 1fb67c5..6edba50 100644 --- a/opensm/opensm/osm_sm_state_mgr.c +++ b/opensm/opensm/osm_sm_state_mgr.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_sm_state_mgr_t. * This file implements the SM State Manager object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.7 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sminfo_rcv.c b/opensm/opensm/osm_sminfo_rcv.c index 96221d6..9a262d2 100644 --- a/opensm/opensm/osm_sminfo_rcv.c +++ b/opensm/opensm/osm_sminfo_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_sminfo_rcv_t. * This object represents the SMInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index d61e1bb..4140b49 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_state_mgr_t. * This file implements the State Manager object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.13 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 38f1e80..e465565 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -38,11 +38,6 @@ * Implementation of osm_subn_t. * This object represents an IBA subnet. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.9 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sw_info_rcv.c b/opensm/opensm/osm_sw_info_rcv.c index 6c9d2b6..d5b28e3 100644 --- a/opensm/opensm/osm_sw_info_rcv.c +++ b/opensm/opensm/osm_sw_info_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_si_rcv_t. * This object represents the SwitchInfo Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_sweep_fail_ctrl.c b/opensm/opensm/osm_sweep_fail_ctrl.c index efe7e65..4f931f5 100644 --- a/opensm/opensm/osm_sweep_fail_ctrl.c +++ b/opensm/opensm/osm_sweep_fail_ctrl.c @@ -36,11 +36,6 @@ /* * Abstract: * Implementation of osm_sweep_fail_ctrl_t. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.5 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index a8e4b72..fdcd109 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -38,11 +38,6 @@ * Implementation of osm_switch_t. * This object represents an Infiniband switch. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.13 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index f27540b..c562e9a 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_trap_rcv_t. * This object represents the Trap Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.12 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_ucast_file.c b/opensm/opensm/osm_ucast_file.c index a6b82f3..43bcbc5 100644 --- a/opensm/opensm/osm_ucast_file.c +++ b/opensm/opensm/osm_ucast_file.c @@ -35,10 +35,6 @@ * Abstract: * Implementation of OpenSM unicast routing module which loads * routes from the dump file - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index caf231c..dc61130 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -36,10 +36,6 @@ /* * Abstract: * Implementation of OpenSM FatTree routing - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c index 74cdd7f..2e67c26 100644 --- a/opensm/opensm/osm_ucast_lash.c +++ b/opensm/opensm/osm_ucast_lash.c @@ -38,10 +38,6 @@ /* * Abstract: * Implementation of LASH algorithm Calculation functions - * - * Environment: - * Linux User Mode - * */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index 938db84..5923638 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of osm_ucast_mgr_t. * This file implements the Unicast Manager object. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.14 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_ucast_updn.c b/opensm/opensm/osm_ucast_updn.c index 0c4df5d..dea27a6 100644 --- a/opensm/opensm/osm_ucast_updn.c +++ b/opensm/opensm/osm_ucast_updn.c @@ -37,11 +37,6 @@ * Abstract: * Implementation of Up Down Algorithm using ranking & Min Hop * Calculation functions - * - * Environment: - * Linux User Mode - * - * $Revision: 1.0 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_vl15intf.c b/opensm/opensm/osm_vl15intf.c index faff2fa..246e6cc 100644 --- a/opensm/opensm/osm_vl15intf.c +++ b/opensm/opensm/osm_vl15intf.c @@ -38,11 +38,6 @@ * Implementation of osm_vl15_t. * This object represents the VL15 Interface object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #if HAVE_CONFIG_H diff --git a/opensm/opensm/osm_vl_arb_rcv.c b/opensm/opensm/osm_vl_arb_rcv.c index c45d805..05d54e2 100644 --- a/opensm/opensm/osm_vl_arb_rcv.c +++ b/opensm/opensm/osm_vl_arb_rcv.c @@ -38,11 +38,6 @@ * Implementation of osm_vla_rcv_t. * This object represents the Vl Arbitration Receiver object. * This object is part of the opensm family of objects. - * - * Environment: - * Linux User Mode - * - * $Revision: 1.6 $ */ #if HAVE_CONFIG_H From televendas.luciano at equagril.com.br Sat Jun 7 11:59:34 2008 From: televendas.luciano at equagril.com.br (Gerry Bautista) Date: Sat, 7 Jun 2008 19:59:34 +0100 Subject: [ofa-general] Wanna have bigger? Message-ID: <01c8c8d9$0206d600$52093a59@televendas.luciano> No problem, just ask me how! You'll find more information here http://www.hyiok.com.cn/a/ Take a break, watch that link. http://www.hyiok.com.cn/a/ From sashak at voltaire.com Sat Jun 7 19:52:33 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 8 Jun 2008 05:52:33 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] OpenSM/osm_sa_mcmember_record.c: Comment reformatting In-Reply-To: <1212843808.6127.244.camel@hrosenstock-ws.xsigo.com> References: <1212843808.6127.244.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080608025233.GO7795@sashak.voltaire.com> On 06:03 Sat 07 Jun , Hal Rosenstock wrote: > OpenSM/osm_sa_mcmember_record.c: Comment reformatting > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 7 19:54:36 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 8 Jun 2008 05:54:36 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] OpenSM: Remove some vestigial comments In-Reply-To: <1212859846.14047.29.camel@hrosenstock-ws.xsigo.com> References: <1212859846.14047.29.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080608025436.GP7795@sashak.voltaire.com> On 10:30 Sat 07 Jun , Hal Rosenstock wrote: > OpenSM: Remove some vestigial comments > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 7 21:29:10 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 8 Jun 2008 07:29:10 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. In-Reply-To: <20080605191251.6882c3d6.weiny2@llnl.gov> References: <20080605191251.6882c3d6.weiny2@llnl.gov> Message-ID: <20080608042910.GS7795@sashak.voltaire.com> On 19:12 Thu 05 Jun , Ira Weiny wrote: > I noticed over the last couple days that ibnetdiscover would print the > incorrect speed on the port I would run ibnetdiscover from. For example: > > Switch 24 "S-0008f10400411f56" # "SW1 wopr ISR9024D (MLX4 FW)" base port 0 lid 11 lmc 0 > ... > [13] "H-0002c90200219e64"[1](2c90200219e65) # "wopri" lid 32 4xSDR > > ^^^ > (Note from the switch side of things it thinks the speed is SDR.) > ... > Ca 2 "H-0002c90200219e64" # "wopri" > [1](2c90200219e65) "S-0008f10400411f56"[13] # lid 32 lmc 0 "SW1 wopr ISR9024D (MLX4 FW)" lid 11 4xDDR > > ^^^ > (but here DDR.) > > It turns out that when you first discover a switch the port object created gets > the PortInfo of Port "0". > > This patch queries again for the PortInfo of the port we "came in on". > > Ira > > > From 5bc66af276c7baabd4d66be9df0379271cb625b4 Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Thu, 5 Jun 2008 19:03:44 -0700 > Subject: [PATCH] infiniband-diags/src/ibnetdiscover.c: Fix the PortInfo data on the port we discover a switch on. > > Previously the portinfo data for that port object would be the PortInfo of > port "0" of the switch. Specifically this would cause the speed of the link > to be printed incorrectly. > > Signed-off-by: Ira K. Weiny Applied. Thanks. Sasha From jgunthorpe at obsidianresearch.com Sat Jun 7 22:44:26 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Sat, 7 Jun 2008 23:44:26 -0600 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080608054426.GK22807@obsidianresearch.com> On Sat, Jun 07, 2008 at 06:03:17AM -0700, Hal Rosenstock wrote: > 2. Add in support for overloading MLIDs. On the configuration side, a > number of additional options would be added to consolidate_ipv6_snm_req. > These include the number of MLIDs to compress down to (default 16), > a multicast group (MGID) base address and (full MGID) mask. this would > default to 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY where Z is the scope, > XXXX is the P_Key, and YYYYYY is the last 24 bits of the port guid ( > the YYYYYY bits would be masked out by default). This is what the > current workaround uses for collapsing the multicast groups. When I wrote originally I was imagining having a list, like: MGID/PREFIX MLIDS ff11:601b::/32 4 ff1e:601b::/32 4 ff00::/8 128 # Everything else Please, please use IPv6 notation for this (and all other GIDs). Since the addressing scheme is taken from IPv6 the interesting bits align sensibly.. The double 64 bit number notation used in osm is horrible, and should be killed off :) inet_pton/inet_ntop w/ AF_INET6 can be used, and is compatible with the existing GID structure. I would recommend a longest prefix bits scheme rather than a mask for administator sanity. The IPv6 addressing scheme that MGID/GID borrows from was designed to work like this, and administrators will be familiar with the concept. If you feel the complexity of non-prefix matching is needed then it is easier on administrators to copy the existing idom used in ACLs and use a two level label/property scheme: Table 1: MGID/PREFIX GROUP NAME ff11:601b::/32 IPV6-SNM ff1e:601b::/32 IPV6-SNM ff1e:601b:89:/48 IPV6-SNM-PKEY_89 ff00::/8 OTHERS Table 2: NAME MLIDS IPV6-SNM 4 IPV6_SNM-PKEY_89 128 OTHERS 128 But I think that is probably overkill, unless you know of other heavy multicast users besides SNM. Also note that a longest prefix match data structure can likely be re-used for future work on IB routing, along with this use for MGID compaction. > The criteria for overloading MLIDs includes any group parameters that > need to be in common (e.g. rate. MTU, perhaps PKey (see below), etc.). It seems reasonable to me that every MTU+rate gets a seperate allocation space, each limited to the configured size. > As PKey is part of the MGID, does this need to be addressed (and if > so) how ? I don't think pkey needs special treatment - prefix matching takes care of it. If you feel seperate allocations for each pkey may someday be necessary then having a table scheme rather than a single value takes care of it nicely. Regards, Jason From ticklednm4 at suzuki.de Sun Jun 8 04:58:55 2008 From: ticklednm4 at suzuki.de (Jody Diaz) Date: , 8 Jun 2008 20:58:55 +0900 Subject: [ofa-general] Those young people taking the position you deserve because you lack a Degree? dkmsdk mhdbnc ndtyqi Message-ID: <01c8c9aa$76835180$74d2283c@ticklednm4> Un dr iv qp ers qt ity De do gr fk ee OBTAIN A PROSP ee EROUS FUTURE, MO vjy NEY-EA bke RNING POWER, AND THE PRESTIGE THAT COMES WITH HAVING THE CA apy REER POSITION YOU HAVE ALWAYS DREAMED OF. DI mo PLO pc MA FROM PRESTIGIOUS NON-AC xxd CRE psi DITED UN fbq VERS wvy ITIES BASED ON YOUR PRESENT KNO eaz WLEDGE AND PR qss OFESS qx IONAL EX zfg PERI nrv ENCE. If you qua vl lify, no required tests, classes, books or exa jxj minat uow ions. Con qag fident bus iality As hv su nv red 1-5 iw 2 xrw 0-2 rx 03-02 mg 13Le eia ave a mes su sage 24 hours a day, 7 days a week incl erg uding Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Sun Jun 8 08:02:51 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sun, 08 Jun 2008 08:02:51 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080608054426.GK22807@obsidianresearch.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080608054426.GK22807@obsidianresearch.com> Message-ID: <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> On Sat, 2008-06-07 at 23:44 -0600, Jason Gunthorpe wrote: > On Sat, Jun 07, 2008 at 06:03:17AM -0700, Hal Rosenstock wrote: > > > 2. Add in support for overloading MLIDs. On the configuration side, a > > number of additional options would be added to consolidate_ipv6_snm_req. > > These include the number of MLIDs to compress down to (default 16), > > a multicast group (MGID) base address and (full MGID) mask. this would > > default to 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY where Z is the scope, > > XXXX is the P_Key, and YYYYYY is the last 24 bits of the port guid ( > > the YYYYYY bits would be masked out by default). This is what the > > current workaround uses for collapsing the multicast groups. > > When I wrote originally I was imagining having a list, like: > > MGID/PREFIX MLIDS > ff11:601b::/32 4 > ff1e:601b::/32 4 > ff00::/8 128 # Everything else > > Please, please use IPv6 notation for this (and all other GIDs). Since > the addressing scheme is taken from IPv6 the interesting bits align > sensibly.. The double 64 bit number notation used in osm is horrible, > and should be killed off :) This is a better approach with the one caveat below: The only issues with the prefix length are the scope and PKey fields potentially making this non contiguous but I think the syntax can be extended for this if needed. In terms of IB routers, if an MLID was overloaded, wouldn't they filter on scope as well (link local scope would be filtered), right ? > inet_pton/inet_ntop w/ AF_INET6 can be used, and is compatible with > the existing GID structure. > > I would recommend a longest prefix bits scheme rather than a mask for > administator sanity. The IPv6 addressing scheme that MGID/GID borrows > from was designed to work like this, and administrators will be > familiar with the concept. If you feel the complexity of non-prefix > matching is needed then it is easier on administrators to copy the > existing idom used in ACLs and use a two level label/property scheme: > > Table 1: > MGID/PREFIX GROUP NAME > ff11:601b::/32 IPV6-SNM > ff1e:601b::/32 IPV6-SNM > ff1e:601b:89:/48 IPV6-SNM-PKEY_89 > ff00::/8 OTHERS > > Table 2: > NAME MLIDS > IPV6-SNM 4 > IPV6_SNM-PKEY_89 128 > OTHERS 128 > > But I think that is probably overkill, unless you know of other heavy > multicast users besides SNM. I'm unaware of any other "applications" which might "require" this but I don't think there's much extra cost in the flexibility provided to accomodate these. > Also note that a longest prefix match data structure can likely be > re-used for future work on IB routing, along with this use for MGID > compaction. > > > The criteria for overloading MLIDs includes any group parameters that > > need to be in common (e.g. rate. MTU, perhaps PKey (see below), etc.). > > It seems reasonable to me that every MTU+rate gets a seperate > allocation space, each limited to the configured size. > > > As PKey is part of the MGID, does this need to be addressed (and if > > so) how ? > > I don't think pkey needs special treatment - prefix matching takes care > of it. If you feel seperate allocations for each pkey may someday be > necessary then having a table scheme rather than a single value takes > care of it nicely. So you think it's "safe" to ignore the related IBA compliance ? It does look to me like that should be changed. I think that was also Roland's take a while ago. -- Hal > Regards, > Jason From dotanba at gmail.com Sun Jun 8 09:10:19 2008 From: dotanba at gmail.com (Dotan Barak) Date: Sun, 08 Jun 2008 18:10:19 +0200 Subject: [ofa-general] SDP and epoll vs select() In-Reply-To: References: Message-ID: <484C046B.2090204@gmail.com> Hi. Bradley Kite wrote: > Hi all, > > Currently my application uses the Linux kernel's epoll interface for > socket event notifications. From what I've read it looks like the SDP > library only works with select()/poll() - is this actually the case or > will epoll work too? > I reviewed the libsdp code and epoll is not implemented in this library. Dotan From sashak at voltaire.com Sun Jun 8 09:03:10 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 8 Jun 2008 19:03:10 +0300 Subject: [ofa-general] [PATCH] opensm/configure.in: remove AM_MAINTAINER_MODE Message-ID: <20080608160310.GU7795@sashak.voltaire.com> This macro disables by default reconfiguration and Makefile.in, Makefile regeneration when configure.in and/or Makefile.am were changed - bad. Also it puts proper definition of ACLOCAL_AMFLAGS variable instead of ACLOCAL redefinition attempts. Signed-off-by: Sasha Khapyorsky --- opensm/Makefile.am | 5 +---- opensm/configure.in | 5 ----- 2 files changed, 1 insertions(+), 9 deletions(-) diff --git a/opensm/Makefile.am b/opensm/Makefile.am index 4c79f49..fca5319 100644 --- a/opensm/Makefile.am +++ b/opensm/Makefile.am @@ -3,10 +3,7 @@ SUBDIRS = complib libvendor opensm osmtest include $(DEFAULT_EVENT_PLUGIN) DIST_SUBDIRS = complib libvendor opensm osmtest include osmeventplugin -# this will control the update of the files in order -MAINTAINERCLEANFILES = Makefile.in aclocal.m4 configure config-h.in - -ACLOCAL = aclocal -I $(ac_aux_dir) +ACLOCAL_AMFLAGS = -I config # we should provide a hint for other apps about the build mode of this project install-exec-hook: diff --git a/opensm/configure.in b/opensm/configure.in index 3608a3a..409e7c6 100644 --- a/opensm/configure.in +++ b/opensm/configure.in @@ -13,11 +13,6 @@ AC_SUBST(TARBALL, ${TARBALL:-${PACKAGE}-${VERSION}.tar.gz}) dnl Defines the Language AC_LANG_C -dnl Provides control over re-making of all auto files -dnl We also use it to define swig dependencies so end -dnl users do not see them. -AM_MAINTAINER_MODE - dnl Required for cases make defines a MAKE=make ??? Why AC_PROG_MAKE_SET AC_PROG_CC -- 1.5.5.1.178.g1f811 From hrosenstock at xsigo.com Sun Jun 8 09:15:34 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sun, 08 Jun 2008 09:15:34 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080608054426.GK22807@obsidianresearch.com> <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> Message-ID: <1212941734.14047.42.camel@hrosenstock-ws.xsigo.com> On Sun, 2008-06-08 at 08:02 -0700, Hal Rosenstock wrote: > In terms of IB routers, if an MLID was overloaded, wouldn't they > filter on scope as well (link local scope would be filtered), right ? Should have said not forward rather than filter on link local scope as these might be locally terminated on the router. -- Hal From hrosenstock at xsigo.com Sun Jun 8 13:17:50 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sun, 08 Jun 2008 13:17:50 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/saquery.c: In dump_results, use query_result rather than query_svc_rec Message-ID: <1212956270.14047.52.camel@hrosenstock-ws.xsigo.com> infiniband-diags/saquery.c: In dump_results, use query_result rather than query_svc_rec Most queries are not for service records. If attribute identifier validation is desired, this should be implemented differently. Signed-off-by: Hal Rosenstock diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c index d97e686..e727940 100644 --- a/infiniband-diags/src/saquery.c +++ b/infiniband-diags/src/saquery.c @@ -659,7 +659,7 @@ static void dump_results(osmv_query_res_t *r, void (*dump_func)(void *)) { int i; for (i = 0; i < r->result_cnt; i++) { - void *data = osmv_get_query_svc_rec(r->p_result_madw, i); + void *data = osmv_get_query_result(r->p_result_madw, i); dump_func(data); } } From Feed at lists.openfabrics.org Sun Jun 8 18:42:59 2008 From: Feed at lists.openfabrics.org (Feed at lists.openfabrics.org) Date: 08 Jun 2008 18:42:59 -0700 Subject: [ofa-general] Feed Blaster puts your ad right to the screens of millions in 15 Minutes ! Message-ID: <20080608184258.8FADFEB916622B37@from.header.has.no.domain> More and more people are subscribing to feeds every day and there are millions who are already subscribed. Thus, your ad will reach a very broad range of potential customers with each use of Feed Blaster! Feed Blaster is the first & only submitter that can submit your ads to thousands of feeds within a few minutes! Post your ads where people read them! - What if you could place your ad into all these feeds ? Right, that would mean you would have millions of sites linking to your ad - and millions of users reading your message within minutes - and my idea actually works For Full details please read the attached .html file Unsubscribe: On Full details page click on contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From PHF at zurich.ibm.com Mon Jun 9 00:23:39 2008 From: PHF at zurich.ibm.com (Philip Frey1) Date: Mon, 9 Jun 2008 09:23:39 +0200 Subject: [ofa-general] missed cq event In-Reply-To: <4849730B.5060008@opengridcomputing.com> Message-ID: > > what is the correct way to get completions from the CQ without missing > > any? > > I am currently using two separate CQs for the SQ and for the RQ. > > In pseudocode I do about the following to wait for a completion: > > > > ibv_get_cq_event(channel, &dst_cq, &ctx); //blocking wait for > > CQ event > > <-- EVENT --> > > ibv_req_notify_cq(dst_cq, 0); // request event > > for next completion > > ibv_poll_cq(dest_cq, 1, wc); // get the WC from > > the CQ > > ibv_ack_cq_events(dst_cq, 1); // ack the event > > < snip > > > you need to poll the cq until it is empty instead of just polling once. > > > get_cq_event() > req_notify_cq() > do { > ne = poll_cq(); > } while (ne != 0); > ack_cq_event() But what if I am just interested in the next work completion? Wouldn't this mean that I have to build a queue in my application, that can hold completions in which I am not yet interested? It sounds like a replication of the completion queue to me. I was more looking for a way to do the 'single poll' on the completion without copying anything into another queue. Many thanks, Philip -------------- next part -------------- An HTML attachment was scrubbed... URL: From fenkes at de.ibm.com Mon Jun 9 04:24:20 2008 From: fenkes at de.ibm.com (Joachim Fenkes) Date: Mon, 9 Jun 2008 13:24:20 +0200 Subject: [ofa-general] [PATCH] IB/ehca: Reject recv WRs if QP is in RESET state In-Reply-To: <48499C11.7030504@gmail.com> References: <200806061835.43802.fenkes@de.ibm.com> <48499C11.7030504@gmail.com> Message-ID: <200806091324.20869.fenkes@de.ibm.com> Signed-off-by: Joachim Fenkes --- On Friday 06 June 2008 22:20, Dotan Barak wrote: > I checked the code in the ehca driver and noticed that > post RR to a QP is being accepted in any state (including the RESET state). You're right, this is only consistent -- thanks for pointing it out! Regards, Joachim drivers/infiniband/hw/ehca/ehca_reqs.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_reqs.c b/drivers/infiniband/hw/ehca/ehca_reqs.c index f093b00..ad197f4 100644 --- a/drivers/infiniband/hw/ehca/ehca_reqs.c +++ b/drivers/infiniband/hw/ehca/ehca_reqs.c @@ -544,8 +544,16 @@ int ehca_post_recv(struct ib_qp *qp, struct ib_recv_wr *recv_wr, struct ib_recv_wr **bad_recv_wr) { - return internal_post_recv(container_of(qp, struct ehca_qp, ib_qp), - qp->device, recv_wr, bad_recv_wr); + struct ehca_qp *my_qp = container_of(qp, struct ehca_qp, ib_qp); + + /* Reject WR if QP is in RESET state */ + if (unlikely(my_qp->state == IB_QPS_RESET)) { + ehca_err(qp->device, "Invalid QP state qp_state=%d qpn=%x", + my_qp->state, qp->qp_num); + return -EINVAL; + } + + return internal_post_recv(my_qp, qp->device, recv_wr, bad_recv_wr); } int ehca_post_srq_recv(struct ib_srq *srq, -- 1.5.5 From hrosenstock at xsigo.com Mon Jun 9 06:24:26 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 06:24:26 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/osm_sa_portinfo_record.c: Cosmetic comment change Message-ID: <1213017866.14047.73.camel@hrosenstock-ws.xsigo.com> OpenSM/osm_sa_portinfo_record.c: Cosmetic comment change Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_portinfo_record.c b/opensm/opensm/osm_sa_portinfo_record.c index 8a5ed6d..88d8c23 100644 --- a/opensm/opensm/osm_sa_portinfo_record.c +++ b/opensm/opensm/osm_sa_portinfo_record.c @@ -422,8 +422,8 @@ __osm_sa_pir_by_comp_mask(IN osm_sa_t * sa, p_physp = osm_node_get_physp_ptr(p_node, p_rcvd_rec->port_num); - /* Check that the p_physp is valid, and that the p_physp and the - p_req_physp share a pkey. */ + /* Check that the p_physp is valid, and that the + p_physp and the p_req_physp share a pkey. */ if (p_physp && osm_physp_share_pkey(sa->p_log, p_req_physp, p_physp)) From hrosenstock at xsigo.com Mon Jun 9 06:24:20 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 06:24:20 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/libvendor/osm_vendor_ibumad_sa.c: Cosmetic changes Message-ID: <1213017860.14047.72.camel@hrosenstock-ws.xsigo.com> opensm/libvendor/osm_vendor_ibumad_sa.c: Cosmetic changes Signed-off-by: Hal Rosenstock diff --git a/opensm/libvendor/osm_vendor_ibumad_sa.c b/opensm/libvendor/osm_vendor_ibumad_sa.c index b109eb2..547d1e5 100644 --- a/opensm/libvendor/osm_vendor_ibumad_sa.c +++ b/opensm/libvendor/osm_vendor_ibumad_sa.c @@ -369,7 +369,7 @@ __osmv_send_sa_req(IN osmv_sa_bind_info_t * p_bind, /* since the sm_lid might change we obtain it every send (actually it is cached in the bind object and refreshed - every 30sec by this proc ) + every 30sec by this proc) */ if (time(NULL) > p_bind->last_lids_update_sec + 30) { status = update_umad_port(p_bind->p_vendor); @@ -462,7 +462,7 @@ __osmv_send_sa_req(IN osmv_sa_bind_info_t * p_bind, /* if synchronous - wait on the event */ if (sync) { osm_log(p_log, OSM_LOG_DEBUG, - "__osmv_send_sa_req: " "Waiting for async event\n"); + "__osmv_send_sa_req: Waiting for async event\n"); cl_event_wait_on(&p_bind->sync_event, EVENT_NO_TIMEOUT, FALSE); cl_event_reset(&p_bind->sync_event); status = p_madw->status; From swise at opengridcomputing.com Mon Jun 9 06:56:46 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 09 Jun 2008 08:56:46 -0500 Subject: [ofa-general] missed cq event In-Reply-To: References: Message-ID: <484D369E.70400@opengridcomputing.com> Philip Frey1 wrote: > > > > what is the correct way to get completions from the CQ without > missing > > > any? > > > I am currently using two separate CQs for the SQ and for the RQ. > > > In pseudocode I do about the following to wait for a completion: > > > > > > ibv_get_cq_event(channel, &dst_cq, &ctx); //blocking wait for > > > CQ event > > > <-- EVENT --> > > > ibv_req_notify_cq(dst_cq, 0); // request event > > > for next completion > > > ibv_poll_cq(dest_cq, 1, wc); // get the WC > from > > > the CQ > > > ibv_ack_cq_events(dst_cq, 1); // ack the event > > > > > < snip > > > > > > you need to poll the cq until it is empty instead of just polling once. > > > > > > get_cq_event() > > req_notify_cq() > > do { > > ne = poll_cq(); > > } while (ne != 0); > > ack_cq_event() > > But what if I am just interested in the next work completion? Wouldn't > this mean > that I have to build a queue in my application, that can hold > completions in which > I am not yet interested? It sounds like a replication of the > completion queue to me. > I was more looking for a way to do the 'single poll' on the completion > without copying > anything into another queue. You can process each polled CQE individually inside the loop, yes? How is that different from getting notified of each CQE insertion and processing it then? From hrosenstock at xsigo.com Mon Jun 9 07:44:27 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 07:44:27 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/osm_sa_mcmember_record.c: Cosmetic change to error log message Message-ID: <1213022667.14047.77.camel@hrosenstock-ws.xsigo.com> OpenSM/osm_sa_mcmember_record.c: Cosmetic change to error log message Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index cc8b1d7..0168021 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1882,7 +1882,7 @@ void osm_mcmr_rcv_process(IN void *context, IN void *data) if (!valid) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B18: " "component mask = 0x%016" PRIx64 ", " - "expected comp mask = 0x%016" PRIx64 " ," + "expected comp mask = 0x%016" PRIx64 ", " "MGID: 0x%016" PRIx64 " : 0x%016" PRIx64 " for PortGID: 0x%016" PRIx64 " : 0x%016" PRIx64 "\n", cl_ntoh64(p_sa_mad->comp_mask), From hrosenstock at xsigo.com Mon Jun 9 07:45:39 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 07:45:39 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <20080606115350.GL7795@sashak.voltaire.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> Message-ID: <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-06 at 14:53 +0300, Sasha Khapyorsky wrote: > Actually there is a better tool under ibsim/tests - mcast_storm. Should this be under OpenFabrics license rather than GPLv2 or later ? -- Hal From PHF at zurich.ibm.com Mon Jun 9 08:40:59 2008 From: PHF at zurich.ibm.com (Philip Frey1) Date: Mon, 9 Jun 2008 17:40:59 +0200 Subject: [ofa-general] missed cq event In-Reply-To: <484D369E.70400@opengridcomputing.com> Message-ID: You are right. Thanks! I have yet another issue: Sometimes I get the following message in /var/log/messages of the local host: post_qp_event - AE qpid 0x4e0 opcode 3 status 0x13 type 0 wrid.hi 0x0 wrid.lo 0x65000000 I was looking for the status and opcode in the source and found that opcode 3 means T3_SEND and status 0x13 means TPT_ERR_OUT_OF_RQE. At the remote host I get and opcode 7 (T3_TERMINATE) and status 0x0 (SUCCESS). Clearly there is someone running out of Receive Queue Elements. The error occurred when doing an ibv_post_send() at the local host. Is this a coincidence or does the local host somehow know that there are not enough RQE's available at the remote host? In other words, does the TPT_ERR_OUT_OF_RQE refer to the local or to the remote receive queue? Many thanks, Philip general-bounces at lists.openfabrics.org wrote on 09.06.2008 15:56:46: > Philip Frey1 wrote: > > > > > > what is the correct way to get completions from the CQ without > > missing > > > > any? > > > > I am currently using two separate CQs for the SQ and for the RQ. > > > > In pseudocode I do about the following to wait for a completion: > > > > > > > > ibv_get_cq_event(channel, &dst_cq, &ctx); //blocking wait for > > > > CQ event > > > > <-- EVENT --> > > > > ibv_req_notify_cq(dst_cq, 0); // request event > > > > for next completion > > > > ibv_poll_cq(dest_cq, 1, wc); // get the WC > > from > > > > the CQ > > > > ibv_ack_cq_events(dst_cq, 1); // ack the event > > > > > > > > < snip > > > > > > > > > you need to poll the cq until it is empty instead of just polling once. > > > > > > > > > get_cq_event() > > > req_notify_cq() > > > do { > > > ne = poll_cq(); > > > } while (ne != 0); > > > ack_cq_event() > > > > But what if I am just interested in the next work completion? Wouldn't > > this mean > > that I have to build a queue in my application, that can hold > > completions in which > > I am not yet interested? It sounds like a replication of the > > completion queue to me. > > I was more looking for a way to do the 'single poll' on the completion > > without copying > > anything into another queue. > > You can process each polled CQE individually inside the loop, yes? How > is that different from getting notified of each CQE insertion and > processing it then? > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From ossrosch at linux.vnet.ibm.com Mon Jun 9 08:42:28 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Mon, 9 Jun 2008 17:42:28 +0200 Subject: [ofa-general] [PATCH 0/2] Prevent loss of interrupts in IB/ehca Message-ID: <200806091742.29421.ossrosch@linux.vnet.ibm.com> This patchset contains two changes for IB/ehca and ibmebus. The first patch enables ibmebus_request_irq() to optionally return the IRQ number, which is used by the second patch to trigger EOI in case of lost interrupts. They should apply cleanly against 2.6.26 git tree. Thanks Stefan From ossrosch at linux.vnet.ibm.com Mon Jun 9 08:44:29 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Mon, 9 Jun 2008 17:44:29 +0200 Subject: [ofa-general] [PATCH 1/2] ibmebus: Change ibmebus_request_irq() to optionally return irq number Message-ID: <200806091744.30930.ossrosch@linux.vnet.ibm.com> Signed-off-by: Stefan Roscher --- arch/powerpc/kernel/ibmebus.c | 5 ++++- drivers/infiniband/hw/ehca/ehca_eq.c | 4 ++-- drivers/net/ehea/ehea_main.c | 6 +++--- include/asm-powerpc/ibmebus.h | 2 +- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/kernel/ibmebus.c b/arch/powerpc/kernel/ibmebus.c index 9971159..a002fdf 100644 --- a/arch/powerpc/kernel/ibmebus.c +++ b/arch/powerpc/kernel/ibmebus.c @@ -208,7 +208,7 @@ void ibmebus_unregister_driver(struct of_platform_driver *drv) } EXPORT_SYMBOL(ibmebus_unregister_driver); -int ibmebus_request_irq(u32 ist, irq_handler_t handler, +int ibmebus_request_irq(u32 ist, int *irq_number, irq_handler_t handler, unsigned long irq_flags, const char *devname, void *dev_id) { @@ -217,6 +217,9 @@ int ibmebus_request_irq(u32 ist, irq_handler_t handler, if (irq == NO_IRQ) return -EINVAL; + if (irq_number) + *irq_number = irq; + return request_irq(irq, handler, irq_flags, devname, dev_id); } EXPORT_SYMBOL(ibmebus_request_irq); diff --git a/drivers/infiniband/hw/ehca/ehca_eq.c b/drivers/infiniband/hw/ehca/ehca_eq.c index 49660df..5bc494f 100644 --- a/drivers/infiniband/hw/ehca/ehca_eq.c +++ b/drivers/infiniband/hw/ehca/ehca_eq.c @@ -122,7 +122,7 @@ int ehca_create_eq(struct ehca_shca *shca, /* register interrupt handlers and initialize work queues */ if (type == EHCA_EQ) { - ret = ibmebus_request_irq(eq->ist, ehca_interrupt_eq, + ret = ibmebus_request_irq(eq->ist, NULL, ehca_interrupt_eq, IRQF_DISABLED, "ehca_eq", (void *)shca); if (ret < 0) @@ -130,7 +130,7 @@ int ehca_create_eq(struct ehca_shca *shca, tasklet_init(&eq->interrupt_task, ehca_tasklet_eq, (long)shca); } else if (type == EHCA_NEQ) { - ret = ibmebus_request_irq(eq->ist, ehca_interrupt_neq, + ret = ibmebus_request_irq(eq->ist, NULL, ehca_interrupt_neq, IRQF_DISABLED, "ehca_neq", (void *)shca); if (ret < 0) diff --git a/drivers/net/ehea/ehea_main.c b/drivers/net/ehea/ehea_main.c index 287a619..102ffeb 100644 --- a/drivers/net/ehea/ehea_main.c +++ b/drivers/net/ehea/ehea_main.c @@ -1216,7 +1216,7 @@ static int ehea_reg_interrupts(struct net_device *dev) snprintf(port->int_aff_name, EHEA_IRQ_NAME_SIZE - 1, "%s-aff", dev->name); - ret = ibmebus_request_irq(port->qp_eq->attr.ist1, + ret = ibmebus_request_irq(port->qp_eq->attr.ist1, NULL, ehea_qp_aff_irq_handler, IRQF_DISABLED, port->int_aff_name, port); if (ret) { @@ -1234,7 +1234,7 @@ static int ehea_reg_interrupts(struct net_device *dev) pr = &port->port_res[i]; snprintf(pr->int_send_name, EHEA_IRQ_NAME_SIZE - 1, "%s-queue%d", dev->name, i); - ret = ibmebus_request_irq(pr->eq->attr.ist1, + ret = ibmebus_request_irq(pr->eq->attr.ist1, NULL, ehea_recv_irq_handler, IRQF_DISABLED, pr->int_send_name, pr); @@ -3414,7 +3414,7 @@ static int __devinit ehea_probe_adapter(struct of_device *dev, tasklet_init(&adapter->neq_tasklet, ehea_neq_tasklet, (unsigned long)adapter); - ret = ibmebus_request_irq(adapter->neq->attr.ist1, + ret = ibmebus_request_irq(adapter->neq->attr.ist1, NULL, ehea_interrupt_neq, IRQF_DISABLED, "ehea_neq", adapter); if (ret) { diff --git a/include/asm-powerpc/ibmebus.h b/include/asm-powerpc/ibmebus.h index 1a9d9ae..3a2618a 100644 --- a/include/asm-powerpc/ibmebus.h +++ b/include/asm-powerpc/ibmebus.h @@ -51,7 +51,7 @@ extern struct bus_type ibmebus_bus_type; int ibmebus_register_driver(struct of_platform_driver *drv); void ibmebus_unregister_driver(struct of_platform_driver *drv); -int ibmebus_request_irq(u32 ist, irq_handler_t handler, +int ibmebus_request_irq(u32 ist, int *irq_number, irq_handler_t handler, unsigned long irq_flags, const char *devname, void *dev_id); void ibmebus_free_irq(u32 ist, void *dev_id); -- 1.5.5 From ossrosch at linux.vnet.ibm.com Mon Jun 9 08:45:17 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Mon, 9 Jun 2008 17:45:17 +0200 Subject: [ofa-general] [PATCH 2/2] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts Message-ID: <200806091745.19147.ossrosch@linux.vnet.ibm.com> Signed-off-by: Stefan Roscher --- drivers/infiniband/hw/ehca/ehca_classes.h | 1 + drivers/infiniband/hw/ehca/ehca_eq.c | 6 ++++-- drivers/infiniband/hw/ehca/ehca_main.c | 12 ++++++++++-- 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_classes.h b/drivers/infiniband/hw/ehca/ehca_classes.h index 1e9e99a..4de363d 100644 --- a/drivers/infiniband/hw/ehca/ehca_classes.h +++ b/drivers/infiniband/hw/ehca/ehca_classes.h @@ -86,6 +86,7 @@ struct ehca_eq { u32 ist; spinlock_t irq_spinlock; struct ehca_eqe_cache_entry eqe_cache[EHCA_EQE_CACHE_SIZE]; + int irq_number; }; struct ehca_sma_attr { diff --git a/drivers/infiniband/hw/ehca/ehca_eq.c b/drivers/infiniband/hw/ehca/ehca_eq.c index 5bc494f..b70e5e5 100644 --- a/drivers/infiniband/hw/ehca/ehca_eq.c +++ b/drivers/infiniband/hw/ehca/ehca_eq.c @@ -122,7 +122,8 @@ int ehca_create_eq(struct ehca_shca *shca, /* register interrupt handlers and initialize work queues */ if (type == EHCA_EQ) { - ret = ibmebus_request_irq(eq->ist, NULL, ehca_interrupt_eq, + ret = ibmebus_request_irq(eq->ist, &eq->irq_number, + ehca_interrupt_eq, IRQF_DISABLED, "ehca_eq", (void *)shca); if (ret < 0) @@ -130,7 +131,8 @@ int ehca_create_eq(struct ehca_shca *shca, tasklet_init(&eq->interrupt_task, ehca_tasklet_eq, (long)shca); } else if (type == EHCA_NEQ) { - ret = ibmebus_request_irq(eq->ist, NULL, ehca_interrupt_neq, + ret = ibmebus_request_irq(eq->ist, &eq->irq_number, + ehca_interrupt_neq, IRQF_DISABLED, "ehca_neq", (void *)shca); if (ret < 0) diff --git a/drivers/infiniband/hw/ehca/ehca_main.c b/drivers/infiniband/hw/ehca/ehca_main.c index 482103e..d713317 100644 --- a/drivers/infiniband/hw/ehca/ehca_main.c +++ b/drivers/infiniband/hw/ehca/ehca_main.c @@ -44,6 +44,7 @@ #include #endif +#include #include "ehca_classes.h" #include "ehca_iverbs.h" #include "ehca_mrmw.h" @@ -937,6 +938,8 @@ static struct of_platform_driver ehca_driver = { void ehca_poll_eqs(unsigned long data) { struct ehca_shca *shca; + int irq; + irq_desc_t *desc; spin_lock(&shca_list_lock); list_for_each_entry(shca, &shca_list, shca_list) { @@ -955,8 +958,13 @@ void ehca_poll_eqs(unsigned long data) spin_unlock_irqrestore(&eq->spinlock, flags); max--; } while (q_ofs == q_ofs2 && max > 0); - if (q_ofs == q_ofs2) - ehca_process_eq(shca, 0); + if (q_ofs == q_ofs2) { + irq = shca->eq.irq_number; + desc = get_irq_desc(irq); + if (desc->chip && desc->chip->eoi) + desc->chip->eoi(irq); + tasklet_hi_schedule(&shca->eq.interrupt_task); + } } } mod_timer(&poll_eqs_timer, round_jiffies(jiffies + HZ)); -- 1.5.5 From THEMANN at de.ibm.com Mon Jun 9 08:57:39 2008 From: THEMANN at de.ibm.com (Jan-Bernd Themann) Date: Mon, 9 Jun 2008 17:57:39 +0200 Subject: [ofa-general] Re: [PATCH 1/2] ibmebus: Change ibmebus_request_irq() to optionally return irq number In-Reply-To: <200806091744.30930.ossrosch@linux.vnet.ibm.com> Message-ID: Stefan Roscher wrote on 09.06.2008 17:44:29: > Signed-off-by: Stefan Roscher > --- > arch/powerpc/kernel/ibmebus.c | 5 ++++- > drivers/infiniband/hw/ehca/ehca_eq.c | 4 ++-- > drivers/net/ehea/ehea_main.c | 6 +++--- > include/asm-powerpc/ibmebus.h | 2 +- > 4 files changed, 10 insertions(+), 7 deletions(-) > > diff --git a/arch/powerpc/kernel/ibmebus.c b/arch/powerpc/kernel/ibmebus.c > index 9971159..a002fdf 100644 > --- a/arch/powerpc/kernel/ibmebus.c > +++ b/arch/powerpc/kernel/ibmebus.c > @@ -208,7 +208,7 @@ void ibmebus_unregister_driver(struct > of_platform_driver *drv) > } > EXPORT_SYMBOL(ibmebus_unregister_driver); > > -int ibmebus_request_irq(u32 ist, irq_handler_t handler, > +int ibmebus_request_irq(u32 ist, int *irq_number, irq_handler_t handler, > unsigned long irq_flags, const char *devname, > void *dev_id) > { > @@ -217,6 +217,9 @@ int ibmebus_request_irq(u32 ist, irq_handler_t handler, > if (irq == NO_IRQ) > return -EINVAL; > > + if (irq_number) > + *irq_number = irq; > + > return request_irq(irq, handler, irq_flags, devname, dev_id); > } > EXPORT_SYMBOL(ibmebus_request_irq); > diff --git a/drivers/infiniband/hw/ehca/ehca_eq.c > b/drivers/infiniband/hw/ehca/ehca_eq.c > index 49660df..5bc494f 100644 > --- a/drivers/infiniband/hw/ehca/ehca_eq.c > +++ b/drivers/infiniband/hw/ehca/ehca_eq.c > @@ -122,7 +122,7 @@ int ehca_create_eq(struct ehca_shca *shca, > > /* register interrupt handlers and initialize work queues */ > if (type == EHCA_EQ) { > - ret = ibmebus_request_irq(eq->ist, ehca_interrupt_eq, > + ret = ibmebus_request_irq(eq->ist, NULL, ehca_interrupt_eq, > IRQF_DISABLED, "ehca_eq", > (void *)shca); > if (ret < 0) > @@ -130,7 +130,7 @@ int ehca_create_eq(struct ehca_shca *shca, > > tasklet_init(&eq->interrupt_task, ehca_tasklet_eq, (long)shca); > } else if (type == EHCA_NEQ) { > - ret = ibmebus_request_irq(eq->ist, ehca_interrupt_neq, > + ret = ibmebus_request_irq(eq->ist, NULL, ehca_interrupt_neq, > IRQF_DISABLED, "ehca_neq", > (void *)shca); > if (ret < 0) > diff --git a/drivers/net/ehea/ehea_main.c b/drivers/net/ehea/ehea_main.c > index 287a619..102ffeb 100644 > --- a/drivers/net/ehea/ehea_main.c > +++ b/drivers/net/ehea/ehea_main.c > @@ -1216,7 +1216,7 @@ static int ehea_reg_interrupts(struct net_device *dev) > snprintf(port->int_aff_name, EHEA_IRQ_NAME_SIZE - 1, "%s-aff", > dev->name); > > - ret = ibmebus_request_irq(port->qp_eq->attr.ist1, > + ret = ibmebus_request_irq(port->qp_eq->attr.ist1, NULL, > ehea_qp_aff_irq_handler, > IRQF_DISABLED, port->int_aff_name, port); > if (ret) { > @@ -1234,7 +1234,7 @@ static int ehea_reg_interrupts(struct net_device *dev) > pr = &port->port_res[i]; > snprintf(pr->int_send_name, EHEA_IRQ_NAME_SIZE - 1, > "%s-queue%d", dev->name, i); > - ret = ibmebus_request_irq(pr->eq->attr.ist1, > + ret = ibmebus_request_irq(pr->eq->attr.ist1, NULL, > ehea_recv_irq_handler, > IRQF_DISABLED, pr->int_send_name, > pr); > @@ -3414,7 +3414,7 @@ static int __devinit ehea_probe_adapter(struct > of_device *dev, > tasklet_init(&adapter->neq_tasklet, ehea_neq_tasklet, > (unsigned long)adapter); > > - ret = ibmebus_request_irq(adapter->neq->attr.ist1, > + ret = ibmebus_request_irq(adapter->neq->attr.ist1, NULL, > ehea_interrupt_neq, IRQF_DISABLED, > "ehea_neq", adapter); > if (ret) { > diff --git a/include/asm-powerpc/ibmebus.h b/include/asm-powerpc/ibmebus.h > index 1a9d9ae..3a2618a 100644 > --- a/include/asm-powerpc/ibmebus.h > +++ b/include/asm-powerpc/ibmebus.h > @@ -51,7 +51,7 @@ extern struct bus_type ibmebus_bus_type; > int ibmebus_register_driver(struct of_platform_driver *drv); > void ibmebus_unregister_driver(struct of_platform_driver *drv); > > -int ibmebus_request_irq(u32 ist, irq_handler_t handler, > +int ibmebus_request_irq(u32 ist, int *irq_number, irq_handler_t handler, > unsigned long irq_flags, const char *devname, > void *dev_id); > void ibmebus_free_irq(u32 ist, void *dev_id); > -- > 1.5.5 > Concerning the eHEA part: Acked-by: Jan-Bernd Themann Regards, Jan-Bernd -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Mon Jun 9 09:20:17 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 09:20:17 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/osm_sa_path_record.c: Add some information to some error log messages Message-ID: <1213028417.14047.95.camel@hrosenstock-ws.xsigo.com> OpenSM/osm_sa_path_record.c: Add some information to some error log messages Also, some commentary changes to this and osm_sa_multipath_record.c Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_multipath_record.c b/opensm/opensm/osm_sa_multipath_record.c index ad2e1a0..f8da64d 100644 --- a/opensm/opensm/osm_sa_multipath_record.c +++ b/opensm/opensm/osm_sa_multipath_record.c @@ -1177,8 +1177,8 @@ __osm_mpr_rcv_get_gids(IN osm_sa_t * sa, (ib_gid_get_subnet_prefix(gids) != sa->p_subn->opt.subnet_prefix)) { /* - This 'error' is the client's fault (bad gid) so - don't enter it as an error in our own log. + This 'error' is the client's fault (bad gid) + so don't enter it as an error in our own log. Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, "ERR 451B: " diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 65e6a74..b2ce54d 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -1176,8 +1176,8 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, if (ib_gid_get_subnet_prefix(&p_pr->sgid) != sa->p_subn->opt.subnet_prefix) { /* - This 'error' is the client's fault (bad gid) so - don't enter it as an error in our own log. + This 'error' is the client's fault (bad gid) + so don't enter it as an error in our own log. Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, @@ -1220,7 +1220,7 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No source port with LID = 0x%X\n", + "No source port with LID 0x%X\n", cl_ntoh16(p_pr->slid)); sa_status = IB_SA_MAD_STATUS_NO_RECORDS; @@ -1243,8 +1243,8 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, PRIx64 "\n", cl_ntoh64(p_pr->dgid.unicast.prefix)); - /* Find the router port that is configured to handle - this prefix, if any: */ + /* Find the router port that is configured to + handle this prefix, if any */ osm_prefix_route_t *route = NULL; osm_prefix_route_t *r = (osm_prefix_route_t *) cl_qlist_head(&sa->p_subn->prefix_routes_list); @@ -1326,12 +1326,12 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, if ((status != CL_SUCCESS) || (*pp_dest_port == NULL)) { /* - This 'error' is the client's fault (bad lid) so - don't enter it as an error in our own log. + This 'error' is the client's fault (bad lid) + so don't enter it as an error in our own log. Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No dest port with LID = 0x%X\n", + "No dest port with LID 0x%X\n", cl_ntoh16(p_pr->dlid)); sa_status = IB_SA_MAD_STATUS_NO_RECORDS; @@ -1504,7 +1504,9 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, status = osm_get_mgrp_by_mgid(sa, &p_pr->dgid, pp_mgrp); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F09: " - "No MC group found for PathRecord destination GID\n"); + "No MC group found for PathRecord destination GID 0x%016" PRIx64 " : " "0x%016" PRIx64 "\n", + cl_ntoh64(p_pr->dgid.unicast.prefix), + cl_ntoh64(p_pr->dgid.unicast.interface_id)); goto Exit; } } @@ -1516,7 +1518,7 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, if ((*pp_mgrp)->mlid != p_pr->dlid) { /* Note: perhaps this might be better indicated as an invalid request */ OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F10: " - "MC group MLID does not match PathRecord destination LID\n"); + "MC group MLID 0x%x does not match PathRecord destination LID 0x%x\n", (*pp_mgrp)->mlid, p_pr->dlid); *pp_mgrp = NULL; goto Exit; } @@ -1524,7 +1526,7 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, *pp_mgrp = __get_mgrp_by_mlid(sa, p_pr->dlid); if (*pp_mgrp == NULL) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F11: " - "No MC group found for PathRecord destination LID\n"); + "No MC group found for PathRecord destination LID 0x%x\n", p_pr->dlid); } } From swise at opengridcomputing.com Mon Jun 9 09:28:08 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 09 Jun 2008 11:28:08 -0500 Subject: [ofa-general] missed cq event In-Reply-To: References: Message-ID: <484D5A18.8070701@opengridcomputing.com> An HTML attachment was scrubbed... URL: From rdreier at cisco.com Mon Jun 9 09:38:59 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Jun 2008 09:38:59 -0700 Subject: [ofa-general] Re: [PATCH 2/2] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806091745.19147.ossrosch@linux.vnet.ibm.com> (Stefan Roscher's message of "Mon, 9 Jun 2008 17:45:17 +0200") References: <200806091745.19147.ossrosch@linux.vnet.ibm.com> Message-ID: > + if (desc->chip && desc->chip->eoi) > + desc->chip->eoi(irq); This doesn't seem like the type of thing a device driver should be doing. Both patches are rather short on changelog text -- what is the issue here and how does this fix it? - R. From sashak at voltaire.com Mon Jun 9 10:57:46 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 20:57:46 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/saquery.c: In dump_results, use query_result rather than query_svc_rec In-Reply-To: <1212956270.14047.52.camel@hrosenstock-ws.xsigo.com> References: <1212956270.14047.52.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609175746.GB24307@sashak.voltaire.com> On 13:17 Sun 08 Jun , Hal Rosenstock wrote: > infiniband-diags/saquery.c: In dump_results, use query_result rather > than query_svc_rec > > Most queries are not for service records. > > If attribute identifier validation is desired, this should be > implemented differently. > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Jun 9 11:02:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 21:02:21 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/libvendor/osm_vendor_ibumad_sa.c: Cosmetic changes In-Reply-To: <1213017860.14047.72.camel@hrosenstock-ws.xsigo.com> References: <1213017860.14047.72.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609180221.GC24307@sashak.voltaire.com> On 06:24 Mon 09 Jun , Hal Rosenstock wrote: > opensm/libvendor/osm_vendor_ibumad_sa.c: Cosmetic changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Jun 9 11:10:15 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 21:10:15 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609181015.GD24307@sashak.voltaire.com> On 07:45 Mon 09 Jun , Hal Rosenstock wrote: > > Should this be under OpenFabrics license rather than GPLv2 or later ? What is wrong with GPLv2? In worst case we can consider this as non-OFA software - it is not compilable by default and not part of OFED RPM. Sasha From sashak at voltaire.com Mon Jun 9 11:22:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 21:22:02 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] OpenSM/osm_sa_path_record.c: Add some information to some error log messages In-Reply-To: <1213028417.14047.95.camel@hrosenstock-ws.xsigo.com> References: <1213028417.14047.95.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609182202.GE24307@sashak.voltaire.com> On 09:20 Mon 09 Jun , Hal Rosenstock wrote: > OpenSM/osm_sa_path_record.c: Add some information to some error log > messages > Also, some commentary changes to this and osm_sa_multipath_record.c > > Signed-off-by: Hal Rosenstock Applied. Thanks. And few comments are below. > @@ -1220,7 +1220,7 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, > Return an error response to the client. > */ > OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, > - "No source port with LID = 0x%X\n", > + "No source port with LID 0x%X\n", AFAIR we decided to use decimal representation for unicast LIDs. I know that this patch has another goal, just reminder... (for me :)) > @@ -1516,7 +1518,7 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, > if ((*pp_mgrp)->mlid != p_pr->dlid) { > /* Note: perhaps this might be better indicated as an invalid request */ > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F10: " > - "MC group MLID does not match PathRecord destination LID\n"); > + "MC group MLID 0x%x does not match PathRecord destination LID 0x%x\n", (*pp_mgrp)->mlid, p_pr->dlid); > *pp_mgrp = NULL; > goto Exit; > } > @@ -1524,7 +1526,7 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, > *pp_mgrp = __get_mgrp_by_mlid(sa, p_pr->dlid); > if (*pp_mgrp == NULL) > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F11: " > - "No MC group found for PathRecord destination LID\n"); > + "No MC group found for PathRecord destination LID 0x%x\n", p_pr->dlid); Break long lines, please. Sasha From hrosenstock at xsigo.com Mon Jun 9 11:24:37 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 11:24:37 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <20080609181015.GD24307@sashak.voltaire.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> <20080609181015.GD24307@sashak.voltaire.com> Message-ID: <1213035877.14047.114.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-09 at 21:10 +0300, Sasha Khapyorsky wrote: > On 07:45 Mon 09 Jun , Hal Rosenstock wrote: > > > > Should this be under OpenFabrics license rather than GPLv2 or later ? > > What is wrong with GPLv2? Isn't this OFA software ? Also, didn't this stem from mcm_rereg_test.c which is under OFA license ? > In worst case we can consider this as non-OFA > software - it is not compilable by default and not part of OFED RPM. Looks to be part of tar ball; not sure about RPM. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Mon Jun 9 11:31:48 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 21:31:48 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <1213035877.14047.114.camel@hrosenstock-ws.xsigo.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> <20080609181015.GD24307@sashak.voltaire.com> <1213035877.14047.114.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609183148.GF24307@sashak.voltaire.com> On 11:24 Mon 09 Jun , Hal Rosenstock wrote: > > Isn't this OFA software ? Also, didn't this stem from mcm_rereg_test.c > which is under OFA license ? No, I wrote both from scratch. Also the license it is copyright holder choice. > > In worst case we can consider this as non-OFA > > software - it is not compilable by default and not part of OFED RPM. > > Looks to be part of tar ball; > not sure about RPM. It is in tarball, but since it is not really needed part of ibsim (rather reference code examples) it is not compilable by default there and it doesn't enter RPM (for this spec file change would be needed). Sasha From hrosenstock at xsigo.com Mon Jun 9 11:51:09 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 11:51:09 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <20080609181015.GD24307@sashak.voltaire.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> <20080609181015.GD24307@sashak.voltaire.com> Message-ID: <1213037469.14047.121.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-09 at 21:10 +0300, Sasha Khapyorsky wrote: > What is wrong with GPLv2? It wasn't sufficient for OpenFabrics as there are some people who prefer to take software under BSD r.t. GPL. So is there a reason this can't be changed ? -- Hal From hrosenstock at xsigo.com Mon Jun 9 12:07:19 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 12:07:19 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/TODO: Eliminate inet sockets from todo per previous README change Message-ID: <1213038439.14047.125.camel@hrosenstock-ws.xsigo.com> ibsim/TODO: Eliminate inet sockets from todo per previous README change Signed-off-by: Hal Rosenstock diff --git a/TODO b/TODO index f7be326..616ed71 100644 --- a/TODO +++ b/TODO @@ -1,4 +1,3 @@ -* inet sockets support * transaction management - multiple transactions per client - hops simulation From bradley.kite at gmail.com Mon Jun 9 12:09:17 2008 From: bradley.kite at gmail.com (Bradley Kite) Date: Mon, 9 Jun 2008 20:09:17 +0100 Subject: [ofa-general] SDP and epoll vs select() In-Reply-To: <484C046B.2090204@gmail.com> References: <484C046B.2090204@gmail.com> Message-ID: On 08/06/2008, Dotan Barak wrote: > Hi. > > > Bradley Kite wrote: > > > Hi all, > > > > Currently my application uses the Linux kernel's epoll interface for > > socket event notifications. From what I've read it looks like the SDP > > library only works with select()/poll() - is this actually the case or > > will epoll work too? > > > > > I reviewed the libsdp code and epoll is not implemented in this library. > > > Dotan > Hi Dotan. I too had looked in libsdp and found no epoll implementation. I thought I'd ask tho, because it could be that the direct implementation of SDP (by using AF_INET_SDP for socket() calls) causes epoll event notifications within the kernel itself - ie without the need to use libsdp. I've looked in the ofa_kernel package and found no reference to epoll so it doesn't look like its implemented there either, but don't know if I'm looking in the right place. For now I think I will be restrained to using TCP over infiniband because I don't like the idea of mixing epoll with select() in my applications event loop. But still, epoll() coupled with AF_INET_SDP would be better. Many thanks -- Brad. From sashak at voltaire.com Mon Jun 9 12:26:35 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 22:26:35 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <1213037469.14047.121.camel@hrosenstock-ws.xsigo.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> <20080609181015.GD24307@sashak.voltaire.com> <1213037469.14047.121.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609192635.GA10301@sashak.voltaire.com> On 11:51 Mon 09 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-09 at 21:10 +0300, Sasha Khapyorsky wrote: > > What is wrong with GPLv2? > > It wasn't sufficient for OpenFabrics as there are some people who prefer > to take software under BSD r.t. GPL. I know example when BSD software was copied as base for proprietary stuff and in this way this open source at the moment lost some critical development resources, attention, etc.. > So is there a reason this can't be changed ? Personally I prefer to get a changes back to community and GPLv2 works better here. In this specific case copyright holder is Voltaire (not Sasha K), we will need to ask there. Sasha From sashak at voltaire.com Mon Jun 9 12:28:46 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Jun 2008 22:28:46 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/TODO: Eliminate inet sockets from todo per previous README change In-Reply-To: <1213038439.14047.125.camel@hrosenstock-ws.xsigo.com> References: <1213038439.14047.125.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609192846.GB10301@sashak.voltaire.com> On 12:07 Mon 09 Jun , Hal Rosenstock wrote: > ibsim/TODO: Eliminate inet sockets from todo per previous README change > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hrosenstock at xsigo.com Mon Jun 9 12:33:27 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 09 Jun 2008 12:33:27 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <20080609192635.GA10301@sashak.voltaire.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> <20080609181015.GD24307@sashak.voltaire.com> <1213037469.14047.121.camel@hrosenstock-ws.xsigo.com> <20080609192635.GA10301@sashak.voltaire.com> Message-ID: <1213040007.14047.139.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-09 at 22:26 +0300, Sasha Khapyorsky wrote: > > So is there a reason this can't be changed ? > > Personally I prefer to get a changes back to community and GPLv2 works > better here. Me too but OpenFabrics has had this dual license since day one AFAIR. > In this specific case copyright holder is Voltaire (not > Sasha K), we will need to ask there. Thanks. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Mon Jun 9 13:10:15 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Jun 2008 13:10:15 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This will get a fix for a crash/lockup triggerable by buggy userspace (that I already sent you as a patch via email), and remove one API that only appeared in 2.6.26-rc kernels and that we decided should be exposed in a different way (already queued for 2.6.27 in linux-next). Roland Dreier (2): IB/umem: Avoid sign problems when demoting npages to integer IB/core: Remove IB_DEVICE_SEND_W_INV capability flag drivers/infiniband/core/umem.c | 2 +- drivers/infiniband/hw/amso1100/c2_rnic.c | 3 +-- include/rdma/ib_verbs.h | 1 - 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index fe78f7d..a1768db 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -150,7 +150,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr, ret = 0; while (npages) { ret = get_user_pages(current, current->mm, cur_base, - min_t(int, npages, + min_t(unsigned long, npages, PAGE_SIZE / sizeof (struct page *)), 1, !umem->writable, page_list, vma_list); diff --git a/drivers/infiniband/hw/amso1100/c2_rnic.c b/drivers/infiniband/hw/amso1100/c2_rnic.c index 9a054c6..b1441ae 100644 --- a/drivers/infiniband/hw/amso1100/c2_rnic.c +++ b/drivers/infiniband/hw/amso1100/c2_rnic.c @@ -455,8 +455,7 @@ int __devinit c2_rnic_init(struct c2_dev *c2dev) IB_DEVICE_CURR_QP_STATE_MOD | IB_DEVICE_SYS_IMAGE_GUID | IB_DEVICE_ZERO_STAG | - IB_DEVICE_MEM_WINDOW | - IB_DEVICE_SEND_W_INV); + IB_DEVICE_MEM_WINDOW); /* Allocate the qptr_array */ c2dev->qptr_array = vmalloc(C2_MAX_CQS * sizeof(void *)); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 911a661..31d30b1 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -105,7 +105,6 @@ enum ib_device_cap_flags { */ IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), - IB_DEVICE_SEND_W_INV = (1<<21), }; enum ib_atomic_cap { From rdreier at cisco.com Mon Jun 9 13:14:26 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Jun 2008 13:14:26 -0700 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> (Or Gerlitz's message of "Tue, 3 Jun 2008 21:59:50 +0300") References: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> Message-ID: > (A) from application point of view, setting this property per QP and not per > QP per attach to group makes enough sense, specifically, IPoIB wants this > flag to be set for all its attaches since the network stack does loopback in > memory. Yes, it's adequate for the current use but I'm wondering why we want to choose a more complex implementation that hides hardware capabilities? > (B) moving forward with the patches, we'd like to export this feature to > user space, and from the previous threads on "special QP features" (eg XRC, > Low Latency) it sounds like adding some sort of creation flags (even in the > price of new verb such that the existing ABI for qp create still works) > might be the way to go. It's just as easy to add a flag to the multicast attach verb -- maybe easier, in fact, since there are 16 reserved bits in struct ib_uverbs_attach_mcast. From rdreier at cisco.com Mon Jun 9 13:15:25 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Jun 2008 13:15:25 -0700 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: <15ddcffd0806031202l248b4fdbgcd6cd98f5c22ce65@mail.gmail.com> (Or Gerlitz's message of "Tue, 3 Jun 2008 22:02:48 +0300") References: <15ddcffd0806031202l248b4fdbgcd6cd98f5c22ce65@mail.gmail.com> Message-ID: > For IPoIB a hint might be enough, since the code to handle such packets is > there anyway and what this code does is simply to drop them. For other > applications, I think that their design might rely deeper on whether this > functionality is provided by their QP or not. MCG attach verb could return whether the "block" succeeded or not. But that might be awkward for the app to handle. Do we have any idea on anyone other than IPoIB who wants this feature? From rdreier at cisco.com Mon Jun 9 13:48:24 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Jun 2008 13:48:24 -0700 Subject: [ofa-general] enabling invalidate operations in libibverbs API Message-ID: I'm thinking about how to add support for the various invalidate operations to libibverbs (ie send with invalidate, RDMA read with inv, local inv, and receive completions with an invalidated key in them). My current feeling is that using unnamed unions is the way to go -- it preserves source and binary compatibility, and pretty much any compiler that anyone is ever going to use will support them. I feel a little bad about using non-standard C, but I don't see a better way -- there's room in the ibv_send_wr structure, but I don't see any way to squeeze another 32 bits into struct ibv_wc. ie I'm proposing the change below (of course I'm leaving out the new opcode etc definitions, this is just the part I don't feel confident about). Any suggestions of a better way or comments pro or con would be nice to get... Thanks diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index acc1b82..fcf587e 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -265,7 +265,10 @@ struct ibv_wc { enum ibv_wc_opcode opcode; uint32_t vendor_err; uint32_t byte_len; - uint32_t imm_data; /* in network byte order */ + union { + uint32_t imm_data; /* in network byte order */ + uint32_t invalidate_rkey; + }; uint32_t qp_num; uint32_t src_qp; enum ibv_wc_flags wc_flags; @@ -508,7 +511,10 @@ struct ibv_send_wr { int num_sge; enum ibv_wr_opcode opcode; enum ibv_send_flags send_flags; - uint32_t imm_data; /* in network byte order */ + union { + uint32_t imm_data; /* in network byte order */ + uint32_t invalidate_rkey; + }; union { struct { uint64_t remote_addr; From sean.hefty at intel.com Mon Jun 9 14:05:24 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 9 Jun 2008 14:05:24 -0700 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: References: Message-ID: <000801c8ca74$89497950$bb37170a@amr.corp.intel.com> I agree that unnamed unions work well in this case. I don't see any strong alternatives for ibv_wc. - Sean From sashak at voltaire.com Mon Jun 9 14:33:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 00:33:02 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Handle error when guid file not found In-Reply-To: <1213040007.14047.139.camel@hrosenstock-ws.xsigo.com> References: <1212695509.6127.161.camel@hrosenstock-ws.xsigo.com> <20080606115350.GL7795@sashak.voltaire.com> <1213022739.14047.79.camel@hrosenstock-ws.xsigo.com> <20080609181015.GD24307@sashak.voltaire.com> <1213037469.14047.121.camel@hrosenstock-ws.xsigo.com> <20080609192635.GA10301@sashak.voltaire.com> <1213040007.14047.139.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080609213302.GG10301@sashak.voltaire.com> On 12:33 Mon 09 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-09 at 22:26 +0300, Sasha Khapyorsky wrote: > > > So is there a reason this can't be changed ? > > > > Personally I prefer to get a changes back to community and GPLv2 works > > better here. > > Me too but OpenFabrics has had this dual license since day one AFAIR. I know (but not for Windows btw). And this is what I meant by "non-OFA software" - I think some other code will not be able to enter OFA due to this dual license restriction. Sasha From rdreier at cisco.com Mon Jun 9 14:52:13 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Jun 2008 14:52:13 -0700 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: (Roland Dreier's message of "Mon, 09 Jun 2008 13:48:24 -0700") References: Message-ID: Here's the complete (untested) libibverbs patch I am working on now... I think I got everything needed... diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h index 0db083a..b503a28 100644 --- a/include/infiniband/kern-abi.h +++ b/include/infiniband/kern-abi.h @@ -306,7 +306,10 @@ struct ibv_kern_wc { __u32 opcode; __u32 vendor_err; __u32 byte_len; - __u32 imm_data; + union { + __u32 imm_data; + __u32 invalidate_rkey; + }; __u32 qp_num; __u32 src_qp; __u32 wc_flags; @@ -572,7 +575,10 @@ struct ibv_kern_send_wr { __u32 num_sge; __u32 opcode; __u32 send_flags; - __u32 imm_data; + union { + __u32 imm_data; + __u32 invalidate_rkey; + }; union { struct { __u64 remote_addr; diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index a04cc62..e6e2b10 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -92,7 +92,17 @@ enum ibv_device_cap_flags { IBV_DEVICE_SYS_IMAGE_GUID = 1 << 11, IBV_DEVICE_RC_RNR_NAK_GEN = 1 << 12, IBV_DEVICE_SRQ_RESIZE = 1 << 13, - IBV_DEVICE_N_NOTIFY_CQ = 1 << 14 + IBV_DEVICE_N_NOTIFY_CQ = 1 << 14, + /* + * IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS is used by libibverbs to + * signal to low-level driver libraries that the kernel set + * the "send with invalidate" capaibility bit. Applications + * should only test IBV_DEVICE_MEM_MGT_EXTENSIONS and never + * look at IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS. + */ + IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS = 1 << 16, + IBV_DEVICE_MEM_WINDOW = 1 << 17, + IBV_DEVICE_MEM_MGT_EXTENSIONS = 1 << 21, }; enum ibv_atomic_cap { @@ -257,7 +267,8 @@ enum ibv_wc_opcode { enum ibv_wc_flags { IBV_WC_GRH = 1 << 0, - IBV_WC_WITH_IMM = 1 << 1 + IBV_WC_WITH_IMM = 1 << 1, + IBV_WC_WITH_INVALIDATE = 1 << 2, }; struct ibv_wc { @@ -266,7 +277,10 @@ struct ibv_wc { enum ibv_wc_opcode opcode; uint32_t vendor_err; uint32_t byte_len; - uint32_t imm_data; /* in network byte order */ + union { + uint32_t imm_data; /* in network byte order */ + uint32_t invalidate_rkey; + }; uint32_t qp_num; uint32_t src_qp; enum ibv_wc_flags wc_flags; @@ -486,7 +500,11 @@ enum ibv_wr_opcode { IBV_WR_SEND_WITH_IMM, IBV_WR_RDMA_READ, IBV_WR_ATOMIC_CMP_AND_SWP, - IBV_WR_ATOMIC_FETCH_AND_ADD + IBV_WR_ATOMIC_FETCH_AND_ADD, + IBV_WR_LSO, + IBV_WR_SEND_WITH_INV, + IBV_WR_RDMA_READ_WITH_INV, + IBV_WR_LOCAL_INV, }; enum ibv_send_flags { @@ -509,7 +527,10 @@ struct ibv_send_wr { int num_sge; enum ibv_wr_opcode opcode; enum ibv_send_flags send_flags; - uint32_t imm_data; /* in network byte order */ + union { + uint32_t imm_data; /* in network byte order */ + uint32_t invalidate_rkey; + }; union { struct { uint64_t remote_addr; diff --git a/src/cmd.c b/src/cmd.c index 66d7134..1945143 100644 --- a/src/cmd.c +++ b/src/cmd.c @@ -159,6 +159,18 @@ int ibv_cmd_query_device(struct ibv_context *context, device_attr->local_ca_ack_delay = resp.local_ca_ack_delay; device_attr->phys_port_cnt = resp.phys_port_cnt; + /* + * If the kernel driver says that it supports memory + * management extensions, then move the flag to + * IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS so that the low-level + * driver needs to move the flag back to show it supports the + * operations as well. + */ + if (device_attr->device_cap_flags & IBV_DEVICE_MEM_MGT_EXTENSIONS) { + device_attr->device_cap_flags &= ~IBV_DEVICE_MEM_MGT_EXTENSIONS; + device_attr->device_cap_flags |= IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS; + } + return 0; } diff --git a/src/compat-1_0.c b/src/compat-1_0.c index 459ade9..0df8b68 100644 --- a/src/compat-1_0.c +++ b/src/compat-1_0.c @@ -535,7 +535,18 @@ symver(__ibv_ack_async_event_1_0, ibv_ack_async_event, IBVERBS_1.0); int __ibv_query_device_1_0(struct ibv_context_1_0 *context, struct ibv_device_attr *device_attr) { - return ibv_query_device(context->real_context, device_attr); + int ret; + + ret = ibv_query_device(context->real_context, device_attr); + + /* + * ABI 1.0 consumers are never expecting memory management + * extension support. + */ + device_attr->device_cap_flags &= ~(IBV_DEVICE_MEM_MGT_EXTENSIONS | + IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS); + + return ret; } symver(__ibv_query_device_1_0, ibv_query_device, IBVERBS_1.0); diff --git a/src/verbs.c b/src/verbs.c index 9e370ce..4ea342f 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -79,7 +79,13 @@ enum ibv_rate mult_to_ibv_rate(int mult) int __ibv_query_device(struct ibv_context *context, struct ibv_device_attr *device_attr) { - return context->ops.query_device(context, device_attr); + int ret; + + ret = context->ops.query_device(context, device_attr); + + device_attr->device_cap_flags &= ~IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS; + + return ret; } default_symver(__ibv_query_device, ibv_query_device); From paulus at samba.org Mon Jun 9 15:28:16 2008 From: paulus at samba.org (Paul Mackerras) Date: Tue, 10 Jun 2008 08:28:16 +1000 Subject: [ofa-general] Re: [PATCH 0/2] Prevent loss of interrupts in IB/ehca In-Reply-To: <200806091742.29421.ossrosch@linux.vnet.ibm.com> References: <200806091742.29421.ossrosch@linux.vnet.ibm.com> Message-ID: <18509.44672.668946.207110@cargo.ozlabs.ibm.com> Stefan Roscher writes: > This patchset contains two changes for IB/ehca and ibmebus. > > The first patch enables ibmebus_request_irq() to optionally return the > IRQ number, which is used by the second patch to trigger EOI in case of > lost interrupts. At first sight it seems like a very bad idea for a driver to be poking into the internals of the interrupt subsystem like this. Under what circumstances do interrupts get lost, and why does doing an extra EOI like this fix the problem? Paul. From sfr at canb.auug.org.au Mon Jun 9 16:15:45 2008 From: sfr at canb.auug.org.au (Stephen Rothwell) Date: Tue, 10 Jun 2008 09:15:45 +1000 Subject: [ofa-general] Re: [PATCH 1/2] ibmebus: Change ibmebus_request_irq() to optionally return irq number In-Reply-To: <200806091744.30930.ossrosch@linux.vnet.ibm.com> References: <200806091744.30930.ossrosch@linux.vnet.ibm.com> Message-ID: <20080610091545.eb36d72f.sfr@canb.auug.org.au> Hi Stefan, On Mon, 9 Jun 2008 17:44:29 +0200 Stefan Roscher wrote: > > Signed-off-by: Stefan Roscher Before Paulus gets to you: Why are we doing this? Please write an explanatory commit message. -- Cheers, Stephen Rothwell sfr at canb.auug.org.au http://www.canb.auug.org.au/~sfr/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available URL: From aafabbri at cisco.com Mon Jun 9 18:20:25 2008 From: aafabbri at cisco.com (Aaron Fabbri (aafabbri)) Date: Mon, 9 Jun 2008 18:20:25 -0700 Subject: [ofa-general] librdmacm: rdma_bind_addr() doesn't return port Message-ID: <3B6F5C9068D5864096EB291236C3386F03E9AC28@xmb-sjc-21d.amer.cisco.com> >From the man page, and from this past discussion: http://article.gmane.org/gmane.linux.drivers.openib/45444 I was under the impression that rdma_bind_addr(), called with a zero port, would modify the caller's address with the assigned port. However, this is not the case. As Roland and Sean pointed out to me, rdma_get_src_port() can be used to get the assigned port, and this is analogous to the regular socket bind() + getsockname() API. Keeping with that convention, I recommend just changing the man page as follows: diff --git a/man/rdma_bind_addr.3 b/man/rdma_bind_addr.3 index 46c48a3..55d1b9c 100644 --- a/man/rdma_bind_addr.3 +++ b/man/rdma_bind_addr.3 @@ -22,8 +22,8 @@ to a specific port number, but it may also be called on the active side of a connection before calling rdma_resolve_addr to bind to a specific address. .P -If used to bind to port 0, the rdma_cm will select an available port -and return it to the user. +If used to bind to port 0, the rdma_cm will select an available port, which can +be retrieved with rdma_get_src_port(3). .SH "SEE ALSO" rdma_create_id(3), rdma_listen(3), rdma_resolve_addr(3), rdma_create_qp(3), rdma_get_local_addr(3), rdma_get_src_port(3) Another option would be to make rdma_bind_addr() return the effective address (but just changing the man page is easier): diff --git a/src/cma.c b/src/cma.c index fc98c8f..2ec5ba5 100644 --- a/src/cma.c +++ b/src/cma.c @@ -531,7 +531,12 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr) if (ret != size) return (ret > 0) ? -ENODATA : ret; - return ucma_query_route(id); + ret = ucma_query_route(id); + if (ret) + return ret; + + memcpy(addr, &id->route.addr.src_addr, addrlen); + return 0; From Instant at lists.openfabrics.org Mon Jun 9 19:08:08 2008 From: Instant at lists.openfabrics.org (Instant at lists.openfabrics.org) Date: 09 Jun 2008 19:08:08 -0700 Subject: [ofa-general] Can you afford to lose 300, 000 potential customers per year ? Message-ID: <20080609190807.197884EE4E0E5973@from.header.has.no.domain> Can you afford to lose 300,000 potential customers per year ? How would You like to divert 1000s of fresh, new visitors daily to Your web site or affiliate web site from Google, Yahoo, MSN and others At $0 cost to you...? ...iNSTANT BOOSTER diverts 1000s of fresh, new visitors daily to Your web site or affiliate web site from Google, Yahoo, MSN and others at $0 cost to you! ...No matter what you are selling or offering - INTSANT BOOSTER will pull in hordes of potential customers to your website - instantly! For Full Details Please read the attached .html file Unsubscribe: Please read the attached .html file and click contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Mon Jun 9 19:56:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 05:56:21 +0300 Subject: [ofa-general] [PATCH] opensm: move remote guids counting to ucast_mgr Message-ID: <20080610025621.GJ10301@sashak.voltaire.com> This is pretty mechanical change - move remote sys/node guids preallocation and counting (used when for lmc > 0 balancing) to ucast_mgr. It is needed for future "preserve base lid routing" change. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_port.h | 1 + opensm/include/opensm/osm_switch.h | 45 ++++++---------- opensm/opensm/osm_dump.c | 3 +- opensm/opensm/osm_switch.c | 101 ++++++++---------------------------- opensm/opensm/osm_ucast_mgr.c | 99 ++++++++++++++++++++++++++--------- 5 files changed, 114 insertions(+), 135 deletions(-) diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index 3d501d8..c1cbd39 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -1149,6 +1149,7 @@ typedef struct _osm_port { unsigned is_new; osm_physp_t *p_physp; cl_qlist_t mcm_list; + void *priv; } osm_port_t; /* * FIELDS diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index f5455af..0e9c5fa 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -153,31 +153,33 @@ typedef struct _osm_switch { * Switch object *********/ -/****s* OpenSM: Switch/osm_switch_guid_count_t +/****s* OpenSM: Switch/struct osm_remote_guids_count * NAME -* osm_switch_guid_count_t +* struct osm_remote_guids_count * * DESCRIPTION -* Stores system and node guids and the number of +* Stores array of pointers to remote node and the numbers of * times a switch has forwarded to it. * * SYNOPSIS */ -typedef struct _osm_switch_guid_count { - uint64_t sys_guid; - uint64_t node_guid; - unsigned int forwarded_to; -} osm_switch_guid_count_t; +struct osm_remote_guids_count { + unsigned count; + struct osm_remote_node { + osm_node_t *node; + unsigned forwarded_to; + } guids[0]; +}; /* * FIELDS -* sys_guid -* A system guid. +* count +* A number of used entries in array. * -* node_guid -* A node guid. +* node +* A pointer to node. * * forwarded_to -* A count of lids forwarded to the sys_guid/node_guid. +* A count of lids forwarded to this node. *********/ /****f* OpenSM: Switch/osm_switch_delete @@ -980,10 +982,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, IN osm_port_t * p_port, IN const uint16_t lid_ho, IN const boolean_t ignore_existing, - IN const boolean_t dor, - IN OUT osm_switch_guid_count_t * remote_guids, - IN OUT uint16_t * p_num_remote_guids, - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used); + IN const boolean_t dor); /* * PARAMETERS * p_sw @@ -1005,18 +1004,6 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, * dor * [in] If TRUE, Dimension Order Routing will be done. * -* remote_guids -* [in out] The array of remote guids already used to route -* the other lids of the same target port (if LMC > 0) -* -* p_num_remote_guids -* [in out] The number of remote guids used for routing to -* the port. -* -* p_remote_guid_count_used -* [in out] The specific osm_switch_guid_count_t used -* in switch recommendations. -* * RETURN VALUE * Returns the recommended port on which to route this LID. * diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c index 2bac75a..b96984b 100644 --- a/opensm/opensm/osm_dump.c +++ b/opensm/opensm/osm_dump.c @@ -218,8 +218,7 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) else { /* No LMC Optimization */ best_port = osm_switch_recommend_path(p_sw, p_port, - lid_ho, TRUE, dor, - NULL, NULL, NULL); + lid_ho, TRUE, dor); fprintf(file, "No %u hop path possible via port %u!", best_hops, best_port); } diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index fdcd109..58936e3 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -214,15 +214,14 @@ osm_switch_get_fwd_tbl_block(IN const osm_switch_t * const p_sw, /********************************************************************** **********************************************************************/ -static osm_switch_guid_count_t * +static struct osm_remote_node * osm_switch_find_guid_common(IN const osm_switch_t * const p_sw, - IN osm_switch_guid_count_t * remote_guids, - IN uint16_t * p_num_remote_guids, + IN struct osm_remote_guids_count *r, IN uint8_t port_num, IN int find_sys_guid, IN int find_node_guid) { - osm_switch_guid_count_t *p_remote_guid = NULL; + struct osm_remote_node *p_remote_guid = NULL; osm_physp_t *p_physp; osm_physp_t *p_rem_physp; osm_node_t *p_rem_node; @@ -240,12 +239,12 @@ osm_switch_find_guid_common(IN const osm_switch_t * const p_sw, sys_guid = p_rem_node->node_info.sys_guid; node_guid = p_rem_node->node_info.node_guid; - for (i = 0; i < *p_num_remote_guids; i++) { + for (i = 0; i < r->count; i++) { if ((!find_sys_guid - || remote_guids[i].sys_guid == sys_guid) + || r->guids[i].node->node_info.sys_guid == sys_guid) && (!find_node_guid - || remote_guids[i].node_guid == node_guid)) { - p_remote_guid = &remote_guids[i]; + || r->guids[i].node->node_info.node_guid == node_guid)) { + p_remote_guid = &r->guids[i]; break; } } @@ -253,49 +252,22 @@ osm_switch_find_guid_common(IN const osm_switch_t * const p_sw, return p_remote_guid; } -static osm_switch_guid_count_t * +static struct osm_remote_node * osm_switch_find_sys_guid_count(IN const osm_switch_t * const p_sw, - IN osm_switch_guid_count_t * remote_guids, - IN uint16_t * p_num_remote_guids, + IN struct osm_remote_guids_count *r, IN uint8_t port_num) { - return osm_switch_find_guid_common(p_sw, - remote_guids, - p_num_remote_guids, - port_num, - 1, - 0); + return osm_switch_find_guid_common(p_sw, r, port_num, 1, 0); } -static osm_switch_guid_count_t * +static struct osm_remote_node * osm_switch_find_node_guid_count(IN const osm_switch_t * const p_sw, - IN osm_switch_guid_count_t * remote_guids, - IN uint16_t * p_num_remote_guids, + IN struct osm_remote_guids_count *r, IN uint8_t port_num) { - return osm_switch_find_guid_common(p_sw, - remote_guids, - p_num_remote_guids, - port_num, - 0, - 1); + return osm_switch_find_guid_common(p_sw, r, port_num, 0, 1); } -static osm_switch_guid_count_t * -osm_switch_find_guid_count(IN const osm_switch_t * const p_sw, - IN osm_switch_guid_count_t * remote_guids, - IN uint16_t * p_num_remote_guids, - IN uint8_t port_num) -{ - return osm_switch_find_guid_common(p_sw, - remote_guids, - p_num_remote_guids, - port_num, - 1, - 1); -} - - /********************************************************************** **********************************************************************/ uint8_t @@ -303,10 +275,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, IN osm_port_t * p_port, IN const uint16_t lid_ho, IN const boolean_t ignore_existing, - IN const boolean_t dor, - IN OUT osm_switch_guid_count_t * remote_guids, - IN OUT uint16_t * p_num_remote_guids, - IN OUT osm_switch_guid_count_t ** p_remote_guid_count_used) + IN const boolean_t dor) { /* We support an enhanced LMC aware routing mode: @@ -318,8 +287,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, If this procedure is provided with the tracking array and counter we can conduct this algorithm. */ - boolean_t routing_for_lmc = remote_guids && p_num_remote_guids - && p_remote_guid_count_used; + boolean_t routing_for_lmc = (p_port->priv != NULL); uint16_t base_lid; uint8_t hops; uint8_t least_hops; @@ -346,7 +314,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, osm_physp_t *p_rem_physp; osm_node_t *p_rem_node; osm_node_t *p_rem_node_first = NULL; - osm_switch_guid_count_t *p_remote_guid = NULL; + struct osm_remote_node *p_remote_guid = NULL; CL_ASSERT(lid_ho > 0); @@ -461,8 +429,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, if (routing_for_lmc) { /* Is the sys guid already used ? */ p_remote_guid = osm_switch_find_sys_guid_count(p_sw, - remote_guids, - p_num_remote_guids, + p_port->priv, port_num); /* If not update the least hops for this case */ @@ -475,8 +442,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, } else { /* same sys found - try node */ /* Else is the node guid already used ? */ p_remote_guid = osm_switch_find_node_guid_count(p_sw, - remote_guids, - p_num_remote_guids, + p_port->priv, port_num); /* If not update the least hops for this case */ @@ -517,11 +483,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, && p_remote_guid && p_remote_guid->forwarded_to < least_forwarded_to) least_forwarded_to = p_remote_guid->forwarded_to; - } - else if (routing_for_lmc - && p_remote_guid - && check_count == least_paths - && p_remote_guid->forwarded_to < least_forwarded_to) { + } else if (routing_for_lmc + && p_remote_guid + && check_count == least_paths + && p_remote_guid->forwarded_to < least_forwarded_to) { least_forwarded_to = p_remote_guid->forwarded_to; best_port = port_num; } @@ -540,28 +505,6 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, best_port = best_port_other_sys; else if (best_port_other_node) best_port = best_port_other_node; - - /* track the remote node and system of the port used. */ - p_remote_guid = osm_switch_find_guid_count(p_sw, - remote_guids, - p_num_remote_guids, - best_port); - - if (!p_remote_guid) { - /* track the remote node and system of the port used. */ - p_physp = osm_node_get_physp_ptr(p_sw->p_node, best_port); - p_rem_physp = osm_physp_get_remote(p_physp); - p_rem_node = osm_physp_get_node_ptr(p_rem_physp); - memcpy(&(remote_guids[*p_num_remote_guids].sys_guid), - &(p_rem_node->node_info.sys_guid), - sizeof(uint64_t)); - memcpy(&(remote_guids[*p_num_remote_guids].node_guid), - &(p_rem_node->node_info.node_guid), - sizeof(uint64_t)); - remote_guids[*p_num_remote_guids].forwarded_to = 0; - (*p_num_remote_guids)++; - } - *p_remote_guid_count_used = p_remote_guid; } return (best_port); diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index 5923638..c073037 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -187,6 +187,24 @@ __osm_ucast_mgr_process_neighbor(IN osm_ucast_mgr_t * const p_mgr, /********************************************************************** **********************************************************************/ +static struct osm_remote_node * +find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, + struct osm_remote_guids_count *r) +{ + unsigned i; + osm_physp_t *p = osm_node_get_physp_ptr(sw->p_node, port); + osm_node_t *node = p->p_remote_physp->p_node; + + for (i = 0; i < r->count; i++) + if (r->guids[i].node == node) + return &r->guids[i]; + + r->guids[i].node = node; + r->guids[i].forwarded_to = 0; + r->count++; + return &r->guids[i]; +} + static void __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, IN osm_switch_t * const p_sw, @@ -204,23 +222,10 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, in providing better routing in LMC > 0 situations */ uint16_t lids_per_port = 1 << p_mgr->p_subn->opt.lmc; - osm_switch_guid_count_t *remote_guids = NULL; - uint16_t num_used_guids = 0; - osm_switch_guid_count_t *p_remote_guid_used = NULL; + struct osm_remote_node *p_remote_guid_used = NULL; OSM_LOG_ENTER(p_mgr->p_log); - if (lids_per_port > 1) { - remote_guids = malloc(sizeof(osm_switch_guid_count_t) * lids_per_port); - if (remote_guids == NULL) { - osm_log(p_mgr->p_log, OSM_LOG_ERROR, - "__osm_ucast_mgr_process_port: ERR 3A09: " - "Cannot allocate array. Insufficient memory\n"); - goto Exit; - } - memset(remote_guids, 0, sizeof(osm_switch_guid_count_t) * lids_per_port); - } - osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); /* If the lids are zero - then there was some problem with the initialization. @@ -258,21 +263,19 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, for (lid_ho = min_lid_ho; lid_ho <= max_lid_ho; lid_ho++) { /* Use the enhanced algorithm only for LMC > 0 */ if (lids_per_port > 1) { - p_remote_guid_used = NULL; port = osm_switch_recommend_path(p_sw, p_port, lid_ho, p_mgr->p_subn-> ignore_existing_lfts, - p_mgr->is_dor, - remote_guids, - &num_used_guids, - &p_remote_guid_used); - } - else + p_mgr->is_dor); + if (port > 0 && port != OSM_NO_PATH && p_port->priv) + p_remote_guid_used = + find_and_add_remote_sys(p_sw, port, + p_port->priv); + } else port = osm_switch_recommend_path(p_sw, p_port, lid_ho, p_mgr->p_subn-> ignore_existing_lfts, - p_mgr->is_dor, - NULL, NULL, NULL); + p_mgr->is_dor); /* There might be no path to the target @@ -334,8 +337,6 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, } Exit: - if (remote_guids) - free(remote_guids); OSM_LOG_EXIT(p_mgr->p_log); } @@ -460,6 +461,48 @@ osm_ucast_mgr_set_fwd_table(IN osm_ucast_mgr_t * const p_mgr, /********************************************************************** **********************************************************************/ +static void alloc_ports_priv(osm_ucast_mgr_t *mgr) +{ + cl_qmap_t *port_tbl = &mgr->p_subn->port_guid_tbl; + struct osm_remote_guids_count *r; + osm_port_t *port; + cl_map_item_t *item; + unsigned lmc; + + for (item = cl_qmap_head(port_tbl); item != cl_qmap_end(port_tbl); + item = cl_qmap_next(item)) { + port = (osm_port_t *)item; + lmc = ib_port_info_get_lmc(&port->p_physp->port_info); + if (!lmc) + continue; + r = malloc(sizeof(*r) + sizeof(r->guids[0]) * (1 << lmc)); + if (!r) { + OSM_LOG(mgr->p_log, OSM_LOG_ERROR, "ERR 3A09: " + "cannot allocate memory to track remote" + " systems for lmc > 0\n"); + port->priv = NULL; + continue; + } + memset(r, 0, sizeof(*r) + sizeof(r->guids[0]) * (1 << lmc)); + port->priv = r; + } +} + +static void free_ports_priv(osm_ucast_mgr_t *mgr) +{ + cl_qmap_t *port_tbl = &mgr->p_subn->port_guid_tbl; + osm_port_t *port; + cl_map_item_t *item; + for (item = cl_qmap_head(port_tbl); item != cl_qmap_end(port_tbl); + item = cl_qmap_next(item)) { + port = (osm_port_t *)item; + if (port->priv) { + free(port->priv); + port->priv = NULL; + } + } +} + static void __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, IN void *context) @@ -488,6 +531,9 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, p_port_tbl = &p_mgr->p_subn->port_guid_tbl; + if (p_mgr->p_subn->opt.lmc) + alloc_ports_priv(p_mgr); + /* Iterate through every port setting LID routes for each port based on base LID and LMC value. @@ -501,6 +547,9 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, osm_ucast_mgr_set_fwd_table(p_mgr, p_sw); + if (p_mgr->p_subn->opt.lmc) + free_ports_priv(p_mgr); + OSM_LOG_EXIT(p_mgr->p_log); } -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Mon Jun 9 19:59:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 05:59:39 +0300 Subject: [ofa-general] [PATCH] opensm: preserve base lid routes In-Reply-To: <20080610025621.GJ10301@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> Message-ID: <20080610025939.GK10301@sashak.voltaire.com> Basically this addresses the problem described by Al Chu in: http://lists.openfabrics.org/pipermail/general/2008-April/049132.html When base lid paths become completely disbalanced on a fabrics with lmc > 0. One feedback was from Yiftah Shahar: "I think that our requirements should be that even when you are working with LMC>0 then the base LID routing should not be affected. One way to achieve this goal is to first run the base-LID routing (so all base LID improvement will be also in LMC>0) and then start with the other LIDs as round-robbing starting from the base-lid-port + 1 according current routing algorithm rules (keeping min-hop, up/down...)." We had some discussion with Al and Yiftah about this and considered that in addition to "pure" base lid paths preservation (which is good thing by itself) proposed method solves original lid disbalancing problem as well. This patch is implementation of the idea above. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_switch.h | 4 + opensm/opensm/osm_dump.c | 3 +- opensm/opensm/osm_switch.c | 8 ++- opensm/opensm/osm_ucast_mgr.c | 163 ++++++++++++++++-------------------- 4 files changed, 86 insertions(+), 92 deletions(-) diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index 0e9c5fa..c1521a6 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -981,6 +981,7 @@ uint8_t osm_switch_recommend_path(IN const osm_switch_t * const p_sw, IN osm_port_t * p_port, IN const uint16_t lid_ho, + IN unsigned start_from, IN const boolean_t ignore_existing, IN const boolean_t dor); /* @@ -995,6 +996,9 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, * lid_ho * [in] LID value (host order) for which to get a path advisory. * +* start_from +* [in] Port number from where to start balance counting. +* * ignore_existing * [in] Set to cause the switch to choose the optimal route * regardless of existing paths. diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c index b96984b..60c6d25 100644 --- a/opensm/opensm/osm_dump.c +++ b/opensm/opensm/osm_dump.c @@ -218,7 +218,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) else { /* No LMC Optimization */ best_port = osm_switch_recommend_path(p_sw, p_port, - lid_ho, TRUE, dor); + lid_ho, 1, TRUE, + dor); fprintf(file, "No %u hop path possible via port %u!", best_hops, best_port); } diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index 58936e3..a9d13c8 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -274,6 +274,7 @@ uint8_t osm_switch_recommend_path(IN const osm_switch_t * const p_sw, IN osm_port_t * p_port, IN const uint16_t lid_ho, + IN unsigned start_from, IN const boolean_t ignore_existing, IN const boolean_t dor) { @@ -294,6 +295,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, uint8_t port_num; uint8_t num_ports; uint32_t least_paths = 0xFFFFFFFF; + unsigned i; /* The follwing will track the least paths if the route should go through a new system/node @@ -397,8 +399,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, */ /* port number starts with one and num_ports is 1 + num phys ports */ - for (port_num = 1; port_num < num_ports; port_num++) { - if (osm_switch_get_hop_count(p_sw, base_lid, port_num) != + for (i = start_from; i < start_from + num_ports; i++) { + port_num = i%num_ports; + if (!port_num || + osm_switch_get_hop_count(p_sw, base_lid, port_num) != least_hops) continue; diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index c073037..2aae6d5 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -208,7 +208,8 @@ find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, static void __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, IN osm_switch_t * const p_sw, - IN osm_port_t * const p_port) + IN osm_port_t * const p_port, + IN unsigned lid_offset) { uint16_t min_lid_ho; uint16_t max_lid_ho; @@ -217,19 +218,14 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, boolean_t is_ignored_by_port_prof; ib_net64_t node_guid; struct osm_routing_engine *p_routing_eng; - /* - The following are temporary structures that will aid - in providing better routing in LMC > 0 situations - */ - uint16_t lids_per_port = 1 << p_mgr->p_subn->opt.lmc; - struct osm_remote_node *p_remote_guid_used = NULL; + unsigned start_from = 1; OSM_LOG_ENTER(p_mgr->p_log); osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); - /* If the lids are zero - then there was some problem with the initialization. - Don't handle this port. */ + /* If the lids are zero - then there was some problem with + * the initialization. Don't handle this port. */ if (min_lid_ho == 0 || max_lid_ho == 0) { OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A04: " "Port 0x%" PRIx64 " has LID 0. An initialization " @@ -238,16 +234,22 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, goto Exit; } - if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { + lid_ho = min_lid_ho + lid_offset; + + if (lid_ho > max_lid_ho) + goto Exit; + + if (lid_offset) + /* ignore potential overflow - it is handled in osm_switch.c */ + start_from = osm_switch_get_port_by_lid(p_sw, lid_ho - 1) + 1; + + if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Processing port 0x%" PRIx64 ", LIDs [0x%X,0x%X]\n", - cl_ntoh64(osm_port_get_guid(p_port)), + "Processing port 0x%" PRIx64 ", LID %u [0x%X,0x%X]\n", + cl_ntoh64(osm_port_get_guid(p_port)), lid_ho, min_lid_ho, max_lid_ho); - } - /* - TO DO - This should be runtime error, not a CL_ASSERT() - */ + /* TODO - This should be runtime error, not a CL_ASSERT() */ CL_ASSERT(max_lid_ho < osm_switch_get_fwd_tbl_size(p_sw)); node_guid = osm_node_get_node_guid(p_sw->p_node); @@ -260,80 +262,62 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, how best to distribute the LID range across the ports that can reach those LIDs. */ - for (lid_ho = min_lid_ho; lid_ho <= max_lid_ho; lid_ho++) { - /* Use the enhanced algorithm only for LMC > 0 */ - if (lids_per_port > 1) { - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, - p_mgr->p_subn-> - ignore_existing_lfts, - p_mgr->is_dor); - if (port > 0 && port != OSM_NO_PATH && p_port->priv) - p_remote_guid_used = - find_and_add_remote_sys(p_sw, port, - p_port->priv); + port = osm_switch_recommend_path(p_sw, p_port, lid_ho, start_from, + p_mgr->p_subn->ignore_existing_lfts, + p_mgr->is_dor); + + if (port == OSM_NO_PATH) { + /* do not try to overwrite the ppro of non existing port ... */ + is_ignored_by_port_prof = TRUE; + + /* Up/Down routing can cause unreachable routes between some + switches so we do not report that as an error in that case */ + if (!p_routing_eng->build_lid_matrices) { + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " + "No path to get to LID 0x%X from switch 0x%" + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); + /* trigger a new sweep - try again ... */ + p_mgr->p_subn->subnet_initialization_error = TRUE; } else - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, - p_mgr->p_subn-> - ignore_existing_lfts, - p_mgr->is_dor); + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, + "No path to get to LID 0x%X from switch 0x%" + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); + } else { + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, + "Routing LID 0x%X to port 0x%X" + " for switch 0x%" PRIx64 "\n", + lid_ho, port, cl_ntoh64(node_guid)); /* - There might be no path to the target + we would like to optionally ignore this port in equalization + as in the case of the Mellanox Anafa Internal PCI TCA port */ - if (port == OSM_NO_PATH) { - /* do not try to overwrite the ppro of non existing port ... */ - is_ignored_by_port_prof = TRUE; - - /* Up/Down routing can cause unreachable routes between some - switches so we do not report that as an error in that case */ - if (!p_routing_eng->build_lid_matrices) { - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " - "No path to get to LID 0x%X from switch 0x%" - PRIx64 "\n", lid_ho, - cl_ntoh64(node_guid)); - /* trigger a new sweep - try again ... */ - p_mgr->p_subn->subnet_initialization_error = - TRUE; - } else - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "No path to get to LID 0x%X from switch 0x%" - PRIx64 "\n", lid_ho, - cl_ntoh64(node_guid)); - } else { - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Routing LID 0x%X to port 0x%X" - " for switch 0x%" PRIx64 "\n", - lid_ho, port, cl_ntoh64(node_guid)); - - /* - we would like to optionally ignore this port in equalization - as in the case of the Mellanox Anafa Internal PCI TCA port - */ - is_ignored_by_port_prof = - osm_port_prof_is_ignored_port(p_mgr->p_subn, - node_guid, port); - - /* - We also would ignore this route if the target lid is of a switch - and the port_profile_switch_node is not TRUE - */ - if (!p_mgr->p_subn->opt.port_profile_switch_nodes) { - is_ignored_by_port_prof |= - (osm_node_get_type(p_port->p_node) == - IB_NODE_TYPE_SWITCH); - } - } + is_ignored_by_port_prof = + osm_port_prof_is_ignored_port(p_mgr->p_subn, + node_guid, port); /* - We have selected the port for this LID. - Write it to the forwarding tables. + We also would ignore this route if the target lid is of + a switch and the port_profile_switch_node is not TRUE */ - p_mgr->lft_buf[lid_ho] = port; - if (!is_ignored_by_port_prof) { - osm_switch_count_path(p_sw, port); - if (p_remote_guid_used) - p_remote_guid_used->forwarded_to++; - } + if (!p_mgr->p_subn->opt.port_profile_switch_nodes) + is_ignored_by_port_prof |= + (osm_node_get_type(p_port->p_node) == + IB_NODE_TYPE_SWITCH); + } + + /* + We have selected the port for this LID. + Write it to the forwarding tables. + */ + p_mgr->lft_buf[lid_ho] = port; + if (!is_ignored_by_port_prof) { + struct osm_remote_node *rem_node_used; + osm_switch_count_path(p_sw, port); + if (port > 0 && p_port->priv && + (rem_node_used = find_and_add_remote_sys(p_sw, port, + p_port->priv))) + rem_node_used->forwarded_to++; } Exit: @@ -512,6 +496,7 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, osm_node_t *p_node; osm_port_t *p_port; const cl_qmap_t *p_port_tbl; + unsigned i, lids_per_port; OSM_LOG_ENTER(p_mgr->p_log); @@ -538,12 +523,12 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, Iterate through every port setting LID routes for each port based on base LID and LMC value. */ - - for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); - p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); - p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) { - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port); - } + lids_per_port = 1 << p_mgr->p_subn->opt.lmc; + for (i = 0; i < lids_per_port; i++) + for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); + p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); + p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) + __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port, i); osm_ucast_mgr_set_fwd_table(p_mgr, p_sw); -- 1.5.5.1.178.g1f811 From yiftahs at voltaire.com Mon Jun 9 21:45:09 2008 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Tue, 10 Jun 2008 07:45:09 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c:Handle error when guid file not found In-Reply-To: <20080609213302.GG10301@sashak.voltaire.com> Message-ID: <39C75744D164D948A170E9792AF8E7CA013E68DD@exil.voltaire.com> Hi Sasha, I think we need to keep the dual license in any section of the Linux code we can (officially in the OFED or not) so it can be use by other projects without limitations (like WinOF). I do not know if GPLv2 impose any issues with WinOF stack but why to change something that is working. Yiftah > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general- > bounces at lists.openfabrics.org] On Behalf Of Sasha Khapyorsky > Sent: Tuesday, June 10, 2008 00:33 > To: Hal Rosenstock > Cc: general at lists.openfabrics.org > Subject: Re: [ofa-general] Re: [PATCH] infiniband- > diags/mcm_rereg_test.c:Handle error when guid file not found > > On 12:33 Mon 09 Jun , Hal Rosenstock wrote: > > On Mon, 2008-06-09 at 22:26 +0300, Sasha Khapyorsky wrote: > > > > So is there a reason this can't be changed ? > > > > > > Personally I prefer to get a changes back to community and GPLv2 works > > > better here. > > > > Me too but OpenFabrics has had this dual license since day one AFAIR. > > I know (but not for Windows btw). And this is what I meant by "non-OFA > software" - I think some other code will not be able to enter OFA due to > this dual license restriction. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general From sean.hefty at intel.com Mon Jun 9 23:01:02 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 9 Jun 2008 23:01:02 -0700 Subject: [ofa-general] Re: [PATCH]infiniband-diags/mcm_rereg_test.c:Handle error when guid filenot found In-Reply-To: <39C75744D164D948A170E9792AF8E7CA013E68DD@exil.voltaire.com> References: <20080609213302.GG10301@sashak.voltaire.com> <39C75744D164D948A170E9792AF8E7CA013E68DD@exil.voltaire.com> Message-ID: <000001c8cabf$5d18f4a0$d7c8180a@amr.corp.intel.com> >I think we need to keep the dual license in any section of the Linux >code we can (officially in the OFED or not) so it can be use by other >projects without limitations (like WinOF). The dual license is a requirement of any OFA Linux software. Windows requires a modified version of BSD. Submissions to OFA cannot be accepted unless they meet the license requirements. (I'm not trying to judge this, simply state what's required in OFA bylaws.) - Sean From rfqicuv at boukis.com Mon Jun 9 23:46:53 2008 From: rfqicuv at boukis.com (Jane Owen) Date: Tue, 10 Jun 2008 08:46:53 +0200 Subject: [ofa-general] Update your Penis Message-ID: <01c8cad6$87c97c80$25e60059@rfqicuv> Viagra Super Active - Our price $2.82 - a new viagra for XXL dick Big Penis Pills - Our price $95.66 -new pills to make your dick XXLWe have EVERYTHING that you need. Visit us now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ossrosch at linux.vnet.ibm.com Tue Jun 10 01:21:26 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Tue, 10 Jun 2008 10:21:26 +0200 Subject: [ofa-general] Re: [PATCH 0/2] Prevent loss of interrupts in IB/ehca In-Reply-To: <18509.44672.668946.207110@cargo.ozlabs.ibm.com> References: <200806091742.29421.ossrosch@linux.vnet.ibm.com> <18509.44672.668946.207110@cargo.ozlabs.ibm.com> Message-ID: <200806101021.28795.ossrosch@linux.vnet.ibm.com> On Tuesday 10 June 2008 00:28:16 Paul Mackerras wrote: > Stefan Roscher writes: > > > This patchset contains two changes for IB/ehca and ibmebus. > > > > The first patch enables ibmebus_request_irq() to optionally return the > > IRQ number, which is used by the second patch to trigger EOI in case of > > lost interrupts. > > At first sight it seems like a very bad idea for a driver to be poking > into the internals of the interrupt subsystem like this. Under what > circumstances do interrupts get lost, and why does doing an extra EOI > like this fix the problem? > > Paul. > The processing of events with a timer controlled polling is not the "typical" way how you should handle adapter events. During corner case testing, we noticed that some versions of ehca do not properly transition to interrupt done in special load situations. This can be resolved by periodically triggering EOI through H_EOI, if eqes are pending. Hope this clarifys the backround of the patch. Is there a better way to initiate this type of EOI in a non-irq case? regards Stefan R. and Christoph R. From paulus at samba.org Tue Jun 10 02:21:38 2008 From: paulus at samba.org (Paul Mackerras) Date: Tue, 10 Jun 2008 19:21:38 +1000 Subject: [ofa-general] Re: [PATCH 0/2] Prevent loss of interrupts in IB/ehca In-Reply-To: <200806101021.28795.ossrosch@linux.vnet.ibm.com> References: <200806091742.29421.ossrosch@linux.vnet.ibm.com> <18509.44672.668946.207110@cargo.ozlabs.ibm.com> <200806101021.28795.ossrosch@linux.vnet.ibm.com> Message-ID: <18510.18338.20232.969001@cargo.ozlabs.ibm.com> Stefan Roscher writes: > The processing of events with a timer controlled polling is not the "typical" > way how you should handle adapter events. Do you mean it's not typical in Linux (I would have said it was), or it's not the way that the firmware architects and implementers thought the eHEA hcalls would be used? > During corner case testing, we noticed that some versions of ehca > do not properly transition to interrupt done in special load situations. > This can be resolved by periodically triggering EOI through H_EOI, > if eqes are pending. So in other words, doing the extra EOIs is a workaround for a firmware and/or hardware bug, then? If that's the case then you needed to say that in the commit message. Given that the ehea driver does hcalls itself directly, and this is a workaround for a firmware/hardware bug, it may actually be cleaner just to have the ehea driver do the necessary H_EOI calls directly. Paul. From sashak at voltaire.com Tue Jun 10 05:43:13 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 15:43:13 +0300 Subject: [ofa-general] Re: OSM_DEFAULT_SM_KEY byte order In-Reply-To: <1212503425.31387.54.camel@hrosenstock-ws.xsigo.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> <20080602182922.GC13775@sashak.voltaire.com> <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> <20080602185151.GF13775@sashak.voltaire.com> <1212503425.31387.54.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080610124313.GO10301@sashak.voltaire.com> Hi Hal, On 07:30 Tue 03 Jun , Hal Rosenstock wrote: > > Question: > What is the expected effect on the wire of these changes (separate cases > of little and big endian machines if different) ? On both little and big endian machines default SM Key value on the wire is '1' in network byte order. It changes the behavior of OpenSM on a little endian machine, where by mistake actual value was '0x0100000000000000' and leaves it as is on a big endian machines. > Request: > If your proposal changes default wire behavior, please make sure this is > documented (in more than just the git log) so it stands less of a chance > of being missed by users who might not follow all these discussions very > closely. Works good for me. Sasha From sashak at voltaire.com Tue Jun 10 06:06:47 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 16:06:47 +0300 Subject: [ofa-general] [PATCH] opensm: fix OSM_DEFAULT_SM_KEY byte order In-Reply-To: <20080610124313.GO10301@sashak.voltaire.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> <20080602182922.GC13775@sashak.voltaire.com> <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> <20080602185151.GF13775@sashak.voltaire.com> <1212503425.31387.54.camel@hrosenstock-ws.xsigo.com> <20080610124313.GO10301@sashak.voltaire.com> Message-ID: <20080610130647.GP10301@sashak.voltaire.com> Keep OSM_DEFAULT_SM_KEY value in network byte order. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_base.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/opensm/include/opensm/osm_base.h b/opensm/include/opensm/osm_base.h index d07ad8c..3793804 100644 --- a/opensm/include/opensm/osm_base.h +++ b/opensm/include/opensm/osm_base.h @@ -112,7 +112,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -#define OSM_DEFAULT_SM_KEY 1 +#define OSM_DEFAULT_SM_KEY CL_HTON64(1) /********/ /****s* OpenSM: Base/OSM_DEFAULT_SA_KEY * NAME @@ -123,7 +123,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -#define OSM_DEFAULT_SA_KEY 1 +#define OSM_DEFAULT_SA_KEY OSM_DEFAULT_SM_KEY /********/ /****s* OpenSM: Base/OSM_DEFAULT_LMC * NAME -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Tue Jun 10 06:07:27 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 16:07:27 +0300 Subject: [ofa-general] [PATCH] opensm: state default SM_Key value change in usage and man In-Reply-To: <20080610130647.GP10301@sashak.voltaire.com> References: <20080522140916.GC32128@sashak.voltaire.com> <1211467961.18236.178.camel@hrosenstock-ws.xsigo.com> <20080531214919.GS22418@sashak.voltaire.com> <1212415598.32691.38.camel@hrosenstock-ws.xsigo.com> <20080602182922.GC13775@sashak.voltaire.com> <1212432068.32691.70.camel@hrosenstock-ws.xsigo.com> <20080602185151.GF13775@sashak.voltaire.com> <1212503425.31387.54.camel@hrosenstock-ws.xsigo.com> <20080610124313.GO10301@sashak.voltaire.com> <20080610130647.GP10301@sashak.voltaire.com> Message-ID: <20080610130727.GQ10301@sashak.voltaire.com> State default SM_Key value change in usage massage, OpenSM man page and generated config template file. Signed-off-by: Sasha Khapyorsky --- opensm/man/opensm.8.in | 3 +++ opensm/opensm/main.c | 8 +++++++- opensm/opensm/osm_subnet.c | 5 +++++ 3 files changed, 15 insertions(+), 1 deletions(-) diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index bb58f8a..b6a9fc7 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -99,6 +99,9 @@ is chosen by priority and GUID. Range goes from 0 \fB\-smkey\fR This option specifies the SM\'s SM_Key (64 bits). This will effect SM authentication. +Note that OpenSM version 3.2.1 and below used the default value '1' +in a host byte order, it is fixed now but you may need this option to +interoperate with old OpenSM running on a little endian machine. .TP \fB\-r\fR, \fB\-\-reassign_lids\fR This option causes OpenSM to reassign LIDs to all diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 040a0d7..48e9e47 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -163,7 +163,13 @@ static void show_usage(void) " from 0 (lowest priority) to 15 (highest).\n\n"); printf("-smkey \n" " This option specifies the SM's SM_Key (64 bits).\n" - " This will effect SM authentication.\n\n"); + " This will effect SM authentication.\n" + " Note that OpenSM version 3.2.1 and below used the\n" + " default value '1' in a host byte order, it is fixed\n" + " now but you may need this option to interoperate\n" + " with old OpenSM running on a little endian machine.\n" + "\n"); + printf("-r\n" "--reassign_lids\n" " This option causes OpenSM to reassign LIDs to all\n" diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index e465565..dd71fa3 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -1403,6 +1403,11 @@ int osm_subn_write_conf_file(char *file_name, IN osm_subn_opt_t *const p_opts) "sm_key 0x%016" PRIx64 "\n\n" "# SM_Key value to qualify rcv SA queries as 'trusted'\n" "sa_key 0x%016" PRIx64 "\n\n" + "# Note that for both values above (sm_key and sa_key)\n" + "# OpenSM version 3.2.1 and below used the default value '1'\n" + "# in a host byte order, it is fixed now but you may need to\n" + "# change the values to interoperate with old OpenSM running\n" + "# on a little endian machine.\n\n" "# Subnet prefix used on this subnet\n" "subnet_prefix 0x%016" PRIx64 "\n\n" "# The LMC value used on this subnet\n" -- 1.5.5.1.178.g1f811 From bart.vanassche at gmail.com Tue Jun 10 06:32:04 2008 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Tue, 10 Jun 2008 15:32:04 +0200 Subject: [ofa-general] Trouble with IPoIB stack ? Message-ID: Hello, While running a stress test of Linux' iSCSI initiator I noticed that the test ran fine over an Ethernet network but that strange memory allocation problems were reported after I switched from Ethernet to IPoIB. A few minutes later the whole system froze. Anyone any hints on how I can pinpoint the cause of this issue ? >From the kernel messages: ... [ 552.238283] scsi_wq_25: page allocation failure. order:0, mode:0x20 [ 551.878249] scsi_wq_39: page allocation failure. order:0, mode:0x20 ... [ 553.369853] Call Trace: [ 551.878264] Pid: 7860, comm: scsi_wq_39 Not tainted 2.6.25.6 #3 [ 552.958273] Pid: 7761, comm: scsi_wq_36 Not tainted 2.6.25.6 #3 [ 553.369866] [__alloc_pages+0x2e8/0x3d0] __alloc_pages+0x2e8/0x3d0 [ 552.418332] [__alloc_pages+0x2e8/0x3d0] __alloc_pages+0x2e8/0x3d0 [ 553.369875] [ib_ipoib:dev_queue_xmit+0xed/0xb410] dev_queue_xmit+0xed/0x380 [ 552.418343] [new_slab+0x22d/0x2b0] new_slab+0x22d/0x2b0 [ 552.418349] [__slab_alloc+0x1f5/0x4c0] __slab_alloc+0x1f5/0x4c0 [ 553.369894] [new_slab+0x22d/0x2b0] new_slab+0x22d/0x2b0 [ 552.778357] Pid: 7306, comm: scsi_wq_23 Not tainted 2.6.25.6 #3 ... See also http://bugzilla.kernel.org/show_bug.cgi?id=10890. Bart. From tziporet at dev.mellanox.co.il Tue Jun 10 07:12:44 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 10 Jun 2008 17:12:44 +0300 Subject: [ofa-general][PATCH 4/12 v2] mlx4: Pre reserved Qp regions In-Reply-To: References: <480F4E98.7010803@mellanox.co.il> Message-ID: <484E8BDC.6090801@mellanox.co.il> Roland Dreier wrote: > Jack is pushing to make these types of module parameters specified using > log2 of the value instead of the absolute value -- seems you guys should > agree on the interface you want to use. > > The parameters that we wanted to be log2 values are those that must be a power of 2. These new parameters do not have this limitation, and thus no need for log2 requirement. Tziporet From rdreier at cisco.com Tue Jun 10 07:17:34 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 07:17:34 -0700 Subject: [ofa-general][PATCH 4/12 v2] mlx4: Pre reserved Qp regions In-Reply-To: <484E8BDC.6090801@mellanox.co.il> (Tziporet Koren's message of "Tue, 10 Jun 2008 17:12:44 +0300") References: <480F4E98.7010803@mellanox.co.il> <484E8BDC.6090801@mellanox.co.il> Message-ID: > The parameters that we wanted to be log2 values are those that must be > a power of 2. > These new parameters do not have this limitation, and thus no need for > log2 requirement. How the heck is the end-user supposed to know or care which parameters are which? Plus: > +module_param_named(num_mac, num_mac, int, 0444); > +MODULE_PARM_DESC(num_mac, "Maximum number of MACs per ETH port " > + "(1-127, default 1)"); > + dev->caps.log_num_macs = ilog2(roundup_pow_of_two(num_mac + 1)); and so on... so it sure looks like a power of two to me. From hrosenstock at xsigo.com Tue Jun 10 07:25:59 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 07:25:59 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/iba/ib_types.h: Fix comment Message-ID: <1213107959.14047.229.camel@hrosenstock-ws.xsigo.com> opensm/include/iba/ib_types.h: Fix comment Signed-off-by: Hal Rosenstock diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index e6bd9ee..09ec257 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -4272,7 +4272,7 @@ ib_sa_mad_get_payload_ptr(IN const ib_sa_mad_t * const p_sa_mad) /* * PARAMETERS -* p_smp +* p_sa_mad * [in] Pointer to the SA MAD packet. * * RETURN VALUES From ogerlitz at voltaire.com Tue Jun 10 07:36:49 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 10 Jun 2008 17:36:49 +0300 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: References: <15ddcffd0806031202l248b4fdbgcd6cd98f5c22ce65@mail.gmail.com> Message-ID: <484E9181.4000107@voltaire.com> Roland Dreier wrote: > MCG attach verb could return whether the "block" succeeded or not. But > that might be awkward for the app to handle. Roland, This way, each call to the MCG attach verb would be duplicated in the code, since if it fails when the "block loopback" param was set it would now be called again unset, etc. I think this turns to be more complex then doing a query for a cap bit and then setting the block mcast through a QP create flag once. To my taste, the fact that the QP create flag is used by the driver at the time of MCG attach does not make it a more complex implementation. > Do we have any idea on anyone other than IPoIB who wants this feature? Yes, we want to use it in a user space multicast app. Or. From ogerlitz at voltaire.com Tue Jun 10 07:40:36 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 10 Jun 2008 17:40:36 +0300 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: References: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> Message-ID: <484E9264.4070708@voltaire.com> Roland Dreier wrote: > Yes, it's adequate for the current use but I'm wondering why we want to > choose a more complex implementation that hides hardware capabilities? I tend to think the suggested way is somehow simpler (vs. the per attach) see my reply on "patch 1/3" > It's just as easy to add a flag to the multicast attach verb -- maybe > easier, in fact, since there are 16 reserved bits in struct > ib_uverbs_attach_mcast. I am concerned about both libibverbs --- uverbs ABI and the app -- libibverbs ABI I understand that with this suggestion of yours, the libibverbs API/ABI of MCG attach to apps would change, isn't it something we want to avoid, eg through introducing a new verbs create_qp_ext that gets also creation flag? Or. From ossrosch at linux.vnet.ibm.com Tue Jun 10 07:44:31 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Tue, 10 Jun 2008 16:44:31 +0200 Subject: [ofa-general] [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts Message-ID: <200806101644.34693.ossrosch@linux.vnet.ibm.com> During corner case testing, we noticed that some versions of ehca do not properly transition to interrupt done in special load situations. This can be resolved by periodically triggering EOI through H_EOI, if eqes are pending. Signed-off-by: Stefan Roscher --- This patch replaces my previous patch-set. As Paul suggested, this version of the patch calls H_EOI directly and doesn't need any ibmebus changes. drivers/infiniband/hw/ehca/ehca_main.c | 11 +++++++++-- drivers/infiniband/hw/ehca/hcp_if.c | 11 +++++++++++ drivers/infiniband/hw/ehca/hcp_if.h | 1 + 3 files changed, 21 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_main.c b/drivers/infiniband/hw/ehca/ehca_main.c index 482103e..add4ff4 100644 --- a/drivers/infiniband/hw/ehca/ehca_main.c +++ b/drivers/infiniband/hw/ehca/ehca_main.c @@ -937,6 +937,7 @@ static struct of_platform_driver ehca_driver = { void ehca_poll_eqs(unsigned long data) { struct ehca_shca *shca; + u64 ret; spin_lock(&shca_list_lock); list_for_each_entry(shca, &shca_list, shca_list) { @@ -955,8 +956,14 @@ void ehca_poll_eqs(unsigned long data) spin_unlock_irqrestore(&eq->spinlock, flags); max--; } while (q_ofs == q_ofs2 && max > 0); - if (q_ofs == q_ofs2) - ehca_process_eq(shca, 0); + if (q_ofs == q_ofs2) { + ret = hipz_h_eoi(eq->ist); + if (ret != H_SUCCESS) + ehca_err(&shca->ib_device, + "bad return code EOI -" + "rc = %ld\n", ret); + tasklet_hi_schedule(&shca->eq.interrupt_task); + } } } mod_timer(&poll_eqs_timer, round_jiffies(jiffies + HZ)); diff --git a/drivers/infiniband/hw/ehca/hcp_if.c b/drivers/infiniband/hw/ehca/hcp_if.c index 5245e13..7084efd 100644 --- a/drivers/infiniband/hw/ehca/hcp_if.c +++ b/drivers/infiniband/hw/ehca/hcp_if.c @@ -933,3 +933,14 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, r_cb, 0, 0, 0, 0); } + +u64 hipz_h_eoi(int irq) +{ + int value; + unsigned long xirr; + + iosync(); + value = (0xff << 24) | irq; + xirr = value & 0xffffffff; + return plpar_hcall_norets(H_EOI, xirr); +} diff --git a/drivers/infiniband/hw/ehca/hcp_if.h b/drivers/infiniband/hw/ehca/hcp_if.h index 60ce02b..2c3c6e0 100644 --- a/drivers/infiniband/hw/ehca/hcp_if.h +++ b/drivers/infiniband/hw/ehca/hcp_if.h @@ -260,5 +260,6 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, const u64 ressource_handle, void *rblock, unsigned long *byte_count); +u64 hipz_h_eoi(int irq); #endif /* __HCP_IF_H__ */ -- 1.5.5 From hrosenstock at xsigo.com Tue Jun 10 07:46:26 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 07:46:26 -0700 Subject: [ofa-general] infiniband-diags/saquery.c:print_multicast_member_record question Message-ID: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> Ira, In saquery.c:print_multicast_member_record, there is the following: /* go through and find the node description for this node GID */ for (i = 0; i < result.result_cnt; i++) { node_record = osmv_get_query_node_rec(result.p_result_madw, i); if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) break; } I have a question about the intention here: The comment mentions the node description for this node GID. Is that what is meant or is the intention to get the node description corresponding to the port GUID for each MCMember record ? -- Hal From eli at dev.mellanox.co.il Tue Jun 10 07:53:50 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 10 Jun 2008 17:53:50 +0300 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: References: Message-ID: <1213109630.12453.64.camel@mtls03> On Mon, 2008-06-09 at 13:48 -0700, Roland Dreier wrote: > I'm thinking about how to add support for the various invalidate > operations to libibverbs (ie send with invalidate, RDMA read with inv, > local inv, and receive completions with an invalidated key in them). Aren't these invalidate operations valid only for physical memory regions? If that is the case, and since we can't have physical memory regions in userspace, I don't see why we need this in userspace. I also don't think IB spec defines RDMA READ with invalidate operations. From rdreier at cisco.com Tue Jun 10 07:52:57 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 07:52:57 -0700 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806101644.34693.ossrosch@linux.vnet.ibm.com> (Stefan Roscher's message of "Tue, 10 Jun 2008 16:44:31 +0200") References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> Message-ID: > During corner case testing, we noticed that some versions of ehca > do not properly transition to interrupt done in special load situations. > This can be resolved by periodically triggering EOI through H_EOI, > if eqes are pending. So just to be clear: this is a workaround for a hardware/firmware bug? - R. From PHF at zurich.ibm.com Tue Jun 10 07:52:00 2008 From: PHF at zurich.ibm.com (Philip Frey1) Date: Tue, 10 Jun 2008 16:52:00 +0200 Subject: [ofa-general] missed cq event In-Reply-To: <484D5A18.8070701@opengridcomputing.com> Message-ID: Steve, thanks for your advice. Is it possible that there is a bug in OFED 1.3 with regard to non-signaled send work requests? I noticed that when I post send work requests onto my send queue, It eventually fills up until I cannot post sends anymore. This happens with the Chelsio T3 RNIC and OFED 1.3 whenever I post send WR's that have their flags set to 0. It does not happen though when I post sends with IBV_SEND_SIGNALED. The CQ is empty in the case of non-signaled WR's (as expected) but they somehow seem to be stuck on the send queue. I use the following code: static struct ibv_send_wr tx_wr, *bad_wr; /* create send work request */ tx_wr.wr_id = tx_wr_id++; tx_wr.next = NULL; tx_wr.sg_list = sg_list; tx_wr.num_sge = num_sge; tx_wr.opcode = IBV_WR_SEND; tx_wr.send_flags = 0; /* post send work request */ ret = ibv_post_send(qp, &tx_wr, &bad_wr); if (ret) { //error } I learned that it might be necessary to post a signaled send WR after posting a number of non-signaled ones in order to clean up the SQ. Is that the case and is there no way to post non-signaled WR's that do not get stuck on the SQ? Cheers, Philip general-bounces at lists.openfabrics.org wrote on 09.06.2008 18:28:08: > Philip Frey1 wrote: > > You are right. Thanks! > > I have yet another issue: > > Sometimes I get the following message in /var/log/messages of the local host: > > post_qp_event - AE qpid 0x4e0 opcode 3 status 0x13 type 0 wrid.hi > 0x0 wrid.lo 0x65000000 > > I was looking for the status and opcode in the source and found that > opcode 3 means T3_SEND and status 0x13 means TPT_ERR_OUT_OF_RQE. > At the remote host I get and opcode 7 (T3_TERMINATE) and status 0x0 (SUCCESS). > > Clearly there is someone running out of Receive Queue Elements. The > error occurred when > doing an ibv_post_send() at the local host. Is this a coincidence or > does the local host > somehow know that there are not enough RQE's available at the remote > host? In other words, > does the TPT_ERR_OUT_OF_RQE refer to the local or to the remote receive queue? > > You have to consider the type too. type 0 indicates ingress errors, > and type 1 indicates egress. > > So the host that logged opcode 3, status 0x13, type 0 received an > incoming SEND but there were no RECV's posted at that time. The > result is a connection termination, which results in the TERMINATE > event on the peer side. > > Steve._______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Tue Jun 10 07:57:05 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 10 Jun 2008 17:57:05 +0300 Subject: [ofa-general] Trouble with IPoIB stack ? In-Reply-To: References: Message-ID: <1213109825.12453.67.camel@mtls03> Does your system respond to ping (over the Eth interface)? It appears like you have a memory leak. Can you send more output and also send more details how to reproduce this? Thanks. On Tue, 2008-06-10 at 15:32 +0200, Bart Van Assche wrote: > Hello, > > While running a stress test of Linux' iSCSI initiator I noticed that > the test ran fine over an Ethernet network but that strange memory > allocation problems were reported after I switched from Ethernet to > IPoIB. A few minutes later the whole system froze. Anyone any hints on > how I can pinpoint the cause of this issue ? > > >From the kernel messages: > > ... > [ 552.238283] scsi_wq_25: page allocation failure. order:0, mode:0x20 > [ 551.878249] scsi_wq_39: page allocation failure. order:0, mode:0x20 > ... > [ 553.369853] Call Trace: > [ 551.878264] Pid: 7860, comm: scsi_wq_39 Not tainted 2.6.25.6 #3 > [ 552.958273] Pid: 7761, comm: scsi_wq_36 Not tainted 2.6.25.6 #3 > [ 553.369866] [__alloc_pages+0x2e8/0x3d0] __alloc_pages+0x2e8/0x3d0 > [ 552.418332] [__alloc_pages+0x2e8/0x3d0] __alloc_pages+0x2e8/0x3d0 > [ 553.369875] [ib_ipoib:dev_queue_xmit+0xed/0xb410] dev_queue_xmit+0xed/0x380 > [ 552.418343] [new_slab+0x22d/0x2b0] new_slab+0x22d/0x2b0 > [ 552.418349] [__slab_alloc+0x1f5/0x4c0] __slab_alloc+0x1f5/0x4c0 > [ 553.369894] [new_slab+0x22d/0x2b0] new_slab+0x22d/0x2b0 > [ 552.778357] Pid: 7306, comm: scsi_wq_23 Not tainted 2.6.25.6 #3 > ... > > See also http://bugzilla.kernel.org/show_bug.cgi?id=10890. > > Bart. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Tue Jun 10 07:56:24 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 07:56:24 -0700 Subject: [ofa-general] missed cq event In-Reply-To: (Philip Frey1's message of "Tue, 10 Jun 2008 16:52:00 +0200") References: Message-ID: > Is it possible that there is a bug in OFED 1.3 with regard to non-signaled > send work requests? > I noticed that when I post send work requests onto my send queue, It > eventually fills up until I > cannot post sends anymore. > This happens with the Chelsio T3 RNIC and OFED 1.3 whenever I post send > WR's that have > their flags set to 0. It does not happen though when I post sends with > IBV_SEND_SIGNALED. > The CQ is empty in the case of non-signaled WR's (as expected) but they > somehow seem to > be stuck on the send queue. This is not a bug. The unsignaled work requests are not considered completed until a later signaled work request completes. So you need to periodically post a signaled work request, so that you know when the unsignaled sends have completed. - R. From rdreier at cisco.com Tue Jun 10 07:58:13 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 07:58:13 -0700 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: <1213109630.12453.64.camel@mtls03> (Eli Cohen's message of "Tue, 10 Jun 2008 17:53:50 +0300") References: <1213109630.12453.64.camel@mtls03> Message-ID: > Aren't these invalidate operations valid only for physical memory > regions? If that is the case, and since we can't have physical memory > regions in userspace, I don't see why we need this in userspace. invalidate operations are also valid for memory windows, which we will probably add support for. And a userspace application could easily be talking to a kernel driver on the other end, so send w/invalidate etc also makes sense in that case. I did leave out all the fast register through a send queue support, of course. > I also don't think IB spec defines RDMA READ with invalidate operations. Correct, but it is an iWARP operation. - R. From ossrosch at linux.vnet.ibm.com Tue Jun 10 08:01:45 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Tue, 10 Jun 2008 17:01:45 +0200 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> Message-ID: <200806101701.47090.ossrosch@linux.vnet.ibm.com> On Tuesday 10 June 2008 16:52:57 Roland Dreier wrote: > > During corner case testing, we noticed that some versions of ehca > > do not properly transition to interrupt done in special load situations. > > This can be resolved by periodically triggering EOI through H_EOI, > > if eqes are pending. > > So just to be clear: this is a workaround for a hardware/firmware bug? > > - R. > Yes it is. regards Stefan From bart.vanassche at gmail.com Tue Jun 10 08:08:45 2008 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Tue, 10 Jun 2008 17:08:45 +0200 Subject: [ofa-general] Trouble with IPoIB stack ? In-Reply-To: <1213109825.12453.67.camel@mtls03> References: <1213109825.12453.67.camel@mtls03> Message-ID: > On Tue, 2008-06-10 at 15:32 +0200, Bart Van Assche wrote: >> While running a stress test of Linux' iSCSI initiator I noticed that >> the test ran fine over an Ethernet network but that strange memory >> allocation problems were reported after I switched from Ethernet to >> IPoIB. A few minutes later the whole system froze. Anyone any hints on >> how I can pinpoint the cause of this issue ? >> >> >From the kernel messages: >> >> ... >> [ 552.238283] scsi_wq_25: page allocation failure. order:0, mode:0x20 >> [ 551.878249] scsi_wq_39: page allocation failure. order:0, mode:0x20 >> ... >> [ 553.369853] Call Trace: >> [ 551.878264] Pid: 7860, comm: scsi_wq_39 Not tainted 2.6.25.6 #3 >> [ 552.958273] Pid: 7761, comm: scsi_wq_36 Not tainted 2.6.25.6 #3 >> [ 553.369866] [__alloc_pages+0x2e8/0x3d0] __alloc_pages+0x2e8/0x3d0 >> [ 552.418332] [__alloc_pages+0x2e8/0x3d0] __alloc_pages+0x2e8/0x3d0 >> [ 553.369875] [ib_ipoib:dev_queue_xmit+0xed/0xb410] dev_queue_xmit+0xed/0x380 >> [ 552.418343] [new_slab+0x22d/0x2b0] new_slab+0x22d/0x2b0 >> [ 552.418349] [__slab_alloc+0x1f5/0x4c0] __slab_alloc+0x1f5/0x4c0 >> [ 553.369894] [new_slab+0x22d/0x2b0] new_slab+0x22d/0x2b0 >> [ 552.778357] Pid: 7306, comm: scsi_wq_23 Not tainted 2.6.25.6 #3 >> ... >> >> See also http://bugzilla.kernel.org/show_bug.cgi?id=10890. On Tue, Jun 10, 2008 at 4:57 PM, Eli Cohen wrote: > Does your system respond to ping (over the Eth interface)? > > It appears like you have a memory leak. Can you send more output and > also send more details how to reproduce this? Hello Eli, I will repeat the test such that I can test whether the system still responds to ping on the Ethernet interface. Regarding the kernel logs and how to reproduce this issue: this information is present as attachments in http://bugzilla.kernel.org/show_bug.cgi?id=10890. Please let me know if you need more information. Bart. From eli at dev.mellanox.co.il Tue Jun 10 08:20:11 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 10 Jun 2008 18:20:11 +0300 Subject: [ofa-general] Trouble with IPoIB stack ? In-Reply-To: References: <1213109825.12453.67.camel@mtls03> Message-ID: <4e6a6b3c0806100820j19849dcfq911b17156996e21d@mail.gmail.com> > I will repeat the test such that I can test whether the system still > responds to ping on the Ethernet interface. > > Regarding the kernel logs and how to reproduce this issue: this > information is present as attachments in > http://bugzilla.kernel.org/show_bug.cgi?id=10890. > > Please let me know if you need more information. > Thanks. I will ask our verification team to reproduce here. From swise at opengridcomputing.com Tue Jun 10 08:21:40 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 10 Jun 2008 10:21:40 -0500 Subject: [ofa-general] missed cq event In-Reply-To: References: Message-ID: <484E9C04.2020007@opengridcomputing.com> Philip Frey1 wrote: > > Steve, thanks for your advice. > > Is it possible that there is a bug in OFED 1.3 with regard to > non-signaled send work requests? > I noticed that when I post send work requests onto my send queue, It > eventually fills up until I > cannot post sends anymore. > This happens with the Chelsio T3 RNIC and OFED 1.3 whenever I post > send WR's that have > their flags set to 0. It does not happen though when I post sends with > IBV_SEND_SIGNALED. > The CQ is empty in the case of non-signaled WR's (as expected) but > they somehow seem to > be stuck on the send queue. > > I use the following code: > static struct ibv_send_wr tx_wr, *bad_wr; > > /* create send work request */ > tx_wr.wr_id = tx_wr_id++; > tx_wr.next = NULL; > tx_wr.sg_list = sg_list; > tx_wr.num_sge = num_sge; > tx_wr.opcode = IBV_WR_SEND; > tx_wr.send_flags = 0; > > /* post send work request */ > ret = ibv_post_send(qp, &tx_wr, &bad_wr); > if (ret) { > //error > } > > I learned that it might be necessary to post a signaled send WR after > posting a number of non-signaled > ones in order to clean up the SQ. Is that the case and is there no way > to post non-signaled WR's that > do not get stuck on the SQ? > That is the case. You must post at least one signaled WR every SQ-depth's worth of posts to force the SQ to be cleaned up. From the driver's perspective, it cannot tell whether an unsignaled WR completed successfully until a subsequent signaled work request completes. This boils down to a requirement that you post at least one signaled WR before filling up the SQ. Steve. From thomas.eggestad at gmail.com Tue Jun 10 09:02:17 2008 From: thomas.eggestad at gmail.com (Thomas Eggestad) Date: Tue, 10 Jun 2008 18:02:17 +0200 Subject: [ofa-general] unsing ibverbs 1.0 with rdmacm Message-ID: hello, I apologize if this is the wrong list to ask my question. If it is wrong please point me to where I should look. I am in the process of implementing support for the rdmacm. For backwards compatibility and code reuse I want to continue to use the ibverbs 1.0 abi for the communication. In pseudo code I do: dl__create_event_channel = dlvsym((handle1), "create_event_channel" , "RDMACM_1.0"); dl_rdma_create_id = dlvsym((handle1), "rdma_create_id" , "RDMACM_1.0"); dl_rdma_bind_addr = dlvsym((handle1), "rdma_bind_addr" , "RDMACM_1.0"); dl_rdma_resolve_route = dlvsym((handle1), "rdma_resolve_route" , "RDMACM_1.0"); dl_ibv_alloc_pd = dlvsym((handle2), "ibv_alloc_pd", "IBVERBS_1.0") cm_event_channel = rdma_create_event_channel() dl_rdma_create_id( cm_event_channel, &cm_id, foo, RDMA_PS_TCP); dl_rdma_bind_addr (cm_id, (struct sockaddr*) &sin); dl_rdma_resolve_route(cm_id, 2000); pd = dl_ibv_alloc_pd(cm_id->verbs); The program get a sigsegv in: 0 __ibv_alloc_pd (context=0x2aaaabb5de40) at src/verbs.c:143 1 0x0000003f7f405ff1 in __ibv_alloc_pd_1_0 (context=0x1e7504e0) [...] Is this supported? If yes, do anybody know of a test program that do this and can point me to it? I have a strong feeling that there is something I have not understood. The system is: rhel5 x86_64 offed 1.3.1 If you believe you need more info please tell me and I will provide it. If this is a case of read the manual please could you give me a link. Regards, Thomas From weiny2 at llnl.gov Tue Jun 10 09:05:34 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 10 Jun 2008 09:05:34 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080610090534.3197494c.weiny2@llnl.gov> Yea, that comment does not make sense. I am searching the node records for the node which has the port defined in the member record. The interface ID is simply the port guid, is it not? uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); I guess the comments should read. /* go through the node records searching for a port guid which matches * this port gid interface id. * This gives us a node description to print. */ Does that make more sense? Ira On Tue, 10 Jun 2008 07:46:26 -0700 Hal Rosenstock wrote: > Ira, > > In saquery.c:print_multicast_member_record, there is the following: > > /* go through and find the node description for this node GID */ > for (i = 0; i < result.result_cnt; i++) { > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > break; > } > > I have a question about the intention here: > > The comment mentions the node description for this node GID. Is that > what is meant or is the intention to get the node description > corresponding to the port GUID for each MCMember record ? > > -- Hal > > From rdreier at cisco.com Tue Jun 10 09:13:15 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 09:13:15 -0700 Subject: [ofa-general] unsing ibverbs 1.0 with rdmacm In-Reply-To: (Thomas Eggestad's message of "Tue, 10 Jun 2008 18:02:17 +0200") References: Message-ID: > I am in the process of implementing support for the rdmacm. > For backwards compatibility and code reuse I want to continue > to use the ibverbs 1.0 abi for the communication. I suspect that this is more trouble than it's worth, because librdmacm requires and uses the libibverbs 1.1 ABI. I think the problem you run into is: > pd = dl_ibv_alloc_pd(cm_id->verbs); librdmacm is giving you an IBVERBS_1.1 context structure in cm_id->verbs, but then you pass it into an IBVERBS_1.0 function. Maybe you could come up with some hack that manually converts back and forth between the two versions of various structures, but as I said before, I think it would be much simpler to just link against and use the 1.1 ABI if you want to use librdmacm. - R. From rdreier at cisco.com Tue Jun 10 09:18:50 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 09:18:50 -0700 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806101701.47090.ossrosch@linux.vnet.ibm.com> (Stefan Roscher's message of "Tue, 10 Jun 2008 17:01:45 +0200") References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> <200806101701.47090.ossrosch@linux.vnet.ibm.com> Message-ID: > > So just to be clear: this is a workaround for a hardware/firmware bug? > Yes it is. OK, so paulus et al... does it seem like a good approach to call H_EOI from driver code (given that this driver makes tons of other hcalls)? How critical is this? Since you said "corner case testing" I suspect we can defer this to 2.6.27 and maybe get it into -stable later? Also, out of curiousity: > +u64 hipz_h_eoi(int irq) > +{ > + int value; > + unsigned long xirr; > + > + iosync(); what is the iosync() required for here? > + value = (0xff << 24) | irq; > + xirr = value & 0xffffffff; given that irq and value are ints, is there any possible way value could have bits outside of the low 32 set? If you're worried about sign extension isn't it simpler to just make value unsigned? > + return plpar_hcall_norets(H_EOI, xirr); > +} ie why not: u64 hipz_h_eoi(int irq) { unsigned xirr; iosync(); xirr = (0xff << 24) | irq; return plpar_hcall_norets(H_EOI, xirr); } From hrosenstock at xsigo.com Tue Jun 10 10:23:24 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 10:23:24 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <20080610090534.3197494c.weiny2@llnl.gov> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> Message-ID: <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-10 at 09:05 -0700, Ira Weiny wrote: > Yea, that comment does not make sense. > > I am searching the node records for the node which has the port defined in the > member record. The interface ID is simply the port guid, is it not? > > uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); > > I guess the comments should read. > > /* go through the node records searching for a port guid which matches > * this port gid interface id. > * This gives us a node description to print. > */ > > Does that make more sense? The makes more sense but I don't think that the code does what's intended. -- Hal > Ira > > On Tue, 10 Jun 2008 07:46:26 -0700 > Hal Rosenstock wrote: > > > Ira, > > > > In saquery.c:print_multicast_member_record, there is the following: > > > > /* go through and find the node description for this node GID */ > > for (i = 0; i < result.result_cnt; i++) { > > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > > break; > > } > > > > I have a question about the intention here: > > > > The comment mentions the node description for this node GID. Is that > > what is meant or is the intention to get the node description > > corresponding to the port GUID for each MCMember record ? > > > > -- Hal > > > > From sashak at voltaire.com Tue Jun 10 10:35:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Jun 2008 20:35:39 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/include/iba/ib_types.h: Fix comment In-Reply-To: <1213107959.14047.229.camel@hrosenstock-ws.xsigo.com> References: <1213107959.14047.229.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080610173539.GA12567@sashak.voltaire.com> On 07:25 Tue 10 Jun , Hal Rosenstock wrote: > opensm/include/iba/ib_types.h: Fix comment > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sean.hefty at intel.com Tue Jun 10 10:41:33 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 10 Jun 2008 10:41:33 -0700 Subject: [ofa-general] librdmacm: rdma_bind_addr() doesn't return port In-Reply-To: <3B6F5C9068D5864096EB291236C3386F03E9AC28@xmb-sjc-21d.amer.cisco.com> References: <3B6F5C9068D5864096EB291236C3386F03E9AC28@xmb-sjc-21d.amer.cisco.com> Message-ID: <000001c8cb21$3961d010$9d37170a@amr.corp.intel.com> >I recommend just changing the man page Thanks - I'll pull in the man page change into my git tree. Can I get a signed-off-by line? - Sean From aafabbri at cisco.com Tue Jun 10 11:06:10 2008 From: aafabbri at cisco.com (Aaron Fabbri (aafabbri)) Date: Tue, 10 Jun 2008 11:06:10 -0700 Subject: [ofa-general] [PATCH] RE: librdmacm: rdma_bind_addr() doesn't return port Message-ID: <3B6F5C9068D5864096EB291236C3386F03EEF9C7@xmb-sjc-21d.amer.cisco.com> Make manpage reflect the fact that rdma_bind_addr(port=0) does not return the assigned port. Signed-off-by: Aaron Fabbri cisco.com> diff --git a/man/rdma_bind_addr.3 b/man/rdma_bind_addr.3 index 46c48a3..55d1b9c 100644 --- a/man/rdma_bind_addr.3 +++ b/man/rdma_bind_addr.3 @@ -22,8 +22,8 @@ to a specific port number, but it may also be called on the ac of a connection before calling rdma_resolve_addr to bind to a specific address. .P -If used to bind to port 0, the rdma_cm will select an available port -and return it to the user. +If used to bind to port 0, the rdma_cm will select an available port, which can +be retrieved with rdma_get_src_port(3). .SH "SEE ALSO" rdma_create_id(3), rdma_listen(3), rdma_resolve_addr(3), rdma_create_qp(3), rdma_get_local_addr(3), rdma_get_src_port(3) From weiny2 at llnl.gov Tue Jun 10 11:11:31 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 10 Jun 2008 11:11:31 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080610111131.039f399e.weiny2@llnl.gov> On Tue, 10 Jun 2008 10:23:24 -0700 Hal Rosenstock wrote: > On Tue, 2008-06-10 at 09:05 -0700, Ira Weiny wrote: > > Yea, that comment does not make sense. > > > > I am searching the node records for the node which has the port defined in the > > member record. The interface ID is simply the port guid, is it not? > > > > uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); > > > > I guess the comments should read. > > > > /* go through the node records searching for a port guid which matches > > * this port gid interface id. > > * This gives us a node description to print. > > */ > > > > Does that make more sense? > > The makes more sense but I don't think that the code does what's > intended. Why do you say that? Have you found some configuration in which this does not work? >From the spec pg 905: "Note: If a channel adapter or router has multiple ports on the same subnet, there will be multiple NodeRecords available for that node from SA, one for each possible PortGUID value of NodeInfo for that node on that subnet." So each NodeRecord should have a unique port GUID in it. Or do you think the GID interface ID will not always equal the GUID? I thought I read that somewhere but perhaps I am wrong. I don't find it right now. :-/ Ira > > -- Hal > > > Ira > > > > On Tue, 10 Jun 2008 07:46:26 -0700 > > Hal Rosenstock wrote: > > > > > Ira, > > > > > > In saquery.c:print_multicast_member_record, there is the following: > > > > > > /* go through and find the node description for this node GID */ > > > for (i = 0; i < result.result_cnt; i++) { > > > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > > > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > > > break; > > > } > > > > > > I have a question about the intention here: > > > > > > The comment mentions the node description for this node GID. Is that > > > what is meant or is the intention to get the node description > > > corresponding to the port GUID for each MCMember record ? > > > > > > -- Hal > > > > > > > From hrosenstock at xsigo.com Tue Jun 10 11:12:31 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 11:12:31 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/saquery.c: In print_multicast_group_records, only query NodeRecords when needed Message-ID: <1213121551.14047.265.camel@hrosenstock-ws.xsigo.com> infiniband-diags/saquery.c: In print_multicast_group_records, only query NodeRecords when needed. This is a minor optimization. Signed-off-by: Hal Rosenstock diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c index e727940..13c4ea0 100644 --- a/infiniband-diags/src/saquery.c +++ b/infiniband-diags/src/saquery.c @@ -1048,9 +1048,12 @@ print_multicast_group_records(osm_bind_handle_t bind_handle, int members) return (status); mc_group_result = result; - status = get_all_records(bind_handle, IB_MAD_ATTR_NODE_RECORD, node_attr_offset, 0); - if (status != IB_SUCCESS) - goto return_mc; + if (members) { + status = get_all_records(bind_handle, IB_MAD_ATTR_NODE_RECORD, + node_attr_offset, 0); + if (status != IB_SUCCESS) + goto return_mc; + } for (i = 0; i < mc_group_result.result_cnt; i++) { mcast_record = osmv_get_query_mc_rec(mc_group_result.p_result_madw, i); @@ -1060,7 +1063,8 @@ print_multicast_group_records(osm_bind_handle_t bind_handle, int members) print_multicast_member_record(mcast_record); } - return_mad(); + if (members) + return_mad(); return_mc: /* return_mad for the mc_group_result */ From weiny2 at llnl.gov Tue Jun 10 11:16:17 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 10 Jun 2008 11:16:17 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <20080610111131.039f399e.weiny2@llnl.gov> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> Message-ID: <20080610111617.40f85eef.weiny2@llnl.gov> On Tue, 10 Jun 2008 11:11:31 -0700 Ira Weiny wrote: > On Tue, 10 Jun 2008 10:23:24 -0700 > Hal Rosenstock wrote: > > > On Tue, 2008-06-10 at 09:05 -0700, Ira Weiny wrote: > > > Yea, that comment does not make sense. > > > > > > I guess the comments should read. > > > > > > /* go through the node records searching for a port guid which matches > > > * this port gid interface id. > > > * This gives us a node description to print. > > > */ > > > > > > Does that make more sense? > > > > The makes more sense but I don't think that the code does what's > > intended. > > Why do you say that? Have you found some configuration in which this does not > work? > > From the spec pg 905: ^^^ Ooops this is misquoted... (I said this) sorry, Ira > > "Note: If a channel adapter or router has multiple ports on the same subnet, > there will be multiple NodeRecords available for that node from SA, one for > each possible PortGUID value of NodeInfo for that node on that subnet." > > So each NodeRecord should have a unique port GUID in it. > > Or do you think the GID interface ID will not always equal the GUID? I thought > I read that somewhere but perhaps I am wrong. I don't find it right now. :-/ > > Ira > > > > > -- Hal > > > > > Ira > > > > > > On Tue, 10 Jun 2008 07:46:26 -0700 > > > Hal Rosenstock wrote: > > > > > > > Ira, > > > > > > > > In saquery.c:print_multicast_member_record, there is the following: > > > > > > > > /* go through and find the node description for this node GID */ > > > > for (i = 0; i < result.result_cnt; i++) { > > > > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > > > > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > > > > break; > > > > } > > > > > > > > I have a question about the intention here: > > > > > > > > The comment mentions the node description for this node GID. Is that > > > > what is meant or is the intention to get the node description > > > > corresponding to the port GUID for each MCMember record ? > > > > > > > > -- Hal > > > > > > > > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Tue Jun 10 11:16:24 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 11:16:24 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <20080610111131.039f399e.weiny2@llnl.gov> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> Message-ID: <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-10 at 11:11 -0700, Ira Weiny wrote: > On Tue, 10 Jun 2008 10:23:24 -0700 > Hal Rosenstock wrote: > > > On Tue, 2008-06-10 at 09:05 -0700, Ira Weiny wrote: > > > Yea, that comment does not make sense. > > > > > > I am searching the node records for the node which has the port defined in the > > > member record. The interface ID is simply the port guid, is it not? > > > > > > uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); > > > > > > I guess the comments should read. > > > > > > /* go through the node records searching for a port guid which matches > > > * this port gid interface id. > > > * This gives us a node description to print. > > > */ > > > > > > Does that make more sense? > > > > The makes more sense but I don't think that the code does what's > > intended. > > Why do you say that? Looks to me like it relies on some node GUID being same as port GUID. While that's allowable for one port, it won't always be the case. > Have you found some configuration in which this does not > work? Yes. -- Hal > From the spec pg 905: > > "Note: If a channel adapter or router has multiple ports on the same subnet, > there will be multiple NodeRecords available for that node from SA, one for > each possible PortGUID value of NodeInfo for that node on that subnet." > > So each NodeRecord should have a unique port GUID in it. > > Or do you think the GID interface ID will not always equal the GUID? I thought > I read that somewhere but perhaps I am wrong. I don't find it right now. :-/ > > Ira > > > > > -- Hal > > > > > Ira > > > > > > On Tue, 10 Jun 2008 07:46:26 -0700 > > > Hal Rosenstock wrote: > > > > > > > Ira, > > > > > > > > In saquery.c:print_multicast_member_record, there is the following: > > > > > > > > /* go through and find the node description for this node GID */ > > > > for (i = 0; i < result.result_cnt; i++) { > > > > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > > > > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > > > > break; > > > > } > > > > > > > > I have a question about the intention here: > > > > > > > > The comment mentions the node description for this node GID. Is that > > > > what is meant or is the intention to get the node description > > > > corresponding to the port GUID for each MCMember record ? > > > > > > > > -- Hal > > > > > > > > > > From weiny2 at llnl.gov Tue Jun 10 11:26:17 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 10 Jun 2008 11:26:17 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080610112617.3563f3ed.weiny2@llnl.gov> On Tue, 10 Jun 2008 11:16:24 -0700 Hal Rosenstock wrote: > On Tue, 2008-06-10 at 11:11 -0700, Ira Weiny wrote: > > On Tue, 10 Jun 2008 10:23:24 -0700 > > Hal Rosenstock wrote: > > > > > On Tue, 2008-06-10 at 09:05 -0700, Ira Weiny wrote: > > > > Yea, that comment does not make sense. > > > > > > > > I am searching the node records for the node which has the port defined in the > > > > member record. The interface ID is simply the port guid, is it not? > > > > > > > > uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); > > > > > > > > I guess the comments should read. > > > > > > > > /* go through the node records searching for a port guid which matches > > > > * this port gid interface id. > > > > * This gives us a node description to print. > > > > */ > > > > > > > > Does that make more sense? > > > > > > The makes more sense but I don't think that the code does what's > > > intended. > > > > Why do you say that? > > Looks to me like it relies on some node GUID being same as port GUID. > While that's allowable for one port, it won't always be the case. In my test system the Node GUID and port GUID are different and this works. I am specifically using the port guid out of the NodeInfo struct of the NodeRecord. So I should be using the port guid vs port GID ID _only_. > > > Have you found some configuration in which this does not > > work? > > Yes. :-( Sorry, I'm pretty sure this should be ok... Ira > > -- Hal > > > From the spec pg 905: > > > > "Note: If a channel adapter or router has multiple ports on the same subnet, > > there will be multiple NodeRecords available for that node from SA, one for > > each possible PortGUID value of NodeInfo for that node on that subnet." > > > > So each NodeRecord should have a unique port GUID in it. > > > > Or do you think the GID interface ID will not always equal the GUID? I thought > > I read that somewhere but perhaps I am wrong. I don't find it right now. :-/ > > > > Ira > > > > > > > > -- Hal > > > > > > > Ira > > > > > > > > On Tue, 10 Jun 2008 07:46:26 -0700 > > > > Hal Rosenstock wrote: > > > > > > > > > Ira, > > > > > > > > > > In saquery.c:print_multicast_member_record, there is the following: > > > > > > > > > > /* go through and find the node description for this node GID */ > > > > > for (i = 0; i < result.result_cnt; i++) { > > > > > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > > > > > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > > > > > break; > > > > > } > > > > > > > > > > I have a question about the intention here: > > > > > > > > > > The comment mentions the node description for this node GID. Is that > > > > > what is meant or is the intention to get the node description > > > > > corresponding to the port GUID for each MCMember record ? > > > > > > > > > > -- Hal > > > > > > > > > > > > > > From pw at osc.edu Tue Jun 10 11:26:55 2008 From: pw at osc.edu (Pete Wyckoff) Date: Tue, 10 Jun 2008 14:26:55 -0400 Subject: [ofa-general] missed cq event In-Reply-To: References: Message-ID: <20080610182655.GE22762@osc.edu> rdreier at cisco.com wrote on Tue, 10 Jun 2008 07:56 -0700: > > Is it possible that there is a bug in OFED 1.3 with regard to non-signaled > > send work requests? > > I noticed that when I post send work requests onto my send queue, It > > eventually fills up until I > > cannot post sends anymore. > > This happens with the Chelsio T3 RNIC and OFED 1.3 whenever I post send > > WR's that have > > their flags set to 0. It does not happen though when I post sends with > > IBV_SEND_SIGNALED. > > The CQ is empty in the case of non-signaled WR's (as expected) but they > > somehow seem to > > be stuck on the send queue. > > This is not a bug. The unsignaled work requests are not considered > completed until a later signaled work request completes. So you need to > periodically post a signaled work request, so that you know when the > unsignaled sends have completed. You know, I've never quite liked this fact. I've seen multiple people, including myself, get tripped up by this issue. And now we have lots of applications with code to: determine the queue depth, increment a counter on every send and periodically post a SIGNALED one, and handle the uninteresting CQ event. Trying not to think about how the hardware works for a moment, this issue can be seen as yet another little detail that apps people must learn to use RDMA devices. Any ideas on how verbs could handle this case itself? Sorry for the rant. -- Pete From hrosenstock at xsigo.com Tue Jun 10 11:35:19 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 11:35:19 -0700 Subject: [ofa-general] Re: infiniband-diags/saquery.c:print_multicast_member_record question In-Reply-To: <20080610112617.3563f3ed.weiny2@llnl.gov> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> <20080610112617.3563f3ed.weiny2@llnl.gov> Message-ID: <1213122919.14047.280.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-10 at 11:26 -0700, Ira Weiny wrote: > On Tue, 10 Jun 2008 11:16:24 -0700 > Hal Rosenstock wrote: > > > On Tue, 2008-06-10 at 11:11 -0700, Ira Weiny wrote: > > > On Tue, 10 Jun 2008 10:23:24 -0700 > > > Hal Rosenstock wrote: > > > > > > > On Tue, 2008-06-10 at 09:05 -0700, Ira Weiny wrote: > > > > > Yea, that comment does not make sense. > > > > > > > > > > I am searching the node records for the node which has the port defined in the > > > > > member record. The interface ID is simply the port guid, is it not? > > > > > > > > > > uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); > > > > > > > > > > I guess the comments should read. > > > > > > > > > > /* go through the node records searching for a port guid which matches > > > > > * this port gid interface id. > > > > > * This gives us a node description to print. > > > > > */ > > > > > > > > > > Does that make more sense? > > > > > > > > The makes more sense but I don't think that the code does what's > > > > intended. > > > > > > Why do you say that? > > > > Looks to me like it relies on some node GUID being same as port GUID. > > While that's allowable for one port, it won't always be the case. > > In my test system the Node GUID and port GUID are different and this works. I > am specifically using the port guid out of the NodeInfo struct of the > NodeRecord. So I should be using the port guid vs port GID ID _only_. Not sure what you mean; port GID 0 = subnet prefix + port GUID. Not sure what SMs support other port GIDs than this but OpenSM doesn't. > > > Have you found some configuration in which this does not > > > work? > > > > Yes. > > :-( Sorry, I'm pretty sure this should be ok... It's possible it's an environment thing but there's a couple of suboptimialities in print_multicast_member_record: 1. Does osmv_get_query_node_rec always return a non NULL pointer ? 2. If the loop fails to find a match, the last node description is used. -- Hal > Ira > > > > > -- Hal > > > > > From the spec pg 905: > > > > > > "Note: If a channel adapter or router has multiple ports on the same subnet, > > > there will be multiple NodeRecords available for that node from SA, one for > > > each possible PortGUID value of NodeInfo for that node on that subnet." > > > > > > So each NodeRecord should have a unique port GUID in it. > > > > > > Or do you think the GID interface ID will not always equal the GUID? I thought > > > I read that somewhere but perhaps I am wrong. I don't find it right now. :-/ > > > > > > Ira > > > > > > > > > > > -- Hal > > > > > > > > > Ira > > > > > > > > > > On Tue, 10 Jun 2008 07:46:26 -0700 > > > > > Hal Rosenstock wrote: > > > > > > > > > > > Ira, > > > > > > > > > > > > In saquery.c:print_multicast_member_record, there is the following: > > > > > > > > > > > > /* go through and find the node description for this node GID */ > > > > > > for (i = 0; i < result.result_cnt; i++) { > > > > > > node_record = osmv_get_query_node_rec(result.p_result_madw, i); > > > > > > if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > > > > > > break; > > > > > > } > > > > > > > > > > > > I have a question about the intention here: > > > > > > > > > > > > The comment mentions the node description for this node GID. Is that > > > > > > what is meant or is the intention to get the node description > > > > > > corresponding to the port GUID for each MCMember record ? > > > > > > > > > > > > -- Hal > > > > > > > > > > > > > > > > > > From rdreier at cisco.com Tue Jun 10 12:01:43 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Jun 2008 12:01:43 -0700 Subject: [ofa-general] missed cq event In-Reply-To: <20080610182655.GE22762@osc.edu> (Pete Wyckoff's message of "Tue, 10 Jun 2008 14:26:55 -0400") References: <20080610182655.GE22762@osc.edu> Message-ID: > You know, I've never quite liked this fact. I've seen multiple > people, including myself, get tripped up by this issue. And now we > have lots of applications with code to: determine the queue depth, > increment a counter on every send and periodically post a SIGNALED > one, and handle the uninteresting CQ event. > > Trying not to think about how the hardware works for a moment, this > issue can be seen as yet another little detail that apps people must > learn to use RDMA devices. Any ideas on how verbs could handle this > case itself? I don't see a good way that this could be handled at the verbs level. How does the app know when the resources associated with the send request can be freed? How would you size your CQ (given than any and all unsignaled requests might generate a completion with error)? What would you want to happen if an app posts 101 work requests to a queue of depth 100 before any WRs complete? It seems to me that the app needs the periodic completion just as much as the hardware driver needs it for book-keeping. - R. From pw at osc.edu Tue Jun 10 12:58:25 2008 From: pw at osc.edu (Pete Wyckoff) Date: Tue, 10 Jun 2008 15:58:25 -0400 Subject: [ofa-general] missed cq event In-Reply-To: References: <20080610182655.GE22762@osc.edu> Message-ID: <20080610195825.GA22901@osc.edu> rdreier at cisco.com wrote on Tue, 10 Jun 2008 12:01 -0700: > > You know, I've never quite liked this fact. I've seen multiple > > people, including myself, get tripped up by this issue. And now we > > have lots of applications with code to: determine the queue depth, > > increment a counter on every send and periodically post a SIGNALED > > one, and handle the uninteresting CQ event. > > > > Trying not to think about how the hardware works for a moment, this > > issue can be seen as yet another little detail that apps people must > > learn to use RDMA devices. Any ideas on how verbs could handle this > > case itself? > > I don't see a good way that this could be handled at the verbs level. > How does the app know when the resources associated with the send > request can be freed? How would you size your CQ (given than any and > all unsignaled requests might generate a completion with error)? > > What would you want to happen if an app posts 101 work requests to a > queue of depth 100 before any WRs complete? > > It seems to me that the app needs the periodic completion just as much > as the hardware driver needs it for book-keeping. Those are valid comments in general. My thoughts are coming from a particular application mindset: for situations where you need app-level flow control anyway. One example is RPC, where a send going out will always generate a receive coming back in. Remote storage or file systems are a case where we need flow control anyway to avoid overflowing various queueing points in the server, so it is easy to piggy-back freeing the send resources when the write ack or read result comes back. It's not possible to free the send resources before seeing the app-level ack from the other side; something can go wrong even after the send CQ event happens. CQ depth can be chosen based on the maximum number of outstanding file system requests as dictated by the server, as we still need the space for errors as you point out. Outstanding WR limit still applies as a separate per-QP limit and is also sizeable based on app flow control constraints. You're certainly right that the need to reap even unsignaled sends is needed for some apps. I'm just looking for a shortcut if my app does not ever need to find out about send completions. Just to simplify the code, not expecting anything will go faster or require less memory. -- Pete From weiny2 at llnl.gov Tue Jun 10 13:18:03 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 10 Jun 2008 13:18:03 -0700 Subject: [ofa-general] [PATCH] infiniband-diags/src/saquery.c: fix potential core dump (Was: Re: infiniband-diags/saquery.c:print_multicast_member_record question) In-Reply-To: <1213122919.14047.280.camel@hrosenstock-ws.xsigo.com> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> <20080610112617.3563f3ed.weiny2@llnl.gov> <1213122919.14047.280.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080610131803.34c7a889.weiny2@llnl.gov> On Tue, 10 Jun 2008 11:35:19 -0700 Hal Rosenstock wrote: > On Tue, 2008-06-10 at 11:26 -0700, Ira Weiny wrote: > > On Tue, 10 Jun 2008 11:16:24 -0700 > > Hal Rosenstock wrote: > > > > > > Looks to me like it relies on some node GUID being same as port GUID. > > > While that's allowable for one port, it won't always be the case. > > > > In my test system the Node GUID and port GUID are different and this works. I > > am specifically using the port guid out of the NodeInfo struct of the > > NodeRecord. So I should be using the port guid vs port GID ID _only_. > > Not sure what you mean; port GID 0 = subnet prefix + port GUID. Not sure > what SMs support other port GIDs than this but OpenSM doesn't. I mean the port gid == subnet prefix + port GUID. Therefore if you mask off the subnet prefix (ie only use the lower 64bits of the gid) you should end up with the GUID, right? > > > > > Have you found some configuration in which this does not > > > > work? > > > > > > Yes. > > > > :-( Sorry, I'm pretty sure this should be ok... > > It's possible it's an environment thing but there's a couple of > suboptimialities in print_multicast_member_record: That is probably true. I see you have already sent a patch to not do the node record queries unless necessary. > > 1. Does osmv_get_query_node_rec always return a non NULL pointer ? :-/ Yea that is a bug. Yes using node_record would be bad if for some reason the node_record did not exists. However, the port should not be a member of the mcast group if it does not exist in the SA. So... you should not have a mcast member record. I think there is a race condition here that I did not check, that is bad. > 2. If the loop fails to find a match, the last node description is used. Yep, that is a bug as well. I guess this masks the above situation. Would you agree this is a corner case? Unless your fabric is changing quickly there should be a node record for each member record right? Patch below. Should there be a warning printed in the case the node_record is not found? Ira >From ac10f52fd65b0b34b409ca2aa266a7363ae32e2c Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Tue, 10 Jun 2008 13:07:43 -0700 Subject: [PATCH] infiniband-diags/src/saquery.c: fix potential core dump and/or incorrect node descriptions from being printed if a corresponding node_record is not found for a multicast member record. Signed-off-by: Ira K. Weiny --- infiniband-diags/src/saquery.c | 17 +++++++++++------ 1 files changed, 11 insertions(+), 6 deletions(-) diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c index e727940..89754f6 100644 --- a/infiniband-diags/src/saquery.c +++ b/infiniband-diags/src/saquery.c @@ -354,14 +354,19 @@ print_multicast_member_record(ib_member_rec_t *p_mcmr) uint64_t gid_prefix = cl_ntoh64( p_mcmr->port_gid.unicast.prefix ); uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); uint16_t mlid = cl_ntoh16( p_mcmr->mlid ); - ib_node_record_t *node_record = NULL; int i = 0; + char *node_name = ""; - /* go through and find the node description for this node GID */ + /* go through the node records searching for a port guid which matches + * this port gid interface id. + * This gives us a node name to print, if available. + */ for (i = 0; i < result.result_cnt; i++) { - node_record = osmv_get_query_node_rec(result.p_result_madw, i); - if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) + ib_node_record_t *nr = osmv_get_query_node_rec(result.p_result_madw, i); + if (cl_ntoh64(nr->node_info.port_guid) == gid_interface_id) { + node_name = clean_nodedesc((char *)nr->node_desc.description); break; + } } if (requested_name) { @@ -370,7 +375,7 @@ print_multicast_member_record(ib_member_rec_t *p_mcmr) "0x%016" PRIx64 " (%s)\n", gid_prefix, gid_interface_id, - clean_nodedesc((char *)node_record->node_desc.description) + node_name ); } } else { @@ -391,7 +396,7 @@ print_multicast_member_record(ib_member_rec_t *p_mcmr) gid_interface_id, p_mcmr->scope_state, p_mcmr->proxy_join, - clean_nodedesc((char *)node_record->node_desc.description) + node_name ); } } -- 1.5.1 From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:02:14 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:02:14 -0400 Subject: [ofa-general] [PATCH v4 00/14] QLogic VNIC Driver Message-ID: <20080610205633.11186.45499.stgit@dale> Roland, This is the fourth round of QLogic Virtual NIC driver patch series for submission to 2.6.27 kernel. The series has been tested against your for-2.6.27 branch. Based on comments received on third series of patches, following fixes have been introduced in this series: - Single value per sysfs file created by QLogic VNIC Driver. - Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic describing the sysfs interface of the QLogic VNIC Driver. - Use of existing dev->stats instead of driver's private vnic->stats. - Changing more generic name completion_callback_cleanup to the one more driver specific - vnic_completion_cleanup - Removal of the field vnic->open - Minor Modifications to linked list manipulations. The sparse endianness checking for the driver did not give any warnings and checkpatch.pl have few warnings indicating lines slightly longer than 80 columns. Background: As mentioned in the earlier versions of patch series, this series adds QLogic Virtual NIC (VNIC) driver which works in conjunction with the the QLogic Ethernet Virtual I/O Controller (EVIC) hardware. The VNIC driver along with the QLogic EVIC's two 10 Gigabit ethernet ports, enables Infiniband clusters to connect to Ethernet networks. This driver also works with the earlier version of the I/O Controller, the VEx. The QLogic VNIC driver creates virtual ethernet interfaces and tunnels the Ethernet data to/from the EVIC over Infiniband using an Infiniband reliable connection. [PATCH v4 01/14] QLogic VNIC: Driver - netdev implementation. [PATCH v4 02/14] QLogic VNIC: Netpath - abstraction of connection to EVIC/VEx. [PATCH v4 03/14] QLogic VNIC: Implementation of communication protocol with EVIC/VEx. [PATCH v4 04/14] QLogic VNIC: Implementation of Control path of communication protocol. [PATCH v4 05/14] QLogic VNIC: Implementation of Data path of communication protocol. [PATCH v4 06/14] QLogic VNIC: IB core stack interaction. [PATCH v4 07/14] QLogic VNIC: Handling configurable parameters of the driver. [PATCH v4 08/14] QLogic VNIC: sysfs interface implementation for the driver. [PATCH v4 09/14] QLogic VNIC: IB Multicast for Ethernet broadcast/multicast. [PATCH v4 10/14] QLogic VNIC: Driver Statistics collection. [PATCH v4 11/14] QLogic VNIC: Driver utility file - implements various utility macros. [PATCH v4 12/14] QLogic VNIC: Driver Kconfig and Makefile. [PATCH v4 13/14] QLogic VNIC: Modifications to IB Kconfig and Makefile. [PATCH v4 14/14] QLogic VNIC: sysfs Documentation. .../ABI/testing/sysfs-class-infiniband-qlgc-vnic | 179 ++ drivers/infiniband/Kconfig | 2 drivers/infiniband/Makefile | 1 drivers/infiniband/ulp/qlgc_vnic/Kconfig | 19 drivers/infiniband/ulp/qlgc_vnic/Makefile | 13 drivers/infiniband/ulp/qlgc_vnic/vnic_config.c | 379 +++ drivers/infiniband/ulp/qlgc_vnic/vnic_config.h | 240 ++ drivers/infiniband/ulp/qlgc_vnic/vnic_control.c | 2286 ++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_control.h | 180 ++ .../infiniband/ulp/qlgc_vnic/vnic_control_pkt.h | 368 +++ drivers/infiniband/ulp/qlgc_vnic/vnic_data.c | 1492 +++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_data.h | 206 ++ drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c | 1057 +++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h | 207 ++ drivers/infiniband/ulp/qlgc_vnic/vnic_main.c | 1097 ++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_main.h | 159 + drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c | 319 +++ drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h | 77 + drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c | 109 + drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h | 80 + drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c | 234 ++ drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h | 497 ++++ drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c | 1256 +++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h | 52 drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h | 103 + drivers/infiniband/ulp/qlgc_vnic/vnic_util.h | 236 ++ drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c | 1214 +++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h | 175 ++ 28 files changed, 12237 insertions(+), 0 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic create mode 100644 drivers/infiniband/ulp/qlgc_vnic/Kconfig create mode 100644 drivers/infiniband/ulp/qlgc_vnic/Makefile create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_config.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_config.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_control.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_control.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_control_pkt.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_data.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_data.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_main.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_main.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_util.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h -- Regards, Ram From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:02:45 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:02:45 -0400 Subject: [ofa-general] [PATCH v4 01/14] QLogic VNIC: Driver - netdev implementation In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210244.11186.21580.stgit@dale> From: Ramachandra K QLogic Virtual NIC Driver. This patch implements netdev registration, netdev functions and state maintenance of the QLogic Virtual NIC corresponding to the various events associated with the QLogic Ethernet Virtual I/O Controller (EVIC/VEx) connection. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_main.c | 1097 ++++++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_main.h | 159 ++++ 2 files changed, 1256 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_main.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_main.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_main.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_main.c new file mode 100644 index 0000000..665f7dc --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_main.c @@ -0,0 +1,1097 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_main.h" +#include "vnic_netpath.h" +#include "vnic_viport.h" +#include "vnic_ib.h" +#include "vnic_stats.h" + +#define MODULEVERSION "1.3.0.0.4" +#define MODULEDETAILS \ + "QLogic Corp. Virtual NIC (VNIC) driver version " MODULEVERSION + +MODULE_AUTHOR("QLogic Corp."); +MODULE_DESCRIPTION(MODULEDETAILS); +MODULE_LICENSE("Dual BSD/GPL"); +MODULE_SUPPORTED_DEVICE("QLogic Ethernet Virtual I/O Controller"); + +u32 vnic_debug; + +module_param(vnic_debug, uint, 0444); +MODULE_PARM_DESC(vnic_debug, "Enable debug tracing if > 0"); + +LIST_HEAD(vnic_list); + +static DECLARE_WAIT_QUEUE_HEAD(vnic_npevent_queue); +static LIST_HEAD(vnic_npevent_list); +static DECLARE_COMPLETION(vnic_npevent_thread_exit); +static spinlock_t vnic_npevent_list_lock; +static struct task_struct *vnic_npevent_thread; +static int vnic_npevent_thread_end; + +static const char *const vnic_npevent_str[] = { + "PRIMARY CONNECTED", + "PRIMARY DISCONNECTED", + "PRIMARY CARRIER", + "PRIMARY NO CARRIER", + "PRIMARY TIMER EXPIRED", + "PRIMARY SETLINK", + "SECONDARY CONNECTED", + "SECONDARY DISCONNECTED", + "SECONDARY CARRIER", + "SECONDARY NO CARRIER", + "SECONDARY TIMER EXPIRED", + "SECONDARY SETLINK", + "FREE VNIC", +}; + +void vnic_connected(struct vnic *vnic, struct netpath *netpath) +{ + VNIC_FUNCTION("vnic_connected()\n"); + if (netpath->second_bias) + vnic_npevent_queue_evt(netpath, VNIC_SECNP_CONNECTED); + else + vnic_npevent_queue_evt(netpath, VNIC_PRINP_CONNECTED); + + vnic_connected_stats(vnic); +} + +void vnic_disconnected(struct vnic *vnic, struct netpath *netpath) +{ + VNIC_FUNCTION("vnic_disconnected()\n"); + if (netpath->second_bias) + vnic_npevent_queue_evt(netpath, VNIC_SECNP_DISCONNECTED); + else + vnic_npevent_queue_evt(netpath, VNIC_PRINP_DISCONNECTED); +} + +void vnic_link_up(struct vnic *vnic, struct netpath *netpath) +{ + VNIC_FUNCTION("vnic_link_up()\n"); + if (netpath->second_bias) + vnic_npevent_queue_evt(netpath, VNIC_SECNP_LINKUP); + else + vnic_npevent_queue_evt(netpath, VNIC_PRINP_LINKUP); +} + +void vnic_link_down(struct vnic *vnic, struct netpath *netpath) +{ + VNIC_FUNCTION("vnic_link_down()\n"); + if (netpath->second_bias) + vnic_npevent_queue_evt(netpath, VNIC_SECNP_LINKDOWN); + else + vnic_npevent_queue_evt(netpath, VNIC_PRINP_LINKDOWN); +} + +void vnic_stop_xmit(struct vnic *vnic, struct netpath *netpath) +{ + unsigned long flags; + + VNIC_FUNCTION("vnic_stop_xmit()\n"); + spin_lock_irqsave(&vnic->current_path_lock, flags); + if (netpath == vnic->current_path) { + if (!netif_queue_stopped(vnic->netdevice)) { + netif_stop_queue(vnic->netdevice); + vnic->failed_over = 0; + } + + vnic_stop_xmit_stats(vnic); + } + spin_unlock_irqrestore(&vnic->current_path_lock, flags); +} + +void vnic_restart_xmit(struct vnic *vnic, struct netpath *netpath) +{ + unsigned long flags; + + VNIC_FUNCTION("vnic_restart_xmit()\n"); + spin_lock_irqsave(&vnic->current_path_lock, flags); + if (netpath == vnic->current_path) { + if (netif_queue_stopped(vnic->netdevice)) + netif_wake_queue(vnic->netdevice); + + vnic_restart_xmit_stats(vnic); + } + spin_unlock_irqrestore(&vnic->current_path_lock, flags); +} + +void vnic_recv_packet(struct vnic *vnic, struct netpath *netpath, + struct sk_buff *skb) +{ + VNIC_FUNCTION("vnic_recv_packet()\n"); + if ((netpath != vnic->current_path) || + !netif_running(vnic->netdevice)) { + VNIC_INFO("tossing packet\n"); + dev_kfree_skb(skb); + return; + } + + vnic->netdevice->last_rx = jiffies; + skb->dev = vnic->netdevice; + skb->protocol = eth_type_trans(skb, skb->dev); + if (!vnic->config->use_rx_csum) + skb->ip_summed = CHECKSUM_NONE; + netif_rx(skb); + vnic_recv_pkt_stats(vnic); +} + +static struct net_device_stats *vnic_get_stats(struct net_device *device) +{ + struct vnic *vnic; + struct netpath *np; + unsigned long flags; + + VNIC_FUNCTION("vnic_get_stats()\n"); + vnic = netdev_priv(device); + + spin_lock_irqsave(&vnic->current_path_lock, flags); + np = vnic->current_path; + if (np && np->viport) { + atomic_inc(&np->viport->reference_count); + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + viport_get_stats(np->viport, &vnic->netdevice->stats); + atomic_dec(&np->viport->reference_count); + wake_up(&np->viport->reference_queue); + } else + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + + return &vnic->netdevice->stats; +} + +static int vnic_open(struct net_device *device) +{ + struct vnic *vnic; + + VNIC_FUNCTION("vnic_open()\n"); + vnic = netdev_priv(device); + + vnic_npevent_queue_evt(&vnic->primary_path, VNIC_PRINP_SETLINK); + netif_start_queue(vnic->netdevice); + + return 0; +} + +static int vnic_stop(struct net_device *device) +{ + struct vnic *vnic; + int ret = 0; + + VNIC_FUNCTION("vnic_stop()\n"); + vnic = netdev_priv(device); + netif_stop_queue(device); + vnic_npevent_queue_evt(&vnic->primary_path, VNIC_PRINP_SETLINK); + + return ret; +} + +static int vnic_hard_start_xmit(struct sk_buff *skb, + struct net_device *device) +{ + struct vnic *vnic; + struct netpath *np; + cycles_t xmit_time; + int ret = -1; + + VNIC_FUNCTION("vnic_hard_start_xmit()\n"); + vnic = netdev_priv(device); + np = vnic->current_path; + + vnic_pre_pkt_xmit_stats(&xmit_time); + + if (np && np->viport) + ret = viport_xmit_packet(np->viport, skb); + + if (ret) { + vnic_xmit_fail_stats(vnic); + dev_kfree_skb_any(skb); + vnic->netdevice->stats.tx_dropped++; + goto out; + } + + device->trans_start = jiffies; + vnic_post_pkt_xmit_stats(vnic, xmit_time); +out: + return 0; +} + +static void vnic_tx_timeout(struct net_device *device) +{ + struct vnic *vnic; + struct viport *viport = NULL; + unsigned long flags; + + VNIC_FUNCTION("vnic_tx_timeout()\n"); + vnic = netdev_priv(device); + device->trans_start = jiffies; + + spin_lock_irqsave(&vnic->current_path_lock, flags); + if (vnic->current_path && vnic->current_path->viport) { + if (vnic->failed_over) { + if (vnic->current_path == &vnic->primary_path) + viport = vnic->secondary_path.viport; + else if (vnic->current_path == &vnic->secondary_path) + viport = vnic->primary_path.viport; + } else + viport = vnic->current_path->viport; + + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + if (viport) + viport_failure(viport); + } else + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + + VNIC_ERROR("vnic_tx_timeout\n"); +} + +static void vnic_set_multicast_list(struct net_device *device) +{ + struct vnic *vnic; + unsigned long flags; + + VNIC_FUNCTION("vnic_set_multicast_list()\n"); + vnic = netdev_priv(device); + + spin_lock_irqsave(&vnic->lock, flags); + if (device->mc_count == 0) { + if (vnic->mc_list_len) { + vnic->mc_list_len = vnic->mc_count = 0; + kfree(vnic->mc_list); + } + } else { + struct dev_mc_list *mc_list = device->mc_list; + int i; + + if (device->mc_count > vnic->mc_list_len) { + if (vnic->mc_list_len) + kfree(vnic->mc_list); + vnic->mc_list_len = device->mc_count + 10; + vnic->mc_list = kmalloc(vnic->mc_list_len * + sizeof *mc_list, GFP_ATOMIC); + if (!vnic->mc_list) { + vnic->mc_list_len = vnic->mc_count = 0; + VNIC_ERROR("failed allocating mc_list\n"); + goto failure; + } + } + vnic->mc_count = device->mc_count; + for (i = 0; i < device->mc_count; i++) { + vnic->mc_list[i] = *mc_list; + vnic->mc_list[i].next = &vnic->mc_list[i + 1]; + mc_list = mc_list->next; + } + } + spin_unlock_irqrestore(&vnic->lock, flags); + + if (vnic->primary_path.viport) + viport_set_multicast(vnic->primary_path.viport, + vnic->mc_list, vnic->mc_count); + + if (vnic->secondary_path.viport) + viport_set_multicast(vnic->secondary_path.viport, + vnic->mc_list, vnic->mc_count); + + vnic_npevent_queue_evt(&vnic->primary_path, VNIC_PRINP_SETLINK); + return; +failure: + spin_unlock_irqrestore(&vnic->lock, flags); +} + +/** + * Following set of functions queues up the events for EVIC and the + * kernel thread queuing up the event might return. + */ +static int vnic_set_mac_address(struct net_device *device, void *addr) +{ + struct vnic *vnic; + struct sockaddr *sockaddr = addr; + u8 *address; + int ret = -1; + + VNIC_FUNCTION("vnic_set_mac_address()\n"); + vnic = netdev_priv(device); + + if (!is_valid_ether_addr(sockaddr->sa_data)) + return -EADDRNOTAVAIL; + + if (netif_running(device)) + return -EBUSY; + + memcpy(device->dev_addr, sockaddr->sa_data, ETH_ALEN); + address = sockaddr->sa_data; + + if (vnic->primary_path.viport) + ret = viport_set_unicast(vnic->primary_path.viport, + address); + + if (ret) + return ret; + + if (vnic->secondary_path.viport) + viport_set_unicast(vnic->secondary_path.viport, address); + + vnic->mac_set = 1; + return 0; +} + +static int vnic_change_mtu(struct net_device *device, int mtu) +{ + struct vnic *vnic; + int ret = 0; + int pri_max_mtu; + int sec_max_mtu; + + VNIC_FUNCTION("vnic_change_mtu()\n"); + vnic = netdev_priv(device); + + if (vnic->primary_path.viport) + pri_max_mtu = viport_max_mtu(vnic->primary_path.viport); + else + pri_max_mtu = MAX_PARAM_VALUE; + + if (vnic->secondary_path.viport) + sec_max_mtu = viport_max_mtu(vnic->secondary_path.viport); + else + sec_max_mtu = MAX_PARAM_VALUE; + + if ((mtu < pri_max_mtu) && (mtu < sec_max_mtu)) { + device->mtu = mtu; + vnic_npevent_queue_evt(&vnic->primary_path, + VNIC_PRINP_SETLINK); + vnic_npevent_queue_evt(&vnic->secondary_path, + VNIC_SECNP_SETLINK); + } else if (pri_max_mtu < sec_max_mtu) + printk(KERN_WARNING PFX "%s: Maximum " + "supported MTU size is %d. " + "Cannot set MTU to %d\n", + vnic->config->name, pri_max_mtu, mtu); + else + printk(KERN_WARNING PFX "%s: Maximum " + "supported MTU size is %d. " + "Cannot set MTU to %d\n", + vnic->config->name, sec_max_mtu, mtu); + + return ret; +} + +static int vnic_npevent_register(struct vnic *vnic, struct netpath *netpath) +{ + u8 *address; + int ret; + + if (!vnic->mac_set) { + /* if netpath == secondary_path, then the primary path isn't + * connected. MAC address will be set when the primary + * connects. + */ + netpath_get_hw_addr(netpath, vnic->netdevice->dev_addr); + address = vnic->netdevice->dev_addr; + + if (vnic->secondary_path.viport) + viport_set_unicast(vnic->secondary_path.viport, + address); + + vnic->mac_set = 1; + } + ret = register_netdev(vnic->netdevice); + if (ret) { + printk(KERN_ERR PFX "%s failed registering netdev " + "error %d - calling viport_failure\n", + config_viport_name(vnic->primary_path.viport->config), + ret); + vnic_free(vnic); + printk(KERN_ERR PFX "%s DELETED : register_netdev failure\n", + config_viport_name(vnic->primary_path.viport->config)); + return ret; + } + + vnic->state = VNIC_REGISTERED; + vnic->carrier = 2; /*special value to force netif_carrier_(on|off)*/ + return 0; +} + +static void vnic_npevent_dequeue_all(struct vnic *vnic) +{ + unsigned long flags; + struct vnic_npevent *npevt, *tmp; + + spin_lock_irqsave(&vnic_npevent_list_lock, flags); + if (list_empty(&vnic_npevent_list)) + goto out; + list_for_each_entry_safe(npevt, tmp, &vnic_npevent_list, + list_ptrs) { + if ((npevt->vnic == vnic)) { + list_del(&npevt->list_ptrs); + kfree(npevt); + } + } +out: + spin_unlock_irqrestore(&vnic_npevent_list_lock, flags); +} + +static void update_path_and_reconnect(struct netpath *netpath, + struct vnic *vnic) +{ + struct viport_config *config = netpath->viport->config; + int delay = 1; + + if (vnic_ib_get_path(netpath, vnic)) + return; + /* + * tell viport_connect to wait for default_no_path_timeout + * before connecting if we are retrying the same path index + * within default_no_path_timeout. + * This prevents flooding connect requests to a path (or set + * of paths) that aren't successfully connecting for some reason. + */ + if (time_after(jiffies, + (netpath->connect_time + vnic->config->no_path_timeout))) { + netpath->path_idx = config->path_idx; + netpath->connect_time = jiffies; + netpath->delay_reconnect = 0; + delay = 0; + } else if (config->path_idx != netpath->path_idx) { + delay = netpath->delay_reconnect; + netpath->path_idx = config->path_idx; + netpath->delay_reconnect = 1; + } else + delay = 1; + viport_connect(netpath->viport, delay); +} + +static inline void vnic_set_checksum_flag(struct vnic *vnic, + struct netpath *target_path) +{ + unsigned long flags; + + spin_lock_irqsave(&vnic->current_path_lock, flags); + vnic->current_path = target_path; + vnic->failed_over = 1; + if (vnic->config->use_tx_csum && + netpath_can_tx_csum(vnic->current_path)) + vnic->netdevice->features |= NETIF_F_IP_CSUM; + + spin_unlock_irqrestore(&vnic->current_path_lock, flags); +} + +static void vnic_set_uni_multicast(struct vnic *vnic, + struct netpath *netpath) +{ + unsigned long flags; + u8 *address; + + if (vnic->mac_set) { + address = vnic->netdevice->dev_addr; + + if (netpath->viport) + viport_set_unicast(netpath->viport, address); + } + spin_lock_irqsave(&vnic->lock, flags); + + if (vnic->mc_list && netpath->viport) + viport_set_multicast(netpath->viport, vnic->mc_list, + vnic->mc_count); + + spin_unlock_irqrestore(&vnic->lock, flags); + if (vnic->state == VNIC_REGISTERED) { + if (!netpath->viport) + return; + viport_set_link(netpath->viport, + vnic->netdevice->flags & ~IFF_UP, + vnic->netdevice->mtu); + } +} + +static void vnic_set_netpath_timers(struct vnic *vnic, + struct netpath *netpath) +{ + switch (netpath->timer_state) { + case NETPATH_TS_IDLE: + netpath->timer_state = NETPATH_TS_ACTIVE; + if (vnic->state == VNIC_UNINITIALIZED) + netpath_timer(netpath, + vnic->config-> + primary_connect_timeout); + else + netpath_timer(netpath, + vnic->config-> + primary_reconnect_timeout); + break; + case NETPATH_TS_ACTIVE: + /*nothing to do*/ + break; + case NETPATH_TS_EXPIRED: + if (vnic->state == VNIC_UNINITIALIZED) + vnic_npevent_register(vnic, netpath); + + break; + } +} + +static void vnic_check_primary_path_timer(struct vnic *vnic) +{ + switch (vnic->primary_path.timer_state) { + case NETPATH_TS_ACTIVE: + /* nothing to do. just wait */ + break; + case NETPATH_TS_IDLE: + netpath_timer(&vnic->primary_path, + vnic->config-> + primary_switch_timeout); + break; + case NETPATH_TS_EXPIRED: + printk(KERN_INFO PFX + "%s: switching to primary path\n", + vnic->config->name); + + vnic_set_checksum_flag(vnic, &vnic->primary_path); + break; + } +} + +static void vnic_carrier_loss(struct vnic *vnic, + struct netpath *last_path) +{ + if (vnic->primary_path.carrier) { + vnic->carrier = 1; + vnic_set_checksum_flag(vnic, &vnic->primary_path); + + if (last_path && last_path != vnic->current_path) + printk(KERN_INFO PFX + "%s: failing over to primary path\n", + vnic->config->name); + else if (!last_path) + printk(KERN_INFO PFX "%s: using primary path\n", + vnic->config->name); + + } else if ((vnic->secondary_path.carrier) && + (vnic->secondary_path.timer_state != NETPATH_TS_ACTIVE)) { + vnic->carrier = 1; + vnic_set_checksum_flag(vnic, &vnic->secondary_path); + + if (last_path && last_path != vnic->current_path) + printk(KERN_INFO PFX + "%s: failing over to secondary path\n", + vnic->config->name); + else if (!last_path) + printk(KERN_INFO PFX "%s: using secondary path\n", + vnic->config->name); + + } + +} + +static void vnic_handle_path_change(struct vnic *vnic, + struct netpath **path) +{ + struct netpath *last_path = *path; + + if (!last_path) { + if (vnic->current_path == &vnic->primary_path) + last_path = &vnic->secondary_path; + else + last_path = &vnic->primary_path; + + } + + if (vnic->current_path && vnic->current_path->viport) + viport_set_link(vnic->current_path->viport, + vnic->netdevice->flags, + vnic->netdevice->mtu); + + if (last_path->viport) + viport_set_link(last_path->viport, + vnic->netdevice->flags & + ~IFF_UP, vnic->netdevice->mtu); + + vnic_restart_xmit(vnic, vnic->current_path); +} + +static void vnic_report_path_change(struct vnic *vnic, + struct netpath *last_path, + int other_path_ok) +{ + if (!vnic->current_path) { + if (last_path == &vnic->primary_path) + printk(KERN_INFO PFX "%s: primary path lost, " + "no failover path available\n", + vnic->config->name); + else + printk(KERN_INFO PFX "%s: secondary path lost, " + "no failover path available\n", + vnic->config->name); + return; + } + + if (last_path != vnic->current_path) + return; + + if (vnic->current_path == &vnic->secondary_path) { + if (other_path_ok != vnic->primary_path.carrier) { + if (other_path_ok) + printk(KERN_INFO PFX "%s: primary path no" + " longer available for failover\n", + vnic->config->name); + else + printk(KERN_INFO PFX "%s: primary path now" + " available for failover\n", + vnic->config->name); + } + } else { + if (other_path_ok != vnic->secondary_path.carrier) { + if (other_path_ok) + printk(KERN_INFO PFX "%s: secondary path no" + " longer available for failover\n", + vnic->config->name); + else + printk(KERN_INFO PFX "%s: secondary path now" + " available for failover\n", + vnic->config->name); + } + } +} + +static void vnic_handle_free_vnic_evt(struct vnic *vnic) +{ + unsigned long flags; + + if (!netif_queue_stopped(vnic->netdevice)) + netif_stop_queue(vnic->netdevice); + + netpath_timer_stop(&vnic->primary_path); + netpath_timer_stop(&vnic->secondary_path); + spin_lock_irqsave(&vnic->current_path_lock, flags); + vnic->current_path = NULL; + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + netpath_free(&vnic->primary_path); + netpath_free(&vnic->secondary_path); + if (vnic->state == VNIC_REGISTERED) + unregister_netdev(vnic->netdevice); + + vnic_npevent_dequeue_all(vnic); + kfree(vnic->config); + if (vnic->mc_list_len) { + vnic->mc_list_len = vnic->mc_count = 0; + kfree(vnic->mc_list); + } + + sysfs_remove_group(&vnic->dev_info.dev.kobj, + &vnic_dev_attr_group); + vnic_cleanup_stats_files(vnic); + device_unregister(&vnic->dev_info.dev); + wait_for_completion(&vnic->dev_info.released); + free_netdev(vnic->netdevice); +} + +static struct vnic *vnic_handle_npevent(struct vnic *vnic, + enum vnic_npevent_type npevt_type) +{ + struct netpath *netpath; + const char *netpath_str; + + if (npevt_type <= VNIC_PRINP_LASTTYPE) + netpath_str = netpath_to_string(vnic, &vnic->primary_path); + else if (npevt_type <= VNIC_SECNP_LASTTYPE) + netpath_str = netpath_to_string(vnic, &vnic->secondary_path); + else + netpath_str = netpath_to_string(vnic, vnic->current_path); + + VNIC_INFO("%s: processing %s, netpath=%s, carrier=%d\n", + vnic->config->name, vnic_npevent_str[npevt_type], + netpath_str, vnic->carrier); + + switch (npevt_type) { + case VNIC_PRINP_CONNECTED: + netpath = &vnic->primary_path; + if (vnic->state == VNIC_UNINITIALIZED) { + if (vnic_npevent_register(vnic, netpath)) + break; + } + vnic_set_uni_multicast(vnic, netpath); + break; + case VNIC_SECNP_CONNECTED: + vnic_set_uni_multicast(vnic, &vnic->secondary_path); + break; + case VNIC_PRINP_TIMEREXPIRED: + netpath = &vnic->primary_path; + netpath->timer_state = NETPATH_TS_EXPIRED; + if (!netpath->carrier) + update_path_and_reconnect(netpath, vnic); + break; + case VNIC_SECNP_TIMEREXPIRED: + netpath = &vnic->secondary_path; + netpath->timer_state = NETPATH_TS_EXPIRED; + if (!netpath->carrier) + update_path_and_reconnect(netpath, vnic); + else { + if (vnic->state == VNIC_UNINITIALIZED) + vnic_npevent_register(vnic, netpath); + } + break; + case VNIC_PRINP_LINKUP: + vnic->primary_path.carrier = 1; + break; + case VNIC_SECNP_LINKUP: + netpath = &vnic->secondary_path; + netpath->carrier = 1; + if (!vnic->carrier) + vnic_set_netpath_timers(vnic, netpath); + break; + case VNIC_PRINP_LINKDOWN: + vnic->primary_path.carrier = 0; + break; + case VNIC_SECNP_LINKDOWN: + if (vnic->state == VNIC_UNINITIALIZED) + netpath_timer_stop(&vnic->secondary_path); + vnic->secondary_path.carrier = 0; + break; + case VNIC_PRINP_DISCONNECTED: + netpath = &vnic->primary_path; + netpath_timer_stop(netpath); + netpath->carrier = 0; + update_path_and_reconnect(netpath, vnic); + break; + case VNIC_SECNP_DISCONNECTED: + netpath = &vnic->secondary_path; + netpath_timer_stop(netpath); + netpath->carrier = 0; + update_path_and_reconnect(netpath, vnic); + break; + case VNIC_PRINP_SETLINK: + netpath = vnic->current_path; + if (!netpath || !netpath->viport) + break; + viport_set_link(netpath->viport, + vnic->netdevice->flags, + vnic->netdevice->mtu); + break; + case VNIC_SECNP_SETLINK: + netpath = &vnic->secondary_path; + if (!netpath || !netpath->viport) + break; + viport_set_link(netpath->viport, + vnic->netdevice->flags, + vnic->netdevice->mtu); + break; + case VNIC_NP_FREEVNIC: + vnic_handle_free_vnic_evt(vnic); + vnic = NULL; + break; + } + return vnic; +} + +static int vnic_npevent_statemachine(void *context) +{ + struct vnic_npevent *vnic_link_evt; + enum vnic_npevent_type npevt_type; + struct vnic *vnic; + int last_carrier; + int other_path_ok = 0; + struct netpath *last_path; + + while (!vnic_npevent_thread_end || + !list_empty(&vnic_npevent_list)) { + unsigned long flags; + + wait_event_interruptible(vnic_npevent_queue, + !list_empty(&vnic_npevent_list) + || vnic_npevent_thread_end); + spin_lock_irqsave(&vnic_npevent_list_lock, flags); + if (list_empty(&vnic_npevent_list)) { + spin_unlock_irqrestore(&vnic_npevent_list_lock, + flags); + VNIC_INFO("netpath statemachine wake" + " on empty list\n"); + continue; + } + + vnic_link_evt = list_first_entry(&vnic_npevent_list, + struct vnic_npevent, + list_ptrs); + list_del(&vnic_link_evt->list_ptrs); + spin_unlock_irqrestore(&vnic_npevent_list_lock, flags); + vnic = vnic_link_evt->vnic; + npevt_type = vnic_link_evt->event_type; + kfree(vnic_link_evt); + + if (vnic->current_path == &vnic->secondary_path) + other_path_ok = vnic->primary_path.carrier; + else if (vnic->current_path == &vnic->primary_path) + other_path_ok = vnic->secondary_path.carrier; + + vnic = vnic_handle_npevent(vnic, npevt_type); + + if (!vnic) + continue; + + last_carrier = vnic->carrier; + last_path = vnic->current_path; + + if (!vnic->current_path || + !vnic->current_path->carrier) { + vnic->carrier = 0; + vnic->current_path = NULL; + vnic->netdevice->features &= ~NETIF_F_IP_CSUM; + } + + if (!vnic->carrier) + vnic_carrier_loss(vnic, last_path); + else if ((vnic->current_path != &vnic->primary_path) && + (vnic->config->prefer_primary) && + (vnic->primary_path.carrier)) + vnic_check_primary_path_timer(vnic); + + if (last_path) + vnic_report_path_change(vnic, last_path, + other_path_ok); + + VNIC_INFO("new netpath=%s, carrier=%d\n", + netpath_to_string(vnic, vnic->current_path), + vnic->carrier); + + if (vnic->current_path != last_path) + vnic_handle_path_change(vnic, &last_path); + + if (vnic->carrier != last_carrier) { + if (vnic->carrier) { + VNIC_INFO("netif_carrier_on\n"); + netif_carrier_on(vnic->netdevice); + vnic_carrier_loss_stats(vnic); + } else { + VNIC_INFO("netif_carrier_off\n"); + netif_carrier_off(vnic->netdevice); + vnic_disconn_stats(vnic); + } + + } + } + complete_and_exit(&vnic_npevent_thread_exit, 0); + return 0; +} + +void vnic_npevent_queue_evt(struct netpath *netpath, + enum vnic_npevent_type evt) +{ + struct vnic_npevent *npevent; + unsigned long flags; + + npevent = kmalloc(sizeof *npevent, GFP_ATOMIC); + if (!npevent) { + VNIC_ERROR("Could not allocate memory for vnic event\n"); + return; + } + npevent->vnic = netpath->parent; + npevent->event_type = evt; + INIT_LIST_HEAD(&npevent->list_ptrs); + spin_lock_irqsave(&vnic_npevent_list_lock, flags); + list_add_tail(&npevent->list_ptrs, &vnic_npevent_list); + spin_unlock_irqrestore(&vnic_npevent_list_lock, flags); + wake_up(&vnic_npevent_queue); +} + +void vnic_npevent_dequeue_evt(struct netpath *netpath, + enum vnic_npevent_type evt) +{ + unsigned long flags; + struct vnic_npevent *npevt, *tmp; + struct vnic *vnic = netpath->parent; + + spin_lock_irqsave(&vnic_npevent_list_lock, flags); + if (list_empty(&vnic_npevent_list)) + goto out; + list_for_each_entry_safe(npevt, tmp, &vnic_npevent_list, + list_ptrs) { + if ((npevt->vnic == vnic) && + (npevt->event_type == evt)) { + list_del(&npevt->list_ptrs); + kfree(npevt); + break; + } + } +out: + spin_unlock_irqrestore(&vnic_npevent_list_lock, flags); +} + +static int vnic_npevent_start(void) +{ + VNIC_FUNCTION("vnic_npevent_start()\n"); + + spin_lock_init(&vnic_npevent_list_lock); + vnic_npevent_thread = kthread_run(vnic_npevent_statemachine, NULL, + "qlgc_vnic_npevent_s_m"); + if (IS_ERR(vnic_npevent_thread)) { + printk(KERN_WARNING PFX "failed to create vnic npevent" + " thread; error %d\n", + (int) PTR_ERR(vnic_npevent_thread)); + vnic_npevent_thread = NULL; + return 1; + } + + return 0; +} + +void vnic_npevent_cleanup(void) +{ + if (vnic_npevent_thread) { + vnic_npevent_thread_end = 1; + wake_up(&vnic_npevent_queue); + wait_for_completion(&vnic_npevent_thread_exit); + vnic_npevent_thread = NULL; + } +} + +static void vnic_setup(struct net_device *device) +{ + ether_setup(device); + + /* ether_setup is used to fill + * device parameters for ethernet devices. + * We override some of the parameters + * which are specific to VNIC. + */ + device->get_stats = vnic_get_stats; + device->open = vnic_open; + device->stop = vnic_stop; + device->hard_start_xmit = vnic_hard_start_xmit; + device->tx_timeout = vnic_tx_timeout; + device->set_multicast_list = vnic_set_multicast_list; + device->set_mac_address = vnic_set_mac_address; + device->change_mtu = vnic_change_mtu; + device->watchdog_timeo = 10 * HZ; + device->features = 0; +} + +struct vnic *vnic_allocate(struct vnic_config *config) +{ + struct vnic *vnic = NULL; + struct net_device *netdev; + + VNIC_FUNCTION("vnic_allocate()\n"); + netdev = alloc_netdev((int) sizeof(*vnic), config->name, vnic_setup); + if (!netdev) { + VNIC_ERROR("failed allocating vnic structure\n"); + return NULL; + } + + vnic = netdev_priv(netdev); + vnic->netdevice = netdev; + spin_lock_init(&vnic->lock); + spin_lock_init(&vnic->current_path_lock); + vnic_alloc_stats(vnic); + vnic->state = VNIC_UNINITIALIZED; + vnic->config = config; + + netpath_init(&vnic->primary_path, vnic, 0); + netpath_init(&vnic->secondary_path, vnic, 1); + + vnic->current_path = NULL; + vnic->failed_over = 0; + + list_add_tail(&vnic->list_ptrs, &vnic_list); + + return vnic; +} + +void vnic_free(struct vnic *vnic) +{ + VNIC_FUNCTION("vnic_free()\n"); + list_del(&vnic->list_ptrs); + vnic_npevent_queue_evt(&vnic->primary_path, VNIC_NP_FREEVNIC); +} + +static void __exit vnic_cleanup(void) +{ + VNIC_FUNCTION("vnic_cleanup()\n"); + + VNIC_INIT("unloading %s\n", MODULEDETAILS); + + while (!list_empty(&vnic_list)) { + struct vnic *vnic = + list_first_entry(&vnic_list, struct vnic, list_ptrs); + vnic_free(vnic); + } + + vnic_npevent_cleanup(); + viport_cleanup(); + vnic_ib_cleanup(); +} + +static int __init vnic_init(void) +{ + int ret; + VNIC_FUNCTION("vnic_init()\n"); + VNIC_INIT("Initializing %s\n", MODULEDETAILS); + + ret = config_start(); + if (ret) { + VNIC_ERROR("config_start failed\n"); + goto failure; + } + + ret = vnic_ib_init(); + if (ret) { + VNIC_ERROR("ib_start failed\n"); + goto failure; + } + + ret = viport_start(); + if (ret) { + VNIC_ERROR("viport_start failed\n"); + goto failure; + } + + ret = vnic_npevent_start(); + if (ret) { + VNIC_ERROR("vnic_npevent_start failed\n"); + goto failure; + } + + return 0; +failure: + vnic_cleanup(); + return ret; +} + +module_init(vnic_init); +module_exit(vnic_cleanup); diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_main.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_main.h new file mode 100644 index 0000000..568e045 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_main.h @@ -0,0 +1,159 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_MAIN_H_INCLUDED +#define VNIC_MAIN_H_INCLUDED + +#include +#include +#include +#include + +#include "vnic_config.h" +#include "vnic_netpath.h" + +extern u16 vnic_max_mtu; +extern struct list_head vnic_list; +extern struct attribute_group vnic_stats_attr_group; +extern cycles_t vnic_recv_ref; + +enum vnic_npevent_type { + VNIC_PRINP_CONNECTED = 0, + VNIC_PRINP_DISCONNECTED = 1, + VNIC_PRINP_LINKUP = 2, + VNIC_PRINP_LINKDOWN = 3, + VNIC_PRINP_TIMEREXPIRED = 4, + VNIC_PRINP_SETLINK = 5, + + /* used to figure out PRI vs SEC types for dbg msg*/ + VNIC_PRINP_LASTTYPE = VNIC_PRINP_SETLINK, + + VNIC_SECNP_CONNECTED = 6, + VNIC_SECNP_DISCONNECTED = 7, + VNIC_SECNP_LINKUP = 8, + VNIC_SECNP_LINKDOWN = 9, + VNIC_SECNP_TIMEREXPIRED = 10, + VNIC_SECNP_SETLINK = 11, + + /* used to figure out PRI vs SEC types for dbg msg*/ + VNIC_SECNP_LASTTYPE = VNIC_SECNP_SETLINK, + + VNIC_NP_FREEVNIC = 12, + + /* + * NOTE : If any new netpath event is being added, don't forget to + * add corresponding netpath event string into vnic_main.c. + */ +}; + +struct vnic_npevent { + struct list_head list_ptrs; + struct vnic *vnic; + enum vnic_npevent_type event_type; +}; + +void vnic_npevent_queue_evt(struct netpath *netpath, + enum vnic_npevent_type evt); +void vnic_npevent_dequeue_evt(struct netpath *netpath, + enum vnic_npevent_type evt); + +enum vnic_state { + VNIC_UNINITIALIZED = 0, + VNIC_REGISTERED = 1 +}; + +struct vnic_param { + int rx_csum; + int tx_csum; + struct vnic_path_param primary_path; + struct vnic_path_param secondary_path; +}; + +struct vnic { + struct list_head list_ptrs; + enum vnic_state state; + struct vnic_param params; + struct vnic_config *config; + struct netpath *current_path; + struct netpath primary_path; + struct netpath secondary_path; + int carrier; + int failed_over; + int mac_set; + struct net_device *netdevice; + struct dev_info dev_info; + struct dev_mc_list *mc_list; + int mc_list_len; + int mc_count; + spinlock_t lock; + spinlock_t current_path_lock; +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS + struct { + cycles_t start_time; + cycles_t conn_time; + cycles_t disconn_ref; /* intermediate time */ + cycles_t disconn_time; + u32 disconn_num; + cycles_t xmit_time; + u32 xmit_num; + u32 xmit_fail; + cycles_t recv_time; + u32 recv_num; + u32 multicast_recv_num; + cycles_t xmit_ref; /* intermediate time */ + cycles_t xmit_off_time; + u32 xmit_off_num; + cycles_t carrier_ref; /* intermediate time */ + cycles_t carrier_off_time; + u32 carrier_off_num; + } statistics; + struct dev_info stat_info; +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ +}; + +struct vnic *vnic_allocate(struct vnic_config *config); + +void vnic_free(struct vnic *vnic); + +void vnic_connected(struct vnic *vnic, struct netpath *netpath); +void vnic_disconnected(struct vnic *vnic, struct netpath *netpath); + +void vnic_link_up(struct vnic *vnic, struct netpath *netpath); +void vnic_link_down(struct vnic *vnic, struct netpath *netpath); + +void vnic_stop_xmit(struct vnic *vnic, struct netpath *netpath); +void vnic_restart_xmit(struct vnic *vnic, struct netpath *netpath); + +void vnic_recv_packet(struct vnic *vnic, struct netpath *netpath, + struct sk_buff *skb); +void vnic_npevent_cleanup(void); +#endif /* VNIC_MAIN_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:03:45 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:03:45 -0400 Subject: [ofa-general] [PATCH v4 03/14] QLogic VNIC: Implementation of communication protocol with EVIC/VEx In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210345.11186.72548.stgit@dale> From: Poornima Kamath Implementation of the statemachine for the protocol used while communicating with the EVIC. The patch also implements the viport abstraction which represents the virtual ethernet port on EVIC. Signed-off-by: Poornima Kamath Signed-off-by: Ramachandra K Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c | 1214 ++++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h | 175 +++ 2 files changed, 1389 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c new file mode 100644 index 0000000..10a002b --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.c @@ -0,0 +1,1214 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_main.h" +#include "vnic_viport.h" +#include "vnic_netpath.h" +#include "vnic_control.h" +#include "vnic_data.h" +#include "vnic_config.h" +#include "vnic_control_pkt.h" + +#define VIPORT_DISCONN_TIMER 10000 /* 10 seconds */ + +#define MAX_RETRY_INTERVAL 20000 /* 20 seconds */ +#define RETRY_INCREMENT 5000 /* 5 seconds */ +#define MAX_CONNECT_RETRY_TIMEOUT 600000 /* 10 minutes */ + +static DECLARE_WAIT_QUEUE_HEAD(viport_queue); +static LIST_HEAD(viport_list); +static DECLARE_COMPLETION(viport_thread_exit); +static spinlock_t viport_list_lock; + +static struct task_struct *viport_thread; +static int viport_thread_end; + +static void viport_timer(struct viport *viport, int timeout); + +struct viport *viport_allocate(struct viport_config *config) +{ + struct viport *viport; + + VIPORT_FUNCTION("viport_allocate()\n"); + viport = kzalloc(sizeof *viport, GFP_KERNEL); + if (!viport) { + VIPORT_ERROR("failed allocating viport structure\n"); + return NULL; + } + + viport->state = VIPORT_DISCONNECTED; + viport->link_state = LINK_FIRSTCONNECT; + viport->connect = WAIT; + viport->new_mtu = 1500; + viport->new_flags = 0; + viport->config = config; + viport->connect = DELAY; + viport->data.max_mtu = vnic_max_mtu; + spin_lock_init(&viport->lock); + init_waitqueue_head(&viport->stats_queue); + init_waitqueue_head(&viport->disconnect_queue); + init_waitqueue_head(&viport->reference_queue); + INIT_LIST_HEAD(&viport->list_ptrs); + + vnic_mc_init(viport); + + return viport; +} + +void viport_connect(struct viport *viport, int delay) +{ + VIPORT_FUNCTION("viport_connect()\n"); + + if (viport->connect != DELAY) + viport->connect = (delay) ? DELAY : NOW; + if (viport->link_state == LINK_FIRSTCONNECT) { + u32 duration; + duration = (net_random() & 0x1ff); + if (!viport->parent->is_primary_path) + duration += 0x1ff; + viport->link_state = LINK_RETRYWAIT; + viport_timer(viport, duration); + } else + viport_kick(viport); +} + +static void viport_disconnect(struct viport *viport) +{ + VIPORT_FUNCTION("viport_disconnect()\n"); + viport->disconnect = 1; + viport_failure(viport); + wait_event(viport->disconnect_queue, viport->disconnect == 0); +} + +void viport_free(struct viport *viport) +{ + VIPORT_FUNCTION("viport_free()\n"); + viport_disconnect(viport); /* NOTE: this can sleep */ + vnic_mc_uninit(viport); + kfree(viport->config); + kfree(viport); +} + +void viport_set_link(struct viport *viport, u16 flags, u16 mtu) +{ + unsigned long localflags; + int i; + + VIPORT_FUNCTION("viport_set_link()\n"); + if (mtu > data_max_mtu(&viport->data)) { + VIPORT_ERROR("configuration error." + " mtu of %d unsupported by %s\n", mtu, + config_viport_name(viport->config)); + goto failure; + } + + spin_lock_irqsave(&viport->lock, localflags); + flags &= IFF_UP | IFF_ALLMULTI | IFF_PROMISC; + if ((viport->new_flags != flags) + || (viport->new_mtu != mtu)) { + viport->new_flags = flags; + viport->new_mtu = mtu; + viport->updates |= NEED_LINK_CONFIG; + if (viport->features_supported & VNIC_FEAT_INBOUND_IB_MC) { + if (((viport->mtu <= MCAST_MSG_SIZE) && (mtu > MCAST_MSG_SIZE)) || + ((viport->mtu > MCAST_MSG_SIZE) && (mtu <= MCAST_MSG_SIZE))) { + /* + * MTU value will enable/disable the multicast. In + * either case, need to send the CMD_CONFIG_ADDRESS2 to + * EVIC. Hence, setting the NEED_ADDRESS_CONFIG flag. + */ + viport->updates |= NEED_ADDRESS_CONFIG; + if (mtu <= MCAST_MSG_SIZE) { + VIPORT_PRINT("%s: MTU changed; " + "old:%d new:%d (threshold:%d);" + " MULTICAST will be enabled.\n", + config_viport_name(viport->config), + viport->mtu, mtu, + (int)MCAST_MSG_SIZE); + } else { + VIPORT_PRINT("%s: MTU changed; " + "old:%d new:%d (threshold:%d); " + "MULTICAST will be disabled.\n", + config_viport_name(viport->config), + viport->mtu, mtu, + (int)MCAST_MSG_SIZE); + } + /* When we resend these addresses, EVIC will + * send mgid=0 back in response. So no need to + * shutoff ib_multicast. + */ + for (i = MCAST_ADDR_START; i < viport->num_mac_addresses; i++) { + if (viport->mac_addresses[i].valid) + viport->mac_addresses[i].operation = VNIC_OP_SET_ENTRY; + } + } + } + viport_kick(viport); + } + + spin_unlock_irqrestore(&viport->lock, localflags); + return; +failure: + viport_failure(viport); +} + +int viport_set_unicast(struct viport *viport, u8 *address) +{ + unsigned long flags; + int ret = -1; + VIPORT_FUNCTION("viport_set_unicast()\n"); + spin_lock_irqsave(&viport->lock, flags); + + if (!viport->mac_addresses) + goto out; + + if (memcmp(viport->mac_addresses[UNICAST_ADDR].address, + address, ETH_ALEN)) { + memcpy(viport->mac_addresses[UNICAST_ADDR].address, + address, ETH_ALEN); + viport->mac_addresses[UNICAST_ADDR].operation + = VNIC_OP_SET_ENTRY; + viport->updates |= NEED_ADDRESS_CONFIG; + viport_kick(viport); + } + ret = 0; +out: + spin_unlock_irqrestore(&viport->lock, flags); + return ret; +} + +int viport_set_multicast(struct viport *viport, + struct dev_mc_list *mc_list, int mc_count) +{ + u32 old_update_list; + int i; + int ret = -1; + unsigned long flags; + + VIPORT_FUNCTION("viport_set_multicast()\n"); + spin_lock_irqsave(&viport->lock, flags); + + if (!viport->mac_addresses) + goto out; + + old_update_list = viport->updates; + if (mc_count > viport->num_mac_addresses - MCAST_ADDR_START) + viport->updates |= NEED_LINK_CONFIG | MCAST_OVERFLOW; + else { + if (mc_count == 0) { + ret = 0; + goto out; + } + if (viport->updates & MCAST_OVERFLOW) { + viport->updates &= ~MCAST_OVERFLOW; + viport->updates |= NEED_LINK_CONFIG; + } + for (i = MCAST_ADDR_START; i < mc_count + MCAST_ADDR_START; + i++, mc_list = mc_list->next) { + if (viport->mac_addresses[i].valid && + !memcmp(viport->mac_addresses[i].address, + mc_list->dmi_addr, ETH_ALEN)) + continue; + memcpy(viport->mac_addresses[i].address, + mc_list->dmi_addr, ETH_ALEN); + viport->mac_addresses[i].valid = 1; + viport->mac_addresses[i].operation = VNIC_OP_SET_ENTRY; + } + for (; i < viport->num_mac_addresses; i++) { + if (!viport->mac_addresses[i].valid) + continue; + viport->mac_addresses[i].valid = 0; + viport->mac_addresses[i].operation = VNIC_OP_SET_ENTRY; + } + if (mc_count) + viport->updates |= NEED_ADDRESS_CONFIG; + } + + if (viport->updates != old_update_list) + viport_kick(viport); + ret = 0; +out: + spin_unlock_irqrestore(&viport->lock, flags); + return ret; +} + +static inline void viport_disable_multicast(struct viport *viport) +{ + VIPORT_INFO("turned off IB_MULTICAST\n"); + viport->config->control_config.ib_multicast = 0; + viport->config->control_config.ib_config.conn_data.features_supported &= + __constant_cpu_to_be32((u32)~VNIC_FEAT_INBOUND_IB_MC); + viport->link_state = LINK_RESET; +} + +void viport_get_stats(struct viport *viport, + struct net_device_stats *stats) +{ + unsigned long flags; + + VIPORT_FUNCTION("viport_get_stats()\n"); + /* Reference count has been already incremented indicating + * that viport structure is being used, which prevents its + * freeing when this task sleeps + */ + if (time_after(jiffies, + (viport->last_stats_time + viport->config->stats_interval))) { + + spin_lock_irqsave(&viport->lock, flags); + viport->updates |= NEED_STATS; + spin_unlock_irqrestore(&viport->lock, flags); + viport_kick(viport); + wait_event(viport->stats_queue, + !(viport->updates & NEED_STATS) + || (viport->disconnect == 1)); + + if (viport->stats.ethernet_status) + vnic_link_up(viport->vnic, viport->parent); + else + vnic_link_down(viport->vnic, viport->parent); + } + + stats->rx_packets = be64_to_cpu(viport->stats.if_in_ok); + stats->tx_packets = be64_to_cpu(viport->stats.if_out_ok); + stats->rx_bytes = be64_to_cpu(viport->stats.if_in_octets); + stats->tx_bytes = be64_to_cpu(viport->stats.if_out_octets); + stats->rx_errors = be64_to_cpu(viport->stats.if_in_errors); + stats->tx_errors = be64_to_cpu(viport->stats.if_out_errors); + stats->rx_dropped = 0; /* EIOC doesn't track */ + stats->tx_dropped = 0; /* EIOC doesn't track */ + stats->multicast = be64_to_cpu(viport->stats.if_in_nucast_pkts); + stats->collisions = 0; /* EIOC doesn't track */ +} + +int viport_xmit_packet(struct viport *viport, struct sk_buff *skb) +{ + int status = -1; + unsigned long flags; + + VIPORT_FUNCTION("viport_xmit_packet()\n"); + spin_lock_irqsave(&viport->lock, flags); + if (viport->state == VIPORT_CONNECTED) + status = data_xmit_packet(&viport->data, skb); + spin_unlock_irqrestore(&viport->lock, flags); + + return status; +} + +void viport_kick(struct viport *viport) +{ + unsigned long flags; + + VIPORT_FUNCTION("viport_kick()\n"); + spin_lock_irqsave(&viport_list_lock, flags); + if (list_empty(&viport->list_ptrs)) { + list_add_tail(&viport->list_ptrs, &viport_list); + wake_up(&viport_queue); + } + spin_unlock_irqrestore(&viport_list_lock, flags); +} + +void viport_failure(struct viport *viport) +{ + unsigned long flags; + + VIPORT_FUNCTION("viport_failure()\n"); + vnic_stop_xmit(viport->vnic, viport->parent); + spin_lock_irqsave(&viport_list_lock, flags); + viport->errored = 1; + if (list_empty(&viport->list_ptrs)) { + list_add_tail(&viport->list_ptrs, &viport_list); + wake_up(&viport_queue); + } + spin_unlock_irqrestore(&viport_list_lock, flags); +} + +static void viport_timeout(unsigned long data) +{ + struct viport *viport; + + VIPORT_FUNCTION("viport_timeout()\n"); + viport = (struct viport *)data; + viport->timer_active = 0; + viport_kick(viport); +} + +static void viport_timer(struct viport *viport, int timeout) +{ + VIPORT_FUNCTION("viport_timer()\n"); + if (viport->timer_active) + del_timer(&viport->timer); + init_timer(&viport->timer); + viport->timer.expires = jiffies + timeout; + viport->timer.data = (unsigned long)viport; + viport->timer.function = viport_timeout; + viport->timer_active = 1; + add_timer(&viport->timer); +} + +static void viport_timer_stop(struct viport *viport) +{ + VIPORT_FUNCTION("viport_timer_stop()\n"); + if (viport->timer_active) + del_timer(&viport->timer); + viport->timer_active = 0; +} + +static int viport_init_mac_addresses(struct viport *viport) +{ + struct vnic_address_op2 *temp; + unsigned long flags; + int i; + + VIPORT_FUNCTION("viport_init_mac_addresses()\n"); + i = viport->num_mac_addresses * sizeof *temp; + temp = kzalloc(viport->num_mac_addresses * sizeof *temp, + GFP_KERNEL); + if (!temp) { + VIPORT_ERROR("failed allocating MAC address table\n"); + return -ENOMEM; + } + + spin_lock_irqsave(&viport->lock, flags); + viport->mac_addresses = temp; + for (i = 0; i < viport->num_mac_addresses; i++) { + viport->mac_addresses[i].index = cpu_to_be16(i); + viport->mac_addresses[i].vlan = + cpu_to_be16(viport->default_vlan); + } + memset(viport->mac_addresses[BROADCAST_ADDR].address, + 0xFF, ETH_ALEN); + viport->mac_addresses[BROADCAST_ADDR].valid = 1; + memcpy(viport->mac_addresses[UNICAST_ADDR].address, + viport->hw_mac_address, ETH_ALEN); + viport->mac_addresses[UNICAST_ADDR].valid = 1; + + spin_unlock_irqrestore(&viport->lock, flags); + + return 0; +} + +static inline void viport_match_mac_address(struct vnic *vnic, + struct viport *viport) +{ + if (vnic && vnic->current_path && + viport == vnic->current_path->viport && + vnic->mac_set && + memcmp(vnic->netdevice->dev_addr, viport->hw_mac_address, ETH_ALEN)) { + VIPORT_ERROR("*** ERROR MAC address mismatch; " + "current = %02x:%02x:%02x:%02x:%02x:%02x " + "From EVIC = %02x:%02x:%02x:%02x:%02x:%02x\n", + vnic->netdevice->dev_addr[0], + vnic->netdevice->dev_addr[1], + vnic->netdevice->dev_addr[2], + vnic->netdevice->dev_addr[3], + vnic->netdevice->dev_addr[4], + vnic->netdevice->dev_addr[5], + viport->hw_mac_address[0], + viport->hw_mac_address[1], + viport->hw_mac_address[2], + viport->hw_mac_address[3], + viport->hw_mac_address[4], + viport->hw_mac_address[5]); + } +} + +static int viport_handle_init_states(struct viport *viport) +{ + enum link_state old_state; + + do { + switch (old_state = viport->link_state) { + case LINK_UNINITIALIZED: + LINK_STATE("state LINK_UNINITIALIZED\n"); + viport->updates = 0; + spin_lock_irq(&viport_list_lock); + list_del_init(&viport->list_ptrs); + spin_unlock_irq(&viport_list_lock); + if (atomic_read(&viport->reference_count)) { + wake_up(&viport->stats_queue); + wait_event(viport->reference_queue, + atomic_read(&viport->reference_count) == 0); + } + /* No more references to viport structure + * so it is safe to delete it by waking disconnect + * queue + */ + + viport->disconnect = 0; + wake_up(&viport->disconnect_queue); + break; + case LINK_INITIALIZE: + LINK_STATE("state LINK_INITIALIZE\n"); + viport->errored = 0; + viport->connect = WAIT; + viport->last_stats_time = 0; + if (viport->disconnect) + viport->link_state = LINK_UNINITIALIZED; + else + viport->link_state = LINK_INITIALIZECONTROL; + break; + case LINK_INITIALIZECONTROL: + LINK_STATE("state LINK_INITIALIZECONTROL\n"); + viport->pd = ib_alloc_pd(viport->config->ibdev); + if (IS_ERR(viport->pd)) + viport->link_state = LINK_DISCONNECTED; + else if (control_init(&viport->control, viport, + &viport->config->control_config, + viport->pd)) { + ib_dealloc_pd(viport->pd); + viport->link_state = LINK_DISCONNECTED; + + } else + viport->link_state = LINK_INITIALIZEDATA; + break; + case LINK_INITIALIZEDATA: + LINK_STATE("state LINK_INITIALIZEDATA\n"); + if (data_init(&viport->data, viport, + &viport->config->data_config, + viport->pd)) + viport->link_state = LINK_CLEANUPCONTROL; + else + viport->link_state = LINK_CONTROLCONNECT; + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_control_states(struct viport *viport) +{ + enum link_state old_state; + struct vnic *vnic; + + do { + switch (old_state = viport->link_state) { + case LINK_CONTROLCONNECT: + if (vnic_ib_cm_connect(&viport->control.ib_conn)) + viport->link_state = LINK_CLEANUPDATA; + else + viport->link_state = LINK_CONTROLCONNECTWAIT; + break; + case LINK_CONTROLCONNECTWAIT: + LINK_STATE("state LINK_CONTROLCONNECTWAIT\n"); + if (control_is_connected(&viport->control)) + viport->link_state = LINK_INITVNICREQ; + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_CONTROLDISCONNECT; + } + break; + case LINK_INITVNICREQ: + LINK_STATE("state LINK_INITVNICREQ\n"); + if (control_init_vnic_req(&viport->control)) + viport->link_state = LINK_RESETCONTROL; + else + viport->link_state = LINK_INITVNICRSP; + break; + case LINK_INITVNICRSP: + LINK_STATE("state LINK_INITVNICRSP\n"); + control_process_async(&viport->control); + + if (!control_init_vnic_rsp(&viport->control, + &viport->features_supported, + viport->hw_mac_address, + &viport->num_mac_addresses, + &viport->default_vlan)) { + if (viport_init_mac_addresses(viport)) + viport->link_state = + LINK_RESETCONTROL; + else { + viport->link_state = + LINK_BEGINDATAPATH; + /* + * Ensure that the current path's MAC + * address matches the one returned by + * EVIC - we've had cases of mismatch + * which then caused havoc. + */ + vnic = viport->parent->parent; + viport_match_mac_address(vnic, viport); + } + } + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESETCONTROL; + } + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_data_states(struct viport *viport) +{ + enum link_state old_state; + + do { + switch (old_state = viport->link_state) { + case LINK_BEGINDATAPATH: + LINK_STATE("state LINK_BEGINDATAPATH\n"); + viport->link_state = LINK_CONFIGDATAPATHREQ; + break; + case LINK_CONFIGDATAPATHREQ: + LINK_STATE("state LINK_CONFIGDATAPATHREQ\n"); + if (control_config_data_path_req(&viport->control, + data_path_id(&viport-> + data), + data_host_pool_max + (&viport->data), + data_eioc_pool_max + (&viport->data))) + viport->link_state = LINK_RESETCONTROL; + else + viport->link_state = LINK_CONFIGDATAPATHRSP; + break; + case LINK_CONFIGDATAPATHRSP: + LINK_STATE("state LINK_CONFIGDATAPATHRSP\n"); + control_process_async(&viport->control); + + if (!control_config_data_path_rsp(&viport->control, + data_host_pool + (&viport->data), + data_eioc_pool + (&viport->data), + data_host_pool_max + (&viport->data), + data_eioc_pool_max + (&viport->data), + data_host_pool_min + (&viport->data), + data_eioc_pool_min + (&viport->data))) + viport->link_state = LINK_DATACONNECT; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESETCONTROL; + } + break; + case LINK_DATACONNECT: + LINK_STATE("state LINK_DATACONNECT\n"); + if (data_connect(&viport->data)) + viport->link_state = LINK_RESETCONTROL; + else + viport->link_state = LINK_DATACONNECTWAIT; + break; + case LINK_DATACONNECTWAIT: + LINK_STATE("state LINK_DATACONNECTWAIT\n"); + control_process_async(&viport->control); + if (data_is_connected(&viport->data)) + viport->link_state = LINK_XCHGPOOLREQ; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESET; + } + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_xchgpool_states(struct viport *viport) +{ + enum link_state old_state; + + do { + switch (old_state = viport->link_state) { + case LINK_XCHGPOOLREQ: + LINK_STATE("state LINK_XCHGPOOLREQ\n"); + if (control_exchange_pools_req(&viport->control, + data_local_pool_addr + (&viport->data), + data_local_pool_rkey + (&viport->data))) + viport->link_state = LINK_RESET; + else + viport->link_state = LINK_XCHGPOOLRSP; + break; + case LINK_XCHGPOOLRSP: + LINK_STATE("state LINK_XCHGPOOLRSP\n"); + control_process_async(&viport->control); + + if (!control_exchange_pools_rsp(&viport->control, + data_remote_pool_addr + (&viport->data), + data_remote_pool_rkey + (&viport->data))) + viport->link_state = LINK_INITIALIZED; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESET; + } + break; + case LINK_INITIALIZED: + LINK_STATE("state LINK_INITIALIZED\n"); + viport->state = VIPORT_CONNECTED; + printk(KERN_INFO PFX + "%s: connection established\n", + config_viport_name(viport->config)); + data_connected(&viport->data); + vnic_connected(viport->parent->parent, + viport->parent); + if (viport->features_supported & VNIC_FEAT_INBOUND_IB_MC) { + printk(KERN_INFO PFX "%s: Supports Inbound IB " + "Multicast\n", + config_viport_name(viport->config)); + if (mc_data_init(&viport->mc_data, viport, + &viport->config->data_config, + viport->pd)) { + viport_disable_multicast(viport); + break; + } + } + spin_lock_irq(&viport->lock); + viport->mtu = 1500; + viport->flags = 0; + if ((viport->mtu != viport->new_mtu) || + (viport->flags != viport->new_flags)) + viport->updates |= NEED_LINK_CONFIG; + spin_unlock_irq(&viport->lock); + viport->link_state = LINK_IDLE; + viport->retry_duration = 0; + viport->total_retry_duration = 0; + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_idle_states(struct viport *viport) +{ + enum link_state old_state; + int handle_mc_join_compl, handle_mc_join; + + do { + switch (old_state = viport->link_state) { + case LINK_IDLE: + LINK_STATE("state LINK_IDLE\n"); + if (viport->config->hb_interval) + viport_timer(viport, + viport->config->hb_interval); + viport->link_state = LINK_IDLING; + break; + case LINK_IDLING: + LINK_STATE("state LINK_IDLING\n"); + control_process_async(&viport->control); + if (viport->errored) { + viport_timer_stop(viport); + viport->errored = 0; + viport->link_state = LINK_RESET; + break; + } + + spin_lock_irq(&viport->lock); + handle_mc_join = (viport->updates & NEED_MCAST_JOIN); + handle_mc_join_compl = + (viport->updates & NEED_MCAST_COMPLETION); + /* + * Turn off both flags, the handler functions will + * rearm them if necessary. + */ + viport->updates &= ~(NEED_MCAST_JOIN | NEED_MCAST_COMPLETION); + + if (viport->updates & NEED_LINK_CONFIG) { + viport_timer_stop(viport); + viport->link_state = LINK_CONFIGLINKREQ; + } else if (viport->updates & NEED_ADDRESS_CONFIG) { + viport_timer_stop(viport); + viport->link_state = LINK_CONFIGADDRSREQ; + } else if (viport->updates & NEED_STATS) { + viport_timer_stop(viport); + viport->link_state = LINK_REPORTSTATREQ; + } else if (viport->config->hb_interval) { + if (!viport->timer_active) + viport->link_state = + LINK_HEARTBEATREQ; + } + spin_unlock_irq(&viport->lock); + if (handle_mc_join) { + if (vnic_mc_join(viport)) + viport_disable_multicast(viport); + } + if (handle_mc_join_compl) + vnic_mc_join_handle_completion(viport); + + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_config_states(struct viport *viport) +{ + enum link_state old_state; + int res; + + do { + switch (old_state = viport->link_state) { + case LINK_CONFIGLINKREQ: + LINK_STATE("state LINK_CONFIGLINKREQ\n"); + spin_lock_irq(&viport->lock); + viport->updates &= ~NEED_LINK_CONFIG; + viport->flags = viport->new_flags; + if (viport->updates & MCAST_OVERFLOW) + viport->flags |= IFF_ALLMULTI; + viport->mtu = viport->new_mtu; + spin_unlock_irq(&viport->lock); + if (control_config_link_req(&viport->control, + viport->flags, + viport->mtu)) + viport->link_state = LINK_RESET; + else + viport->link_state = LINK_CONFIGLINKRSP; + break; + case LINK_CONFIGLINKRSP: + LINK_STATE("state LINK_CONFIGLINKRSP\n"); + control_process_async(&viport->control); + + if (!control_config_link_rsp(&viport->control, + &viport->flags, + &viport->mtu)) + viport->link_state = LINK_IDLE; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESET; + } + break; + case LINK_CONFIGADDRSREQ: + LINK_STATE("state LINK_CONFIGADDRSREQ\n"); + + spin_lock_irq(&viport->lock); + res = control_config_addrs_req(&viport->control, + viport->mac_addresses, + viport-> + num_mac_addresses); + + if (res > 0) { + viport->updates &= ~NEED_ADDRESS_CONFIG; + viport->link_state = LINK_CONFIGADDRSRSP; + } else if (res == 0) + viport->link_state = LINK_CONFIGADDRSRSP; + else + viport->link_state = LINK_RESET; + spin_unlock_irq(&viport->lock); + break; + case LINK_CONFIGADDRSRSP: + LINK_STATE("state LINK_CONFIGADDRSRSP\n"); + control_process_async(&viport->control); + + if (!control_config_addrs_rsp(&viport->control)) + viport->link_state = LINK_IDLE; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESET; + } + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_stat_states(struct viport *viport) +{ + enum link_state old_state; + + do { + switch (old_state = viport->link_state) { + case LINK_REPORTSTATREQ: + LINK_STATE("state LINK_REPORTSTATREQ\n"); + if (control_report_statistics_req(&viport->control)) + viport->link_state = LINK_RESET; + else + viport->link_state = LINK_REPORTSTATRSP; + break; + case LINK_REPORTSTATRSP: + LINK_STATE("state LINK_REPORTSTATRSP\n"); + control_process_async(&viport->control); + + spin_lock_irq(&viport->lock); + if (control_report_statistics_rsp(&viport->control, + &viport->stats) == 0) { + viport->updates &= ~NEED_STATS; + viport->last_stats_time = jiffies; + wake_up(&viport->stats_queue); + viport->link_state = LINK_IDLE; + } + + spin_unlock_irq(&viport->lock); + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESET; + } + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_heartbeat_states(struct viport *viport) +{ + enum link_state old_state; + + do { + switch (old_state = viport->link_state) { + case LINK_HEARTBEATREQ: + LINK_STATE("state LINK_HEARTBEATREQ\n"); + if (control_heartbeat_req(&viport->control, + viport->config->hb_timeout)) + viport->link_state = LINK_RESET; + else + viport->link_state = LINK_HEARTBEATRSP; + break; + case LINK_HEARTBEATRSP: + LINK_STATE("state LINK_HEARTBEATRSP\n"); + control_process_async(&viport->control); + + if (!control_heartbeat_rsp(&viport->control)) + viport->link_state = LINK_IDLE; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_RESET; + } + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_reset_states(struct viport *viport) +{ + enum link_state old_state; + int handle_mc_join_compl = 0, handle_mc_join = 0; + + do { + switch (old_state = viport->link_state) { + case LINK_RESET: + LINK_STATE("state LINK_RESET\n"); + viport->errored = 0; + spin_lock_irq(&viport->lock); + viport->state = VIPORT_DISCONNECTED; + /* + * Turn off both flags, the handler functions will + * rearm them if necessary + */ + viport->updates &= ~(NEED_MCAST_JOIN | NEED_MCAST_COMPLETION); + + spin_unlock_irq(&viport->lock); + vnic_link_down(viport->vnic, viport->parent); + printk(KERN_INFO PFX + "%s: connection lost\n", + config_viport_name(viport->config)); + if (handle_mc_join) { + if (vnic_mc_join(viport)) + viport_disable_multicast(viport); + } + if (handle_mc_join_compl) + vnic_mc_join_handle_completion(viport); + if (viport->features_supported & VNIC_FEAT_INBOUND_IB_MC) { + vnic_mc_leave(viport); + vnic_mc_data_cleanup(&viport->mc_data); + } + + if (control_reset_req(&viport->control)) + viport->link_state = LINK_DATADISCONNECT; + else + viport->link_state = LINK_RESETRSP; + break; + case LINK_RESETRSP: + LINK_STATE("state LINK_RESETRSP\n"); + control_process_async(&viport->control); + + if (!control_reset_rsp(&viport->control)) + viport->link_state = LINK_DATADISCONNECT; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_DATADISCONNECT; + } + break; + case LINK_RESETCONTROL: + LINK_STATE("state LINK_RESETCONTROL\n"); + if (control_reset_req(&viport->control)) + viport->link_state = LINK_CONTROLDISCONNECT; + else + viport->link_state = LINK_RESETCONTROLRSP; + break; + case LINK_RESETCONTROLRSP: + LINK_STATE("state LINK_RESETCONTROLRSP\n"); + control_process_async(&viport->control); + + if (!control_reset_rsp(&viport->control)) + viport->link_state = LINK_CONTROLDISCONNECT; + + if (viport->errored) { + viport->errored = 0; + viport->link_state = LINK_CONTROLDISCONNECT; + } + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_handle_disconn_states(struct viport *viport) +{ + enum link_state old_state; + + do { + switch (old_state = viport->link_state) { + case LINK_DATADISCONNECT: + LINK_STATE("state LINK_DATADISCONNECT\n"); + data_disconnect(&viport->data); + viport->link_state = LINK_CONTROLDISCONNECT; + break; + case LINK_CONTROLDISCONNECT: + LINK_STATE("state LINK_CONTROLDISCONNECT\n"); + viport->link_state = LINK_CLEANUPDATA; + break; + case LINK_CLEANUPDATA: + LINK_STATE("state LINK_CLEANUPDATA\n"); + data_cleanup(&viport->data); + viport->link_state = LINK_CLEANUPCONTROL; + break; + case LINK_CLEANUPCONTROL: + LINK_STATE("state LINK_CLEANUPCONTROL\n"); + spin_lock_irq(&viport->lock); + kfree(viport->mac_addresses); + viport->mac_addresses = NULL; + spin_unlock_irq(&viport->lock); + control_cleanup(&viport->control); + ib_dealloc_pd(viport->pd); + viport->link_state = LINK_DISCONNECTED; + break; + case LINK_DISCONNECTED: + LINK_STATE("state LINK_DISCONNECTED\n"); + vnic_disconnected(viport->parent->parent, + viport->parent); + if (viport->disconnect != 0) + viport->link_state = LINK_UNINITIALIZED; + else if (viport->retry == 1) { + viport->retry = 0; + /* + * Check if the initial retry interval has crossed + * 20 seconds. + * The retry interval is initially 5 seconds which + * is incremented by 5. Once it is 20 the interval + * is fixed to 20 seconds till 10 minutes, + * after which retrying is stopped + */ + if (viport->retry_duration < MAX_RETRY_INTERVAL) + viport->retry_duration += + RETRY_INCREMENT; + + viport->total_retry_duration += + viport->retry_duration; + + if (viport->total_retry_duration >= + MAX_CONNECT_RETRY_TIMEOUT) { + viport->link_state = LINK_UNINITIALIZED; + printk("Timed out after retrying" + " for retry_duration %d msecs\n" + , viport->total_retry_duration); + } else { + viport->connect = DELAY; + viport->link_state = LINK_RETRYWAIT; + } + viport_timer(viport, + msecs_to_jiffies(viport->retry_duration)); + } else { + u32 duration = 5000 + ((net_random()) & 0x1FF); + if (!viport->parent->is_primary_path) + duration += 0x1ff; + viport_timer(viport, + msecs_to_jiffies(duration)); + viport->connect = DELAY; + viport->link_state = LINK_RETRYWAIT; + } + break; + case LINK_RETRYWAIT: + LINK_STATE("state LINK_RETRYWAIT\n"); + viport->stats.ethernet_status = 0; + viport->updates = 0; + wake_up(&viport->stats_queue); + if (viport->disconnect != 0) { + viport_timer_stop(viport); + viport->link_state = LINK_UNINITIALIZED; + } else if (viport->connect == DELAY) { + if (!viport->timer_active) + viport->link_state = LINK_INITIALIZE; + } else if (viport->connect == NOW) { + viport_timer_stop(viport); + viport->link_state = LINK_INITIALIZE; + } + break; + case LINK_FIRSTCONNECT: + viport->stats.ethernet_status = 0; + viport->updates = 0; + wake_up(&viport->stats_queue); + if (viport->disconnect != 0) { + viport_timer_stop(viport); + viport->link_state = LINK_UNINITIALIZED; + } + + break; + default: + return -1; + } + } while (viport->link_state != old_state); + + return 0; +} + +static int viport_statemachine(void *context) +{ + struct viport *viport; + enum link_state old_link_state; + + VIPORT_FUNCTION("viport_statemachine()\n"); + while (!viport_thread_end || !list_empty(&viport_list)) { + wait_event_interruptible(viport_queue, + !list_empty(&viport_list) + || viport_thread_end); + spin_lock_irq(&viport_list_lock); + if (list_empty(&viport_list)) { + spin_unlock_irq(&viport_list_lock); + continue; + } + viport = list_first_entry(&viport_list, struct viport, + list_ptrs); + list_del_init(&viport->list_ptrs); + spin_unlock_irq(&viport_list_lock); + + do { + old_link_state = viport->link_state; + + /* + * Optimize for the state machine steady state + * by checking for the most common states first. + * + */ + if (viport_handle_idle_states(viport) == 0) + break; + if (viport_handle_heartbeat_states(viport) == 0) + break; + if (viport_handle_stat_states(viport) == 0) + break; + if (viport_handle_config_states(viport) == 0) + break; + + if (viport_handle_init_states(viport) == 0) + break; + if (viport_handle_control_states(viport) == 0) + break; + if (viport_handle_data_states(viport) == 0) + break; + if (viport_handle_xchgpool_states(viport) == 0) + break; + if (viport_handle_reset_states(viport) == 0) + break; + if (viport_handle_disconn_states(viport) == 0) + break; + } while (viport->link_state != old_link_state); + } + + complete_and_exit(&viport_thread_exit, 0); +} + +int viport_start(void) +{ + VIPORT_FUNCTION("viport_start()\n"); + + spin_lock_init(&viport_list_lock); + viport_thread = kthread_run(viport_statemachine, NULL, + "qlgc_vnic_viport_s_m"); + if (IS_ERR(viport_thread)) { + printk(KERN_WARNING PFX "Could not create viport_thread;" + " error %d\n", (int) PTR_ERR(viport_thread)); + viport_thread = NULL; + return 1; + } + + return 0; +} + +void viport_cleanup(void) +{ + VIPORT_FUNCTION("viport_cleanup()\n"); + if (viport_thread) { + viport_thread_end = 1; + wake_up(&viport_queue); + wait_for_completion(&viport_thread_exit); + viport_thread = NULL; + } +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h new file mode 100644 index 0000000..70cdc9f --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_viport.h @@ -0,0 +1,175 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_VIPORT_H_INCLUDED +#define VNIC_VIPORT_H_INCLUDED + +#include "vnic_control.h" +#include "vnic_data.h" +#include "vnic_multicast.h" + +enum viport_state { + VIPORT_DISCONNECTED = 0, + VIPORT_CONNECTED = 1 +}; + +enum link_state { + LINK_UNINITIALIZED = 0, + LINK_INITIALIZE = 1, + LINK_INITIALIZECONTROL = 2, + LINK_INITIALIZEDATA = 3, + LINK_CONTROLCONNECT = 4, + LINK_CONTROLCONNECTWAIT = 5, + LINK_INITVNICREQ = 6, + LINK_INITVNICRSP = 7, + LINK_BEGINDATAPATH = 8, + LINK_CONFIGDATAPATHREQ = 9, + LINK_CONFIGDATAPATHRSP = 10, + LINK_DATACONNECT = 11, + LINK_DATACONNECTWAIT = 12, + LINK_XCHGPOOLREQ = 13, + LINK_XCHGPOOLRSP = 14, + LINK_INITIALIZED = 15, + LINK_IDLE = 16, + LINK_IDLING = 17, + LINK_CONFIGLINKREQ = 18, + LINK_CONFIGLINKRSP = 19, + LINK_CONFIGADDRSREQ = 20, + LINK_CONFIGADDRSRSP = 21, + LINK_REPORTSTATREQ = 22, + LINK_REPORTSTATRSP = 23, + LINK_HEARTBEATREQ = 24, + LINK_HEARTBEATRSP = 25, + LINK_RESET = 26, + LINK_RESETRSP = 27, + LINK_RESETCONTROL = 28, + LINK_RESETCONTROLRSP = 29, + LINK_DATADISCONNECT = 30, + LINK_CONTROLDISCONNECT = 31, + LINK_CLEANUPDATA = 32, + LINK_CLEANUPCONTROL = 33, + LINK_DISCONNECTED = 34, + LINK_RETRYWAIT = 35, + LINK_FIRSTCONNECT = 36 +}; + +enum { + BROADCAST_ADDR = 0, + UNICAST_ADDR = 1, + MCAST_ADDR_START = 2 +}; + +#define current_mac_address mac_addresses[UNICAST_ADDR].address + +enum { + NEED_STATS = 0x00000001, + NEED_ADDRESS_CONFIG = 0x00000002, + NEED_LINK_CONFIG = 0x00000004, + MCAST_OVERFLOW = 0x00000008, + NEED_MCAST_COMPLETION = 0x00000010, + NEED_MCAST_JOIN = 0x00000020 +}; + +struct viport { + struct list_head list_ptrs; + struct netpath *parent; + struct vnic *vnic; + struct viport_config *config; + struct control control; + struct data data; + spinlock_t lock; + struct ib_pd *pd; + enum viport_state state; + enum link_state link_state; + struct vnic_cmd_report_stats_rsp stats; + wait_queue_head_t stats_queue; + unsigned long last_stats_time; + u32 features_supported; + u8 hw_mac_address[ETH_ALEN]; + u16 default_vlan; + u16 num_mac_addresses; + struct vnic_address_op2 *mac_addresses; + u32 updates; + u16 flags; + u16 new_flags; + u16 mtu; + u16 new_mtu; + u32 errored; + enum { WAIT, DELAY, NOW } connect; + u32 disconnect; + u32 retry; + wait_queue_head_t disconnect_queue; + int timer_active; + struct timer_list timer; + u32 retry_duration; + u32 total_retry_duration; + atomic_t reference_count; + wait_queue_head_t reference_queue; + struct mc_info mc_info; + struct mc_data mc_data; +}; + +int viport_start(void); +void viport_cleanup(void); + +struct viport *viport_allocate(struct viport_config *config); +void viport_free(struct viport *viport); + +void viport_connect(struct viport *viport, int delay); + +void viport_set_link(struct viport *viport, u16 flags, u16 mtu); +void viport_get_stats(struct viport *viport, + struct net_device_stats *stats); +int viport_xmit_packet(struct viport *viport, struct sk_buff *skb); +void viport_kick(struct viport *viport); + +void viport_failure(struct viport *viport); + +int viport_set_unicast(struct viport *viport, u8 *address); +int viport_set_multicast(struct viport *viport, + struct dev_mc_list *mc_list, + int mc_count); + +#define viport_max_mtu(viport) data_max_mtu(&(viport)->data) + +#define viport_get_hw_addr(viport, address) \ + memcpy(address, (viport)->hw_mac_address, ETH_ALEN) + +#define viport_features(viport) ((viport)->features_supported) + +#define viport_can_tx_csum(viport) \ + (((viport)->features_supported & \ + (VNIC_FEAT_IPV4_CSUM_TX | VNIC_FEAT_TCP_CSUM_TX | \ + VNIC_FEAT_UDP_CSUM_TX)) == (VNIC_FEAT_IPV4_CSUM_TX | \ + VNIC_FEAT_TCP_CSUM_TX | VNIC_FEAT_UDP_CSUM_TX)) + +#endif /* VNIC_VIPORT_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:03:15 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:03:15 -0400 Subject: [ofa-general] [PATCH v4 02/14] QLogic VNIC: Netpath - abstraction of connection to EVIC/VEx In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210315.11186.97777.stgit@dale> From: Ramachandra K This patch implements the netpath layer of QLogic VNIC. Netpath is an abstraction of a connection to EVIC. It primarily includes the implementation which maintains the timers to monitor the status of the connection to EVIC/VEx. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c | 109 +++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h | 80 +++++++++++++++++ 2 files changed, 189 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c new file mode 100644 index 0000000..273031c --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.c @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include + +#include "vnic_util.h" +#include "vnic_main.h" +#include "vnic_viport.h" +#include "vnic_netpath.h" + +static void vnic_npevent_timeout(unsigned long data) +{ + struct netpath *netpath = (struct netpath *)data; + + if (netpath->second_bias) + vnic_npevent_queue_evt(netpath, VNIC_SECNP_TIMEREXPIRED); + else + vnic_npevent_queue_evt(netpath, VNIC_PRINP_TIMEREXPIRED); +} + +void netpath_timer(struct netpath *netpath, int timeout) +{ + if (netpath->timer_state == NETPATH_TS_ACTIVE) + del_timer_sync(&netpath->timer); + if (timeout) { + init_timer(&netpath->timer); + netpath->timer_state = NETPATH_TS_ACTIVE; + netpath->timer.expires = jiffies + timeout; + netpath->timer.data = (unsigned long)netpath; + netpath->timer.function = vnic_npevent_timeout; + add_timer(&netpath->timer); + } else + vnic_npevent_timeout((unsigned long)netpath); +} + +void netpath_timer_stop(struct netpath *netpath) +{ + if (netpath->timer_state != NETPATH_TS_ACTIVE) + return; + del_timer_sync(&netpath->timer); + if (netpath->second_bias) + vnic_npevent_dequeue_evt(netpath, VNIC_SECNP_TIMEREXPIRED); + else + vnic_npevent_dequeue_evt(netpath, VNIC_PRINP_TIMEREXPIRED); + + netpath->timer_state = NETPATH_TS_IDLE; +} + +void netpath_free(struct netpath *netpath) +{ + vnic_cleanup_path_files(netpath); + if (!netpath->viport) + return; + viport_free(netpath->viport); + netpath->viport = NULL; +} + +void netpath_init(struct netpath *netpath, struct vnic *vnic, + int second_bias) +{ + netpath->parent = vnic; + netpath->carrier = 0; + netpath->viport = NULL; + netpath->second_bias = second_bias; + netpath->timer_state = NETPATH_TS_IDLE; + init_timer(&netpath->timer); +} + +const char *netpath_to_string(struct vnic *vnic, struct netpath *netpath) +{ + if (!netpath) + return "NULL"; + else if (netpath == &vnic->primary_path) + return "PRIMARY"; + else if (netpath == &vnic->secondary_path) + return "SECONDARY"; + else + return "UNKNOWN"; +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h new file mode 100644 index 0000000..0aa3384 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_netpath.h @@ -0,0 +1,80 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_NETPATH_H_INCLUDED +#define VNIC_NETPATH_H_INCLUDED + +#include + +#include "vnic_sys.h" + +struct viport; +struct vnic; + +enum netpath_ts { + NETPATH_TS_IDLE = 0, + NETPATH_TS_ACTIVE = 1, + NETPATH_TS_EXPIRED = 2 +}; + +struct netpath { + int carrier; + struct vnic *parent; + struct viport *viport; + size_t path_idx; + unsigned long connect_time; + int second_bias; + u8 is_primary_path; + u8 delay_reconnect; + struct timer_list timer; + enum netpath_ts timer_state; + struct dev_info dev_info; +}; + +void netpath_init(struct netpath *netpath, struct vnic *vnic, + int second_bias); +void netpath_free(struct netpath *netpath); +void vnic_cleanup_path_files(struct netpath *path); + +void netpath_timer(struct netpath *netpath, int timeout); +void netpath_timer_stop(struct netpath *netpath); + +const char *netpath_to_string(struct vnic *vnic, struct netpath *netpath); + +#define netpath_get_hw_addr(netpath, address) \ + viport_get_hw_addr((netpath)->viport, address) +#define netpath_is_connected(netpath) \ + (netpath->state == NETPATH_CONNECTED) +#define netpath_can_tx_csum(netpath) \ + viport_can_tx_csum(netpath->viport) + +#endif /* VNIC_NETPATH_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:04:15 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:04:15 -0400 Subject: [ofa-general] [PATCH v4 04/14] QLogic VNIC: Implementation of Control path of communication protocol In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210415.11186.95157.stgit@dale> From: Poornima Kamath This patch adds the files that define the control packet formats and implements various control messages that are exchanged as part of the communication protocol with the EVIC/VEx. Signed-off-by: Poornima Kamath Signed-off-by: Ramachandra K Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_control.c | 2286 ++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_control.h | 180 ++ .../infiniband/ulp/qlgc_vnic/vnic_control_pkt.h | 368 +++ 3 files changed, 2834 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_control.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_control.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_control_pkt.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_control.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_control.c new file mode 100644 index 0000000..df4d2a8 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_control.c @@ -0,0 +1,2286 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_main.h" +#include "vnic_viport.h" +#include "vnic_stats.h" + +#define vnic_multicast_address(rsp2_address, index) \ + ((rsp2_address)->list_address_ops[index].address[0] & 0x01) + +static void control_log_control_packet(struct vnic_control_packet *pkt); + +char *control_ifcfg_name(struct control *control) +{ + if (!control) + return "nctl"; + if (!control->parent) + return "np"; + if (!control->parent->parent) + return "npp"; + if (!control->parent->parent->parent) + return "nppp"; + if (!control->parent->parent->parent->config) + return "npppc"; + return control->parent->parent->parent->config->name; +} + +static void control_recv(struct control *control, struct recv_io *recv_io) +{ + if (vnic_ib_post_recv(&control->ib_conn, &recv_io->io)) + viport_failure(control->parent); +} + +static void control_recv_complete(struct io *io) +{ + struct recv_io *recv_io = (struct recv_io *)io; + struct recv_io *last_recv_io; + struct control *control = &io->viport->control; + struct vnic_control_packet *pkt = control_packet(recv_io); + struct vnic_control_header *c_hdr = &pkt->hdr; + unsigned long flags; + cycles_t response_time; + + CONTROL_FUNCTION("%s: control_recv_complete() State=%d\n", + control_ifcfg_name(control), control->req_state); + + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + control_note_rsptime_stats(&response_time); + CONTROL_PACKET(pkt); + spin_lock_irqsave(&control->io_lock, flags); + if (c_hdr->pkt_type == TYPE_INFO) { + last_recv_io = control->info; + control->info = recv_io; + spin_unlock_irqrestore(&control->io_lock, flags); + viport_kick(control->parent); + if (last_recv_io) + control_recv(control, last_recv_io); + } else if (c_hdr->pkt_type == TYPE_RSP) { + u8 repost = 0; + u8 fail = 0; + u8 kick = 0; + + switch (control->req_state) { + case REQ_INACTIVE: + case RSP_RECEIVED: + case REQ_COMPLETED: + CONTROL_ERROR("%s: Unexpected control" + "response received: CMD = %d\n", + control_ifcfg_name(control), + c_hdr->pkt_cmd); + control_log_control_packet(pkt); + control->req_state = REQ_FAILED; + fail = 1; + break; + case REQ_POSTED: + case REQ_SENT: + if (c_hdr->pkt_cmd != control->last_cmd + || c_hdr->pkt_seq_num != control->seq_num) { + CONTROL_ERROR("%s: Incorrect Control Response " + "received\n", + control_ifcfg_name(control)); + CONTROL_ERROR("%s: Sent control request:\n", + control_ifcfg_name(control)); + control_log_control_packet(control_last_req(control)); + CONTROL_ERROR("%s: Received control response:\n", + control_ifcfg_name(control)); + control_log_control_packet(pkt); + control->req_state = REQ_FAILED; + fail = 1; + } else { + control->response = recv_io; + control_update_rsptime_stats(control, + response_time); + if (control->req_state == REQ_POSTED) { + CONTROL_INFO("%s: Recv CMD RSP %d" + "before Send Completion\n", + control_ifcfg_name(control), + c_hdr->pkt_cmd); + control->req_state = RSP_RECEIVED; + } else { + control->req_state = REQ_COMPLETED; + kick = 1; + } + } + break; + case REQ_FAILED: + /* stay in REQ_FAILED state */ + repost = 1; + break; + } + spin_unlock_irqrestore(&control->io_lock, flags); + /* we must do this outside the lock*/ + if (kick) + viport_kick(control->parent); + if (repost || fail) { + control_recv(control, recv_io); + if (fail) + viport_failure(control->parent); + } + + } else { + list_add_tail(&recv_io->io.list_ptrs, + &control->failure_list); + spin_unlock_irqrestore(&control->io_lock, flags); + viport_kick(control->parent); + } + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); +} + +static void control_timeout(unsigned long data) +{ + struct control *control; + unsigned long flags; + u8 fail = 0; + u8 kick = 0; + + control = (struct control *)data; + CONTROL_FUNCTION("%s: control_timeout(), State=%d\n", + control_ifcfg_name(control), control->req_state); + control->timer_state = TIMER_EXPIRED; + + spin_lock_irqsave(&control->io_lock, flags); + switch (control->req_state) { + case REQ_INACTIVE: + kick = 1; + /* stay in REQ_INACTIVE state */ + break; + case REQ_POSTED: + case REQ_SENT: + control->req_state = REQ_FAILED; + CONTROL_ERROR("%s: No send Completion for Cmd=%d \n", + control_ifcfg_name(control), control->last_cmd); + control_timeout_stats(control); + fail = 1; + break; + case RSP_RECEIVED: + control->req_state = REQ_FAILED; + CONTROL_ERROR("%s: No response received from EIOC for Cmd=%d\n", + control_ifcfg_name(control), control->last_cmd); + control_timeout_stats(control); + fail = 1; + break; + case REQ_COMPLETED: + /* stay in REQ_COMPLETED state*/ + kick = 1; + break; + case REQ_FAILED: + /* stay in REQ_FAILED state*/ + break; + } + spin_unlock_irqrestore(&control->io_lock, flags); + /* we must do this outside the lock */ + if (fail) + viport_failure(control->parent); + if (kick) + viport_kick(control->parent); + + return; +} + +static void control_timer(struct control *control, int timeout) +{ + CONTROL_FUNCTION("%s: control_timer()\n", + control_ifcfg_name(control)); + if (control->timer_state == TIMER_ACTIVE) + mod_timer(&control->timer, jiffies + timeout); + else { + init_timer(&control->timer); + control->timer.expires = jiffies + timeout; + control->timer.data = (unsigned long)control; + control->timer.function = control_timeout; + control->timer_state = TIMER_ACTIVE; + add_timer(&control->timer); + } +} + +static void control_timer_stop(struct control *control) +{ + CONTROL_FUNCTION("%s: control_timer_stop()\n", + control_ifcfg_name(control)); + if (control->timer_state == TIMER_ACTIVE) + del_timer_sync(&control->timer); + + control->timer_state = TIMER_IDLE; +} + +static int control_send(struct control *control, struct send_io *send_io) +{ + unsigned long flags; + u8 ret = -1; + u8 fail = 0; + struct vnic_control_packet *pkt = control_packet(send_io); + + CONTROL_FUNCTION("%s: control_send(), State=%d\n", + control_ifcfg_name(control), control->req_state); + spin_lock_irqsave(&control->io_lock, flags); + switch (control->req_state) { + case REQ_INACTIVE: + CONTROL_PACKET(pkt); + control_timer(control, control->config->rsp_timeout); + control_note_reqtime_stats(control); + if (vnic_ib_post_send(&control->ib_conn, &control->send_io.io)) { + CONTROL_ERROR("%s: Failed to post send\n", + control_ifcfg_name(control)); + /* stay in REQ_INACTIVE state*/ + fail = 1; + } else { + control->last_cmd = pkt->hdr.pkt_cmd; + control->req_state = REQ_POSTED; + ret = 0; + } + break; + case REQ_POSTED: + case REQ_SENT: + case RSP_RECEIVED: + case REQ_COMPLETED: + CONTROL_ERROR("%s:Previous Command is not completed." + "New CMD: %d Last CMD: %d Seq: %d\n", + control_ifcfg_name(control), pkt->hdr.pkt_cmd, + control->last_cmd, control->seq_num); + + control->req_state = REQ_FAILED; + fail = 1; + break; + case REQ_FAILED: + /* this can occur after an error when ViPort state machine + * attempts to reset the link. + */ + CONTROL_INFO("%s:Attempt to send in failed state." + "New CMD: %d Last CMD: %d\n", + control_ifcfg_name(control), pkt->hdr.pkt_cmd, + control->last_cmd); + /* stay in REQ_FAILED state*/ + break; + } + spin_unlock_irqrestore(&control->io_lock, flags); + + /* we must do this outside the lock */ + if (fail) + viport_failure(control->parent); + return ret; + +} + +static void control_send_complete(struct io *io) +{ + struct control *control = &io->viport->control; + unsigned long flags; + u8 fail = 0; + u8 kick = 0; + + CONTROL_FUNCTION("%s: control_sendComplete(), State=%d\n", + control_ifcfg_name(control), control->req_state); + spin_lock_irqsave(&control->io_lock, flags); + switch (control->req_state) { + case REQ_INACTIVE: + case REQ_SENT: + case REQ_COMPLETED: + CONTROL_ERROR("%s: Unexpected control send completion\n", + control_ifcfg_name(control)); + fail = 1; + control->req_state = REQ_FAILED; + break; + case REQ_POSTED: + control->req_state = REQ_SENT; + break; + case RSP_RECEIVED: + control->req_state = REQ_COMPLETED; + kick = 1; + break; + case REQ_FAILED: + /* stay in REQ_FAILED state */ + break; + } + spin_unlock_irqrestore(&control->io_lock, flags); + /* we must do this outside the lock */ + if (fail) + viport_failure(control->parent); + if (kick) + viport_kick(control->parent); + + return; +} + +void control_process_async(struct control *control) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + unsigned long flags; + + CONTROL_FUNCTION("%s: control_process_async()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + spin_lock_irqsave(&control->io_lock, flags); + recv_io = control->info; + if (recv_io) { + CONTROL_INFO("%s: processing info packet\n", + control_ifcfg_name(control)); + control->info = NULL; + spin_unlock_irqrestore(&control->io_lock, flags); + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd == CMD_REPORT_STATUS) { + u32 status; + status = + be32_to_cpu(pkt->cmd.report_status.status_number); + switch (status) { + case VNIC_STATUS_LINK_UP: + CONTROL_INFO("%s: link up\n", + control_ifcfg_name(control)); + vnic_link_up(control->parent->vnic, + control->parent->parent); + break; + case VNIC_STATUS_LINK_DOWN: + CONTROL_INFO("%s: link down\n", + control_ifcfg_name(control)); + vnic_link_down(control->parent->vnic, + control->parent->parent); + break; + default: + CONTROL_ERROR("%s: asynchronous status" + " received from EIOC\n", + control_ifcfg_name(control)); + control_log_control_packet(pkt); + break; + } + } + if ((pkt->hdr.pkt_cmd != CMD_REPORT_STATUS) || + pkt->cmd.report_status.is_fatal) + viport_failure(control->parent); + + control_recv(control, recv_io); + spin_lock_irqsave(&control->io_lock, flags); + } + + while (!list_empty(&control->failure_list)) { + CONTROL_INFO("%s: processing error packet\n", + control_ifcfg_name(control)); + recv_io = (struct recv_io *) + list_entry(control->failure_list.next, struct io, + list_ptrs); + list_del(&recv_io->io.list_ptrs); + spin_unlock_irqrestore(&control->io_lock, flags); + pkt = control_packet(recv_io); + CONTROL_ERROR("%s: asynchronous error received from EIOC\n", + control_ifcfg_name(control)); + control_log_control_packet(pkt); + if ((pkt->hdr.pkt_type != TYPE_ERR) + || (pkt->hdr.pkt_cmd != CMD_REPORT_STATUS) + || pkt->cmd.report_status.is_fatal) + viport_failure(control->parent); + + control_recv(control, recv_io); + spin_lock_irqsave(&control->io_lock, flags); + } + spin_unlock_irqrestore(&control->io_lock, flags); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + CONTROL_FUNCTION("%s: done control_process_async\n", + control_ifcfg_name(control)); +} + +static struct send_io *control_init_hdr(struct control *control, u8 cmd) +{ + struct control_config *config; + struct vnic_control_packet *pkt; + struct vnic_control_header *hdr; + + CONTROL_FUNCTION("control_init_hdr()\n"); + config = control->config; + + pkt = control_packet(&control->send_io); + hdr = &pkt->hdr; + + hdr->pkt_type = TYPE_REQ; + hdr->pkt_cmd = cmd; + control->seq_num++; + hdr->pkt_seq_num = control->seq_num; + hdr->pkt_retry_count = 0; + + return &control->send_io; +} + +static struct recv_io *control_get_rsp(struct control *control) +{ + struct recv_io *recv_io = NULL; + unsigned long flags; + u8 fail = 0; + + CONTROL_FUNCTION("%s: control_getRsp(), State=%d\n", + control_ifcfg_name(control), control->req_state); + spin_lock_irqsave(&control->io_lock, flags); + switch (control->req_state) { + case REQ_INACTIVE: + CONTROL_ERROR("%s: Checked for Response with no" + "command pending\n", + control_ifcfg_name(control)); + control->req_state = REQ_FAILED; + fail = 1; + break; + case REQ_POSTED: + case REQ_SENT: + case RSP_RECEIVED: + /* no response available yet + stay in present state*/ + break; + case REQ_COMPLETED: + recv_io = control->response; + if (!recv_io) { + control->req_state = REQ_FAILED; + fail = 1; + break; + } + control->response = NULL; + control->last_cmd = CMD_INVALID; + control_timer_stop(control); + control->req_state = REQ_INACTIVE; + break; + case REQ_FAILED: + control_timer_stop(control); + /* stay in REQ_FAILED state*/ + break; + } + spin_unlock_irqrestore(&control->io_lock, flags); + if (fail) + viport_failure(control->parent); + return recv_io; +} + +int control_init_vnic_req(struct control *control) +{ + struct send_io *send_io; + struct control_config *config = control->config; + struct vnic_control_packet *pkt; + struct vnic_cmd_init_vnic_req *init_vnic_req; + + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_INIT_VNIC); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + init_vnic_req = &pkt->cmd.init_vnic_req; + init_vnic_req->vnic_major_version = + __constant_cpu_to_be16(VNIC_MAJORVERSION); + init_vnic_req->vnic_minor_version = + __constant_cpu_to_be16(VNIC_MINORVERSION); + init_vnic_req->vnic_instance = config->vnic_instance; + init_vnic_req->num_data_paths = 1; + init_vnic_req->num_address_entries = + cpu_to_be16(config->max_address_entries); + + control->last_cmd = pkt->hdr.pkt_cmd; + CONTROL_PACKET(pkt); + + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +static int control_chk_vnic_rsp_values(struct control *control, + u16 *num_addrs, + u8 num_data_paths, + u8 num_lan_switches, + u32 *features) +{ + + struct control_config *config = control->config; + + if ((control->maj_ver > VNIC_MAJORVERSION) + || ((control->maj_ver == VNIC_MAJORVERSION) + && (control->min_ver > VNIC_MINORVERSION))) { + CONTROL_ERROR("%s: unsupported version\n", + control_ifcfg_name(control)); + goto failure; + } + if (num_data_paths != 1) { + CONTROL_ERROR("%s: EIOC returned too many datapaths\n", + control_ifcfg_name(control)); + goto failure; + } + if (*num_addrs > config->max_address_entries) { + CONTROL_ERROR("%s: EIOC returned more address" + " entries than requested\n", + control_ifcfg_name(control)); + goto failure; + } + if (*num_addrs < config->min_address_entries) { + CONTROL_ERROR("%s: not enough address entries\n", + control_ifcfg_name(control)); + goto failure; + } + if (num_lan_switches < 1) { + CONTROL_ERROR("%s: EIOC returned no lan switches\n", + control_ifcfg_name(control)); + goto failure; + } + if (num_lan_switches > 1) { + CONTROL_ERROR("%s: EIOC returned multiple lan switches\n", + control_ifcfg_name(control)); + goto failure; + } + CONTROL_ERROR("%s checking features %x ib_multicast:%d\n", + control_ifcfg_name(control), + *features, config->ib_multicast); + if ((*features & VNIC_FEAT_INBOUND_IB_MC) && !config->ib_multicast) { + /* disable multicast if it is not on in the cfg file, or + if we turned it off because join failed */ + *features &= ~VNIC_FEAT_INBOUND_IB_MC; + } + + return 0; +failure: + return -1; +} + +int control_init_vnic_rsp(struct control *control, u32 *features, + u8 *mac_address, u16 *num_addrs, u16 *vlan) +{ + u8 num_data_paths; + u8 num_lan_switches; + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_init_vnic_rsp *init_vnic_rsp; + + + CONTROL_FUNCTION("%s: control_init_vnic_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_INIT_VNIC) + goto failure; + + init_vnic_rsp = &pkt->cmd.init_vnic_rsp; + control->maj_ver = be16_to_cpu(init_vnic_rsp->vnic_major_version); + control->min_ver = be16_to_cpu(init_vnic_rsp->vnic_minor_version); + num_data_paths = init_vnic_rsp->num_data_paths; + num_lan_switches = init_vnic_rsp->num_lan_switches; + *features = be32_to_cpu(init_vnic_rsp->features_supported); + *num_addrs = be16_to_cpu(init_vnic_rsp->num_address_entries); + + if (control_chk_vnic_rsp_values(control, num_addrs, + num_data_paths, + num_lan_switches, + features)) + goto failure; + + control->lan_switch.lan_switch_num = + init_vnic_rsp->lan_switch[0].lan_switch_num; + control->lan_switch.num_enet_ports = + init_vnic_rsp->lan_switch[0].num_enet_ports; + control->lan_switch.default_vlan = + init_vnic_rsp->lan_switch[0].default_vlan; + *vlan = be16_to_cpu(control->lan_switch.default_vlan); + memcpy(control->lan_switch.hw_mac_address, + init_vnic_rsp->lan_switch[0].hw_mac_address, ETH_ALEN); + memcpy(mac_address, init_vnic_rsp->lan_switch[0].hw_mac_address, + ETH_ALEN); + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +static void copy_recv_pool_config(struct vnic_recv_pool_config *src, + struct vnic_recv_pool_config *dst) +{ + dst->size_recv_pool_entry = src->size_recv_pool_entry; + dst->num_recv_pool_entries = src->num_recv_pool_entries; + dst->timeout_before_kick = src->timeout_before_kick; + dst->num_recv_pool_entries_before_kick = + src->num_recv_pool_entries_before_kick; + dst->num_recv_pool_bytes_before_kick = + src->num_recv_pool_bytes_before_kick; + dst->free_recv_pool_entries_per_update = + src->free_recv_pool_entries_per_update; +} + +static int check_recv_pool_config_value(__be32 *src, __be32 *dst, + __be32 *max, __be32 *min, + char *name) +{ + u32 value; + + value = be32_to_cpu(*src); + if (value > be32_to_cpu(*max)) { + CONTROL_ERROR("value %s too large\n", name); + return -1; + } else if (value < be32_to_cpu(*min)) { + CONTROL_ERROR("value %s too small\n", name); + return -1; + } + + *dst = cpu_to_be32(value); + return 0; +} + +static int check_recv_pool_config(struct vnic_recv_pool_config *src, + struct vnic_recv_pool_config *dst, + struct vnic_recv_pool_config *max, + struct vnic_recv_pool_config *min) +{ + if (check_recv_pool_config_value(&src->size_recv_pool_entry, + &dst->size_recv_pool_entry, + &max->size_recv_pool_entry, + &min->size_recv_pool_entry, + "size_recv_pool_entry") + || check_recv_pool_config_value(&src->num_recv_pool_entries, + &dst->num_recv_pool_entries, + &max->num_recv_pool_entries, + &min->num_recv_pool_entries, + "num_recv_pool_entries") + || check_recv_pool_config_value(&src->timeout_before_kick, + &dst->timeout_before_kick, + &max->timeout_before_kick, + &min->timeout_before_kick, + "timeout_before_kick") + || check_recv_pool_config_value(&src-> + num_recv_pool_entries_before_kick, + &dst-> + num_recv_pool_entries_before_kick, + &max-> + num_recv_pool_entries_before_kick, + &min-> + num_recv_pool_entries_before_kick, + "num_recv_pool_entries_before_kick") + || check_recv_pool_config_value(&src-> + num_recv_pool_bytes_before_kick, + &dst-> + num_recv_pool_bytes_before_kick, + &max-> + num_recv_pool_bytes_before_kick, + &min-> + num_recv_pool_bytes_before_kick, + "num_recv_pool_bytes_before_kick") + || check_recv_pool_config_value(&src-> + free_recv_pool_entries_per_update, + &dst-> + free_recv_pool_entries_per_update, + &max-> + free_recv_pool_entries_per_update, + &min-> + free_recv_pool_entries_per_update, + "free_recv_pool_entries_per_update")) + goto failure; + + if (!is_power_of_2(be32_to_cpu(dst->num_recv_pool_entries))) { + CONTROL_ERROR("num_recv_pool_entries (%d)" + " must be power of 2\n", + dst->num_recv_pool_entries); + goto failure; + } + + if (!is_power_of_2(be32_to_cpu(dst-> + free_recv_pool_entries_per_update))) { + CONTROL_ERROR("free_recv_pool_entries_per_update (%d)" + " must be power of 2\n", + dst->free_recv_pool_entries_per_update); + goto failure; + } + + if (be32_to_cpu(dst->free_recv_pool_entries_per_update) >= + be32_to_cpu(dst->num_recv_pool_entries)) { + CONTROL_ERROR("free_recv_pool_entries_per_update (%d) must" + " be less than num_recv_pool_entries (%d)\n", + dst->free_recv_pool_entries_per_update, + dst->num_recv_pool_entries); + goto failure; + } + + if (be32_to_cpu(dst->num_recv_pool_entries_before_kick) >= + be32_to_cpu(dst->num_recv_pool_entries)) { + CONTROL_ERROR("num_recv_pool_entries_before_kick (%d) must" + " be less than num_recv_pool_entries (%d)\n", + dst->num_recv_pool_entries_before_kick, + dst->num_recv_pool_entries); + goto failure; + } + + return 0; +failure: + return -1; +} + +int control_config_data_path_req(struct control *control, u64 path_id, + struct vnic_recv_pool_config *host, + struct vnic_recv_pool_config *eioc) +{ + struct send_io *send_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_config_data_path *config_data_path; + + CONTROL_FUNCTION("%s: control_config_data_path_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_CONFIG_DATA_PATH); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + config_data_path = &pkt->cmd.config_data_path_req; + config_data_path->data_path = 0; + config_data_path->path_identifier = path_id; + copy_recv_pool_config(host, + &config_data_path->host_recv_pool_config); + copy_recv_pool_config(eioc, + &config_data_path->eioc_recv_pool_config); + CONTROL_PACKET(pkt); + + control->last_cmd = pkt->hdr.pkt_cmd; + + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +int control_config_data_path_rsp(struct control *control, + struct vnic_recv_pool_config *host, + struct vnic_recv_pool_config *eioc, + struct vnic_recv_pool_config *max_host, + struct vnic_recv_pool_config *max_eioc, + struct vnic_recv_pool_config *min_host, + struct vnic_recv_pool_config *min_eioc) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_config_data_path *config_data_path; + + CONTROL_FUNCTION("%s: control_config_data_path_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_CONFIG_DATA_PATH) + goto failure; + + config_data_path = &pkt->cmd.config_data_path_rsp; + if (config_data_path->data_path != 0) { + CONTROL_ERROR("%s: received CMD_CONFIG_DATA_PATH response" + " for wrong data path: %u\n", + control_ifcfg_name(control), + config_data_path->data_path); + goto failure; + } + + if (check_recv_pool_config(&config_data_path-> + host_recv_pool_config, + host, max_host, min_host) + || check_recv_pool_config(&config_data_path-> + eioc_recv_pool_config, + eioc, max_eioc, min_eioc)) { + goto failure; + } + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +int control_exchange_pools_req(struct control *control, u64 addr, u32 rkey) +{ + struct send_io *send_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_exchange_pools *exchange_pools; + + CONTROL_FUNCTION("%s: control_exchange_pools_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_EXCHANGE_POOLS); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + exchange_pools = &pkt->cmd.exchange_pools_req; + exchange_pools->data_path = 0; + exchange_pools->pool_rkey = cpu_to_be32(rkey); + exchange_pools->pool_addr = cpu_to_be64(addr); + + control->last_cmd = pkt->hdr.pkt_cmd; + + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +int control_exchange_pools_rsp(struct control *control, u64 *addr, + u32 *rkey) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_exchange_pools *exchange_pools; + + CONTROL_FUNCTION("%s: control_exchange_pools_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_EXCHANGE_POOLS) + goto failure; + + exchange_pools = &pkt->cmd.exchange_pools_rsp; + *rkey = be32_to_cpu(exchange_pools->pool_rkey); + *addr = be64_to_cpu(exchange_pools->pool_addr); + + if (exchange_pools->data_path != 0) { + CONTROL_ERROR("%s: received CMD_EXCHANGE_POOLS response" + " for wrong data path: %u\n", + control_ifcfg_name(control), + exchange_pools->data_path); + goto failure; + } + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +int control_config_link_req(struct control *control, u16 flags, u16 mtu) +{ + struct send_io *send_io; + struct vnic_cmd_config_link *config_link_req; + struct vnic_control_packet *pkt; + + CONTROL_FUNCTION("%s: control_config_link_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_CONFIG_LINK); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + config_link_req = &pkt->cmd.config_link_req; + config_link_req->lan_switch_num = + control->lan_switch.lan_switch_num; + config_link_req->cmd_flags = VNIC_FLAG_SET_MTU; + if (flags & IFF_UP) + config_link_req->cmd_flags |= VNIC_FLAG_ENABLE_NIC; + else + config_link_req->cmd_flags |= VNIC_FLAG_DISABLE_NIC; + if (flags & IFF_ALLMULTI) + config_link_req->cmd_flags |= VNIC_FLAG_ENABLE_MCAST_ALL; + else + config_link_req->cmd_flags |= VNIC_FLAG_DISABLE_MCAST_ALL; + if (flags & IFF_PROMISC) { + config_link_req->cmd_flags |= VNIC_FLAG_ENABLE_PROMISC; + /* the EIOU doesn't really do PROMISC mode. + * if PROMISC is set, it only receives unicast packets + * I also have to set MCAST_ALL if I want real + * PROMISC mode. + */ + config_link_req->cmd_flags &= ~VNIC_FLAG_DISABLE_MCAST_ALL; + config_link_req->cmd_flags |= VNIC_FLAG_ENABLE_MCAST_ALL; + } else + config_link_req->cmd_flags |= VNIC_FLAG_DISABLE_PROMISC; + + config_link_req->mtu_size = cpu_to_be16(mtu); + + control->last_cmd = pkt->hdr.pkt_cmd; + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +int control_config_link_rsp(struct control *control, u16 *flags, u16 *mtu) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_config_link *config_link_rsp; + + CONTROL_FUNCTION("%s: control_config_link_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_CONFIG_LINK) + goto failure; + config_link_rsp = &pkt->cmd.config_link_rsp; + if (config_link_rsp->cmd_flags & VNIC_FLAG_ENABLE_NIC) + *flags |= IFF_UP; + if (config_link_rsp->cmd_flags & VNIC_FLAG_ENABLE_MCAST_ALL) + *flags |= IFF_ALLMULTI; + if (config_link_rsp->cmd_flags & VNIC_FLAG_ENABLE_PROMISC) + *flags |= IFF_PROMISC; + + *mtu = be16_to_cpu(config_link_rsp->mtu_size); + + if (control->parent->features_supported & VNIC_FEAT_INBOUND_IB_MC) { + /* featuresSupported might include INBOUND_IB_MC but + MTU might cause it to be auto-disabled at embedded */ + if (config_link_rsp->cmd_flags & VNIC_FLAG_ENABLE_MCAST_ALL) { + union ib_gid mgid = config_link_rsp->allmulti_mgid; + if (mgid.raw[0] != 0xff) { + CONTROL_ERROR("%s: invalid formatprefix " + VNIC_GID_FMT "\n", + control_ifcfg_name(control), + VNIC_GID_RAW_ARG(mgid.raw)); + } else { + /* rather than issuing join here, which might + * arrive at SM before EVIC creates the MC + * group, postpone it. + */ + vnic_mc_join_setup(control->parent, &mgid); + CONTROL_ERROR("join setup for ALL_MULTI\n"); + } + } + /* we don't want to leave mcast group if MCAST_ALL is disabled + * because there are no doubt multicast addresses set and we + * want to stay joined so we can get that traffic via the + * mcast group. + */ + } + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +/* control_config_addrs_req: + * return values: + * -1: failure + * 0: incomplete (successful operation, but more address + * table entries to be updated) + * 1: complete + */ +int control_config_addrs_req(struct control *control, + struct vnic_address_op2 *addrs, u16 num) +{ + u16 i; + u8 j; + int ret = 1; + struct send_io *send_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_config_addresses *config_addrs_req; + struct vnic_cmd_config_addresses2 *config_addrs_req2; + + CONTROL_FUNCTION("%s: control_config_addrs_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + if (control->parent->features_supported & VNIC_FEAT_INBOUND_IB_MC) { + CONTROL_INFO("Sending CMD_CONFIG_ADDRESSES2 %lx MAX:%d " + "sizes:%d %d(off:%d) sizes2:%d %d %d" + "(off:%d - %d %d %d %d %d %d %d)\n", jiffies, + (int)MAX_CONFIG_ADDR_ENTRIES2, + (int)sizeof(struct vnic_cmd_config_addresses), + (int)sizeof(struct vnic_address_op), + (int)offsetof(struct vnic_cmd_config_addresses, + list_address_ops), + (int)sizeof(struct vnic_cmd_config_addresses2), + (int)sizeof(struct vnic_address_op2), + (int)sizeof(union ib_gid), + (int)offsetof(struct vnic_cmd_config_addresses2, + list_address_ops), + (int)offsetof(struct vnic_address_op2, index), + (int)offsetof(struct vnic_address_op2, operation), + (int)offsetof(struct vnic_address_op2, valid), + (int)offsetof(struct vnic_address_op2, address), + (int)offsetof(struct vnic_address_op2, vlan), + (int)offsetof(struct vnic_address_op2, reserved), + (int)offsetof(struct vnic_address_op2, mgid) + ); + send_io = control_init_hdr(control, CMD_CONFIG_ADDRESSES2); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + config_addrs_req2 = &pkt->cmd.config_addresses_req2; + memset(pkt->cmd.cmd_data, 0, VNIC_MAX_CONTROLDATASZ); + config_addrs_req2->lan_switch_num = + control->lan_switch.lan_switch_num; + for (i = 0, j = 0; (i < num) && (j < MAX_CONFIG_ADDR_ENTRIES2); i++) { + if (!addrs[i].operation) + continue; + config_addrs_req2->list_address_ops[j].index = + cpu_to_be16(i); + config_addrs_req2->list_address_ops[j].operation = + VNIC_OP_SET_ENTRY; + config_addrs_req2->list_address_ops[j].valid = + addrs[i].valid; + memcpy(config_addrs_req2->list_address_ops[j].address, + addrs[i].address, ETH_ALEN); + config_addrs_req2->list_address_ops[j].vlan = + addrs[i].vlan; + addrs[i].operation = 0; + CONTROL_INFO("%s i=%d " + "addr[%d]=%02x:%02x:%02x:%02x:%02x:%02x " + "valid:%d\n", control_ifcfg_name(control), i, j, + addrs[i].address[0], addrs[i].address[1], + addrs[i].address[2], addrs[i].address[3], + addrs[i].address[4], addrs[i].address[5], + addrs[i].valid); + j++; + } + config_addrs_req2->num_address_ops = j; + } else { + send_io = control_init_hdr(control, CMD_CONFIG_ADDRESSES); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + config_addrs_req = &pkt->cmd.config_addresses_req; + config_addrs_req->lan_switch_num = + control->lan_switch.lan_switch_num; + for (i = 0, j = 0; (i < num) && (j < 16); i++) { + if (!addrs[i].operation) + continue; + config_addrs_req->list_address_ops[j].index = + cpu_to_be16(i); + config_addrs_req->list_address_ops[j].operation = + VNIC_OP_SET_ENTRY; + config_addrs_req->list_address_ops[j].valid = + addrs[i].valid; + memcpy(config_addrs_req->list_address_ops[j].address, + addrs[i].address, ETH_ALEN); + config_addrs_req->list_address_ops[j].vlan = + addrs[i].vlan; + addrs[i].operation = 0; + j++; + } + config_addrs_req->num_address_ops = j; + } + for (; i < num; i++) { + if (addrs[i].operation) { + ret = 0; + break; + } + } + + control->last_cmd = pkt->hdr.pkt_cmd; + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + if (control_send(control, send_io)) + return -1; + return ret; +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +static int process_cmd_config_address2_rsp(struct control *control, + struct vnic_control_packet *pkt, + struct recv_io *recv_io) +{ + struct vnic_cmd_config_addresses2 *config_addrs_rsp2; + int idx, mcaddrs, nomgid; + union ib_gid mgid, rsp_mgid; + + config_addrs_rsp2 = &pkt->cmd.config_addresses_rsp2; + CONTROL_INFO("%s rsp to CONFIG_ADDRESSES2\n", + control_ifcfg_name(control)); + + for (idx = 0, mcaddrs = 0, nomgid = 1; + idx < config_addrs_rsp2->num_address_ops; + idx++) { + if (!config_addrs_rsp2->list_address_ops[idx].valid) + continue; + + /* check if address is multicasts */ + if (!vnic_multicast_address(config_addrs_rsp2, idx)) + continue; + + mcaddrs++; + mgid = config_addrs_rsp2->list_address_ops[idx].mgid; + CONTROL_INFO("%s: got mgid " VNIC_GID_FMT + " MCAST_MSG_SIZE:%d mtu:%d\n", + control_ifcfg_name(control), + VNIC_GID_RAW_ARG(mgid.raw), + (int)MCAST_MSG_SIZE, + control->parent->mtu); + + /* Embedded should have turned off multicast + * due to large MTU size; mgid had better be 0. + */ + if (control->parent->mtu > MCAST_MSG_SIZE) { + if ((mgid.global.subnet_prefix != 0) || + (mgid.global.interface_id != 0)) { + CONTROL_ERROR("%s: invalid mgid; " + "expected 0 " + VNIC_GID_FMT "\n", + control_ifcfg_name(control), + VNIC_GID_RAW_ARG(mgid.raw)); + } + continue; + } + if (mgid.raw[0] != 0xff) { + CONTROL_ERROR("%s: invalid formatprefix " + VNIC_GID_FMT "\n", + control_ifcfg_name(control), + VNIC_GID_RAW_ARG(mgid.raw)); + continue; + } + nomgid = 0; /* got a valid mgid */ + + /* let's verify that all the mgids match this one */ + for (; idx < config_addrs_rsp2->num_address_ops; idx++) { + if (!config_addrs_rsp2->list_address_ops[idx].valid) + continue; + + /* check if address is multicasts */ + if (!vnic_multicast_address(config_addrs_rsp2, idx)) + continue; + + rsp_mgid = config_addrs_rsp2->list_address_ops[idx].mgid; + if (memcmp(&mgid, &rsp_mgid, sizeof(union ib_gid)) == 0) + continue; + + CONTROL_ERROR("%s: Multicast Group MGIDs not " + "unique; mgids: " VNIC_GID_FMT + " " VNIC_GID_FMT "\n", + control_ifcfg_name(control), + VNIC_GID_RAW_ARG(mgid.raw), + VNIC_GID_RAW_ARG(rsp_mgid.raw)); + return 1; + } + + /* rather than issuing join here, which might arrive + * at SM before EVIC creates the MC group, postpone it. + */ + vnic_mc_join_setup(control->parent, &mgid); + + /* there is only one multicast group to join, so we're done. */ + break; + } + + /* we sent atleast one multicast address but got no MGID + * back so, if it is not allmulti case, leave the group + * we joined before. (for allmulti case we have to stay + * joined) + */ + if ((config_addrs_rsp2->num_address_ops > 0) && (mcaddrs > 0) && + nomgid && !(control->parent->flags & IFF_ALLMULTI)) { + CONTROL_INFO("numaddrops:%d mcadrs:%d nomgid:%d\n", + config_addrs_rsp2->num_address_ops, + mcaddrs > 0, nomgid); + + vnic_mc_leave(control->parent); + } + + return 0; +} + +int control_config_addrs_rsp(struct control *control) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + + CONTROL_FUNCTION("%s: control_config_addrs_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if ((pkt->hdr.pkt_cmd != CMD_CONFIG_ADDRESSES) && + (pkt->hdr.pkt_cmd != CMD_CONFIG_ADDRESSES2)) + goto failure; + + if (((pkt->hdr.pkt_cmd == CMD_CONFIG_ADDRESSES2) && + !control->parent->features_supported & VNIC_FEAT_INBOUND_IB_MC) || + ((pkt->hdr.pkt_cmd == CMD_CONFIG_ADDRESSES) && + control->parent->features_supported & VNIC_FEAT_INBOUND_IB_MC)) { + CONTROL_ERROR("%s unexpected response pktCmd:%d flag:%x\n", + control_ifcfg_name(control), pkt->hdr.pkt_cmd, + control->parent->features_supported & + VNIC_FEAT_INBOUND_IB_MC); + goto failure; + } + + if (pkt->hdr.pkt_cmd == CMD_CONFIG_ADDRESSES2) { + if (process_cmd_config_address2_rsp(control, pkt, recv_io)) + goto failure; + } else { + struct vnic_cmd_config_addresses *config_addrs_rsp; + config_addrs_rsp = &pkt->cmd.config_addresses_rsp; + } + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +int control_report_statistics_req(struct control *control) +{ + struct send_io *send_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_report_stats_req *report_statistics_req; + + CONTROL_FUNCTION("%s: control_report_statistics_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_REPORT_STATISTICS); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + report_statistics_req = &pkt->cmd.report_statistics_req; + report_statistics_req->lan_switch_num = + control->lan_switch.lan_switch_num; + + control->last_cmd = pkt->hdr.pkt_cmd; + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +int control_report_statistics_rsp(struct control *control, + struct vnic_cmd_report_stats_rsp *stats) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_report_stats_rsp *rep_stat_rsp; + + CONTROL_FUNCTION("%s: control_report_statistics_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_REPORT_STATISTICS) + goto failure; + + rep_stat_rsp = &pkt->cmd.report_statistics_rsp; + + stats->if_in_broadcast_pkts = rep_stat_rsp->if_in_broadcast_pkts; + stats->if_in_multicast_pkts = rep_stat_rsp->if_in_multicast_pkts; + stats->if_in_octets = rep_stat_rsp->if_in_octets; + stats->if_in_ucast_pkts = rep_stat_rsp->if_in_ucast_pkts; + stats->if_in_nucast_pkts = rep_stat_rsp->if_in_nucast_pkts; + stats->if_in_underrun = rep_stat_rsp->if_in_underrun; + stats->if_in_errors = rep_stat_rsp->if_in_errors; + stats->if_out_errors = rep_stat_rsp->if_out_errors; + stats->if_out_octets = rep_stat_rsp->if_out_octets; + stats->if_out_ucast_pkts = rep_stat_rsp->if_out_ucast_pkts; + stats->if_out_multicast_pkts = rep_stat_rsp->if_out_multicast_pkts; + stats->if_out_broadcast_pkts = rep_stat_rsp->if_out_broadcast_pkts; + stats->if_out_nucast_pkts = rep_stat_rsp->if_out_nucast_pkts; + stats->if_out_ok = rep_stat_rsp->if_out_ok; + stats->if_in_ok = rep_stat_rsp->if_in_ok; + stats->if_out_ucast_bytes = rep_stat_rsp->if_out_ucast_bytes; + stats->if_out_multicast_bytes = rep_stat_rsp->if_out_multicast_bytes; + stats->if_out_broadcast_bytes = rep_stat_rsp->if_out_broadcast_bytes; + stats->if_in_ucast_bytes = rep_stat_rsp->if_in_ucast_bytes; + stats->if_in_multicast_bytes = rep_stat_rsp->if_in_multicast_bytes; + stats->if_in_broadcast_bytes = rep_stat_rsp->if_in_broadcast_bytes; + stats->ethernet_status = rep_stat_rsp->ethernet_status; + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +int control_reset_req(struct control *control) +{ + struct send_io *send_io; + struct vnic_control_packet *pkt; + + CONTROL_FUNCTION("%s: control_reset_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_RESET); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + + control->last_cmd = pkt->hdr.pkt_cmd; + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +int control_reset_rsp(struct control *control) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + + CONTROL_FUNCTION("%s: control_reset_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_RESET) + goto failure; + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +int control_heartbeat_req(struct control *control, u32 hb_interval) +{ + struct send_io *send_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_heartbeat *heartbeat_req; + + CONTROL_FUNCTION("%s: control_heartbeat_req()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + + send_io = control_init_hdr(control, CMD_HEARTBEAT); + if (!send_io) + goto failure; + + pkt = control_packet(send_io); + heartbeat_req = &pkt->cmd.heartbeat_req; + heartbeat_req->hb_interval = cpu_to_be32(hb_interval); + + control->last_cmd = pkt->hdr.pkt_cmd; + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return control_send(control, send_io); +failure: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + return -1; +} + +int control_heartbeat_rsp(struct control *control) +{ + struct recv_io *recv_io; + struct vnic_control_packet *pkt; + struct vnic_cmd_heartbeat *heartbeat_rsp; + + CONTROL_FUNCTION("%s: control_heartbeat_rsp()\n", + control_ifcfg_name(control)); + ib_dma_sync_single_for_cpu(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + + recv_io = control_get_rsp(control); + if (!recv_io) + goto out; + + pkt = control_packet(recv_io); + if (pkt->hdr.pkt_cmd != CMD_HEARTBEAT) + goto failure; + + heartbeat_rsp = &pkt->cmd.heartbeat_rsp; + + control_recv(control, recv_io); + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return 0; +failure: + viport_failure(control->parent); +out: + ib_dma_sync_single_for_device(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + return -1; +} + +static int control_init_recv_ios(struct control *control, + struct viport *viport, + struct vnic_control_packet *pkt) +{ + struct io *io; + struct ib_device *ibdev = viport->config->ibdev; + struct control_config *config = control->config; + dma_addr_t recv_dma; + unsigned int i; + + + control->recv_len = sizeof *pkt * config->num_recvs; + control->recv_dma = ib_dma_map_single(ibdev, + pkt, control->recv_len, + DMA_FROM_DEVICE); + + if (ib_dma_mapping_error(ibdev, control->recv_dma)) { + CONTROL_ERROR("control recv dma map error\n"); + goto failure; + } + + recv_dma = control->recv_dma; + for (i = 0; i < config->num_recvs; i++) { + io = &control->recv_ios[i].io; + io->viport = viport; + io->routine = control_recv_complete; + io->type = RECV; + + control->recv_ios[i].virtual_addr = (u8 *)pkt; + control->recv_ios[i].list.addr = recv_dma; + control->recv_ios[i].list.length = sizeof *pkt; + control->recv_ios[i].list.lkey = control->mr->lkey; + + recv_dma = recv_dma + sizeof *pkt; + pkt++; + + io->rwr.wr_id = (u64)io; + io->rwr.sg_list = &control->recv_ios[i].list; + io->rwr.num_sge = 1; + if (vnic_ib_post_recv(&control->ib_conn, io)) + goto unmap_recv; + } + + return 0; +unmap_recv: + ib_dma_unmap_single(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); +failure: + return -1; +} + +static int control_init_send_ios(struct control *control, + struct viport *viport, + struct vnic_control_packet *pkt) +{ + struct io *io; + struct ib_device *ibdev = viport->config->ibdev; + + control->send_io.virtual_addr = (u8 *)pkt; + control->send_len = sizeof *pkt; + control->send_dma = ib_dma_map_single(ibdev, pkt, + control->send_len, + DMA_TO_DEVICE); + if (ib_dma_mapping_error(ibdev, control->send_dma)) { + CONTROL_ERROR("control send dma map error\n"); + goto failure; + } + + io = &control->send_io.io; + io->viport = viport; + io->routine = control_send_complete; + + control->send_io.list.addr = control->send_dma; + control->send_io.list.length = sizeof *pkt; + control->send_io.list.lkey = control->mr->lkey; + + io->swr.wr_id = (u64)io; + io->swr.sg_list = &control->send_io.list; + io->swr.num_sge = 1; + io->swr.opcode = IB_WR_SEND; + io->swr.send_flags = IB_SEND_SIGNALED; + io->type = SEND; + + return 0; +failure: + return -1; +} + +int control_init(struct control *control, struct viport *viport, + struct control_config *config, struct ib_pd *pd) +{ + struct vnic_control_packet *pkt; + unsigned int sz; + + CONTROL_FUNCTION("%s: control_init()\n", + control_ifcfg_name(control)); + control->parent = viport; + control->config = config; + control->ib_conn.viport = viport; + control->ib_conn.ib_config = &config->ib_config; + control->ib_conn.state = IB_CONN_UNINITTED; + control->ib_conn.callback_thread = NULL; + control->ib_conn.callback_thread_end = 0; + control->req_state = REQ_INACTIVE; + control->last_cmd = CMD_INVALID; + control->seq_num = 0; + control->response = NULL; + control->info = NULL; + INIT_LIST_HEAD(&control->failure_list); + spin_lock_init(&control->io_lock); + + if (vnic_ib_conn_init(&control->ib_conn, viport, pd, + &config->ib_config)) { + CONTROL_ERROR("Control IB connection" + " initialization failed\n"); + goto failure; + } + + control->mr = ib_get_dma_mr(pd, IB_ACCESS_LOCAL_WRITE); + if (IS_ERR(control->mr)) { + CONTROL_ERROR("%s: failed to register memory" + " for control connection\n", + control_ifcfg_name(control)); + goto destroy_conn; + } + + control->ib_conn.cm_id = ib_create_cm_id(viport->config->ibdev, + vnic_ib_cm_handler, + &control->ib_conn); + if (IS_ERR(control->ib_conn.cm_id)) { + CONTROL_ERROR("creating control CM ID failed\n"); + goto destroy_mr; + } + + sz = sizeof(struct recv_io) * config->num_recvs; + control->recv_ios = vmalloc(sz); + + if (!control->recv_ios) { + CONTROL_ERROR("%s: failed allocating space for recv ios\n", + control_ifcfg_name(control)); + goto destroy_cm_id; + } + + memset(control->recv_ios, 0, sz); + /*One send buffer and num_recvs recv buffers */ + control->local_storage = kzalloc(sizeof *pkt * + (config->num_recvs + 1), + GFP_KERNEL); + + if (!control->local_storage) { + CONTROL_ERROR("%s: failed allocating space" + " for local storage\n", + control_ifcfg_name(control)); + goto free_recv_ios; + } + + pkt = control->local_storage; + if (control_init_send_ios(control, viport, pkt)) + goto free_storage; + + pkt++; + if (control_init_recv_ios(control, viport, pkt)) + goto unmap_send; + + return 0; + +unmap_send: + ib_dma_unmap_single(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); +free_storage: + kfree(control->local_storage); +free_recv_ios: + vfree(control->recv_ios); +destroy_cm_id: + ib_destroy_cm_id(control->ib_conn.cm_id); +destroy_mr: + ib_dereg_mr(control->mr); +destroy_conn: + ib_destroy_qp(control->ib_conn.qp); + ib_destroy_cq(control->ib_conn.cq); +failure: + return -1; +} + +void control_cleanup(struct control *control) +{ + CONTROL_FUNCTION("%s: control_disconnect()\n", + control_ifcfg_name(control)); + + if (ib_send_cm_dreq(control->ib_conn.cm_id, NULL, 0)) + CONTROL_ERROR("control CM DREQ sending failed\n"); + + control->ib_conn.state = IB_CONN_DISCONNECTED; + control_timer_stop(control); + control->req_state = REQ_INACTIVE; + control->response = NULL; + control->last_cmd = CMD_INVALID; + vnic_completion_cleanup(&control->ib_conn); + ib_destroy_cm_id(control->ib_conn.cm_id); + ib_destroy_qp(control->ib_conn.qp); + ib_destroy_cq(control->ib_conn.cq); + ib_dereg_mr(control->mr); + ib_dma_unmap_single(control->parent->config->ibdev, + control->send_dma, control->send_len, + DMA_TO_DEVICE); + ib_dma_unmap_single(control->parent->config->ibdev, + control->recv_dma, control->recv_len, + DMA_FROM_DEVICE); + vfree(control->recv_ios); + kfree(control->local_storage); + +} + +static void control_log_report_status_pkt(struct vnic_control_packet *pkt) +{ + printk(KERN_INFO + " pkt_cmd = CMD_REPORT_STATUS\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO + " lan_switch_num = %u, is_fatal = %u\n", + pkt->cmd.report_status.lan_switch_num, + pkt->cmd.report_status.is_fatal); + printk(KERN_INFO + " status_number = %u, status_info = %u\n", + be32_to_cpu(pkt->cmd.report_status.status_number), + be32_to_cpu(pkt->cmd.report_status.status_info)); + pkt->cmd.report_status.file_name[31] = '\0'; + pkt->cmd.report_status.routine[31] = '\0'; + printk(KERN_INFO " filename = %s, routine = %s\n", + pkt->cmd.report_status.file_name, + pkt->cmd.report_status.routine); + printk(KERN_INFO + " line_num = %u, error_parameter = %u\n", + be32_to_cpu(pkt->cmd.report_status.line_num), + be32_to_cpu(pkt->cmd.report_status.error_parameter)); + pkt->cmd.report_status.desc_text[127] = '\0'; + printk(KERN_INFO " desc_text = %s\n", + pkt->cmd.report_status.desc_text); +} + +static void control_log_report_stats_pkt(struct vnic_control_packet *pkt) +{ + printk(KERN_INFO + " pkt_cmd = CMD_REPORT_STATISTICS\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO " lan_switch_num = %u\n", + pkt->cmd.report_statistics_req.lan_switch_num); + if (pkt->hdr.pkt_type == TYPE_REQ) + return; + printk(KERN_INFO " if_in_broadcast_pkts = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_broadcast_pkts)); + printk(" if_in_multicast_pkts = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_multicast_pkts)); + printk(KERN_INFO " if_in_octets = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_octets)); + printk(" if_in_ucast_pkts = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_ucast_pkts)); + printk(KERN_INFO " if_in_nucast_pkts = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_nucast_pkts)); + printk(" if_in_underrun = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_underrun)); + printk(KERN_INFO " if_in_errors = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_errors)); + printk(" if_out_errors = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_errors)); + printk(KERN_INFO " if_out_octets = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_octets)); + printk(" if_out_ucast_pkts = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_ucast_pkts)); + printk(KERN_INFO " if_out_multicast_pkts = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_multicast_pkts)); + printk(" if_out_broadcast_pkts = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_broadcast_pkts)); + printk(KERN_INFO " if_out_nucast_pkts = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_nucast_pkts)); + printk(" if_out_ok = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp.if_out_ok)); + printk(KERN_INFO " if_in_ok = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp.if_in_ok)); + printk(" if_out_ucast_bytes = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_ucast_bytes)); + printk(KERN_INFO " if_out_multicast_bytes = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_multicast_bytes)); + printk(" if_out_broadcast_bytes = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_out_broadcast_bytes)); + printk(KERN_INFO " if_in_ucast_bytes = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_ucast_bytes)); + printk(" if_in_multicast_bytes = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_multicast_bytes)); + printk(KERN_INFO " if_in_broadcast_bytes = %llu", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + if_in_broadcast_bytes)); + printk(" ethernet_status = %llu\n", + be64_to_cpu(pkt->cmd.report_statistics_rsp. + ethernet_status)); +} + +static void control_log_config_link_pkt(struct vnic_control_packet *pkt) +{ + printk(KERN_INFO + " pkt_cmd = CMD_CONFIG_LINK\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO " cmd_flags = %x\n", + pkt->cmd.config_link_req.cmd_flags); + if (pkt->cmd.config_link_req.cmd_flags & VNIC_FLAG_ENABLE_NIC) + printk(KERN_INFO + " VNIC_FLAG_ENABLE_NIC\n"); + if (pkt->cmd.config_link_req.cmd_flags & VNIC_FLAG_DISABLE_NIC) + printk(KERN_INFO + " VNIC_FLAG_DISABLE_NIC\n"); + if (pkt->cmd.config_link_req. + cmd_flags & VNIC_FLAG_ENABLE_MCAST_ALL) + printk(KERN_INFO + " VNIC_FLAG_ENABLE_" + "MCAST_ALL\n"); + if (pkt->cmd.config_link_req. + cmd_flags & VNIC_FLAG_DISABLE_MCAST_ALL) + printk(KERN_INFO + " VNIC_FLAG_DISABLE_" + "MCAST_ALL\n"); + if (pkt->cmd.config_link_req. + cmd_flags & VNIC_FLAG_ENABLE_PROMISC) + printk(KERN_INFO + " VNIC_FLAG_ENABLE_" + "PROMISC\n"); + if (pkt->cmd.config_link_req. + cmd_flags & VNIC_FLAG_DISABLE_PROMISC) + printk(KERN_INFO + " VNIC_FLAG_DISABLE_" + "PROMISC\n"); + if (pkt->cmd.config_link_req.cmd_flags & VNIC_FLAG_SET_MTU) + printk(KERN_INFO + " VNIC_FLAG_SET_MTU\n"); + printk(KERN_INFO + " lan_switch_num = %x, mtu_size = %d\n", + pkt->cmd.config_link_req.lan_switch_num, + be16_to_cpu(pkt->cmd.config_link_req.mtu_size)); + if (pkt->hdr.pkt_type == TYPE_RSP) { + printk(KERN_INFO + " default_vlan = %u," + " hw_mac_address =" + " %02x:%02x:%02x:%02x:%02x:%02x\n", + be16_to_cpu(pkt->cmd.config_link_req. + default_vlan), + pkt->cmd.config_link_req.hw_mac_address[0], + pkt->cmd.config_link_req.hw_mac_address[1], + pkt->cmd.config_link_req.hw_mac_address[2], + pkt->cmd.config_link_req.hw_mac_address[3], + pkt->cmd.config_link_req.hw_mac_address[4], + pkt->cmd.config_link_req.hw_mac_address[5]); + } +} + +static void print_config_addr(struct vnic_address_op *list, + int num_address_ops, size_t mgidoff) +{ + int i = 0; + + while (i < num_address_ops && i < 16) { + printk(KERN_INFO " list_address_ops[%u].index" + " = %u\n", i, be16_to_cpu(list->index)); + switch (list->operation) { + case VNIC_OP_GET_ENTRY: + printk(KERN_INFO " list_address_ops[%u]." + "operation = VNIC_OP_GET_ENTRY\n", i); + break; + case VNIC_OP_SET_ENTRY: + printk(KERN_INFO " list_address_ops[%u]." + "operation = VNIC_OP_SET_ENTRY\n", i); + break; + default: + printk(KERN_INFO " list_address_ops[%u]." + "operation = UNKNOWN(%d)\n", i, + list->operation); + break; + } + printk(KERN_INFO " list_address_ops[%u].valid" + " = %u\n", i, list->valid); + printk(KERN_INFO " list_address_ops[%u].address" + " = %02x:%02x:%02x:%02x:%02x:%02x\n", i, + list->address[0], list->address[1], + list->address[2], list->address[3], + list->address[4], list->address[5]); + printk(KERN_INFO " list_address_ops[%u].vlan" + " = %u\n", i, be16_to_cpu(list->vlan)); + if (mgidoff) { + printk(KERN_INFO + " list_address_ops[%u].mgid" + " = " VNIC_GID_FMT "\n", i, + VNIC_GID_RAW_ARG((char *)list + mgidoff)); + list = (struct vnic_address_op *) + ((char *)list + sizeof(struct vnic_address_op2)); + } else + list = (struct vnic_address_op *) + ((char *)list + sizeof(struct vnic_address_op)); + i++; + } +} + +static void control_log_config_addrs_pkt(struct vnic_control_packet *pkt, + u8 addresses2) +{ + struct vnic_address_op *list; + int no_address_ops; + + if (addresses2) + printk(KERN_INFO + " pkt_cmd = CMD_CONFIG_ADDRESSES2\n"); + else + printk(KERN_INFO + " pkt_cmd = CMD_CONFIG_ADDRESSES\n"); + printk(KERN_INFO " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, pkt->hdr.pkt_retry_count); + if (addresses2) { + printk(KERN_INFO " num_address_ops = %x," + " lan_switch_num = %d\n", + pkt->cmd.config_addresses_req2.num_address_ops, + pkt->cmd.config_addresses_req2.lan_switch_num); + list = (struct vnic_address_op *) + pkt->cmd.config_addresses_req2.list_address_ops; + no_address_ops = pkt->cmd.config_addresses_req2.num_address_ops; + print_config_addr(list, no_address_ops, + offsetof(struct vnic_address_op2, mgid)); + } else { + printk(KERN_INFO " num_address_ops = %x," + " lan_switch_num = %d\n", + pkt->cmd.config_addresses_req.num_address_ops, + pkt->cmd.config_addresses_req.lan_switch_num); + list = pkt->cmd.config_addresses_req.list_address_ops; + no_address_ops = pkt->cmd.config_addresses_req.num_address_ops; + print_config_addr(list, no_address_ops, 0); + } +} + +static void control_log_exch_pools_pkt(struct vnic_control_packet *pkt) +{ + printk(KERN_INFO + " pkt_cmd = CMD_EXCHANGE_POOLS\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO " datapath = %u\n", + pkt->cmd.exchange_pools_req.data_path); + printk(KERN_INFO " pool_rkey = %08x" + " pool_addr = %llx\n", + be32_to_cpu(pkt->cmd.exchange_pools_req.pool_rkey), + be64_to_cpu(pkt->cmd.exchange_pools_req.pool_addr)); +} + +static void control_log_data_path_pkt(struct vnic_control_packet *pkt) +{ + printk(KERN_INFO + " pkt_cmd = CMD_CONFIG_DATA_PATH\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO " path_identifier = %llx," + " data_path = %u\n", + pkt->cmd.config_data_path_req.path_identifier, + pkt->cmd.config_data_path_req.data_path); + printk(KERN_INFO + "host config size_recv_pool_entry = %u," + " num_recv_pool_entries = %u\n", + be32_to_cpu(pkt->cmd.config_data_path_req. + host_recv_pool_config.size_recv_pool_entry), + be32_to_cpu(pkt->cmd.config_data_path_req. + host_recv_pool_config.num_recv_pool_entries)); + printk(KERN_INFO + " timeout_before_kick = %u," + " num_recv_pool_entries_before_kick = %u\n", + be32_to_cpu(pkt->cmd.config_data_path_req. + host_recv_pool_config.timeout_before_kick), + be32_to_cpu(pkt->cmd.config_data_path_req. + host_recv_pool_config. + num_recv_pool_entries_before_kick)); + printk(KERN_INFO + " num_recv_pool_bytes_before_kick = %u," + " free_recv_pool_entries_per_update = %u\n", + be32_to_cpu(pkt->cmd.config_data_path_req. + host_recv_pool_config. + num_recv_pool_bytes_before_kick), + be32_to_cpu(pkt->cmd.config_data_path_req. + host_recv_pool_config. + free_recv_pool_entries_per_update)); + printk(KERN_INFO + "eioc config size_recv_pool_entry = %u," + " num_recv_pool_entries = %u\n", + be32_to_cpu(pkt->cmd.config_data_path_req. + eioc_recv_pool_config.size_recv_pool_entry), + be32_to_cpu(pkt->cmd.config_data_path_req. + eioc_recv_pool_config.num_recv_pool_entries)); + printk(KERN_INFO + " timeout_before_kick = %u," + " num_recv_pool_entries_before_kick = %u\n", + be32_to_cpu(pkt->cmd.config_data_path_req. + eioc_recv_pool_config.timeout_before_kick), + be32_to_cpu(pkt->cmd.config_data_path_req. + eioc_recv_pool_config. + num_recv_pool_entries_before_kick)); + printk(KERN_INFO + " num_recv_pool_bytes_before_kick = %u," + " free_recv_pool_entries_per_update = %u\n", + be32_to_cpu(pkt->cmd.config_data_path_req. + eioc_recv_pool_config. + num_recv_pool_bytes_before_kick), + be32_to_cpu(pkt->cmd.config_data_path_req. + eioc_recv_pool_config. + free_recv_pool_entries_per_update)); +} + +static void control_log_init_vnic_pkt(struct vnic_control_packet *pkt) +{ + printk(KERN_INFO + " pkt_cmd = CMD_INIT_VNIC\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO + " vnic_major_version = %u," + " vnic_minor_version = %u\n", + be16_to_cpu(pkt->cmd.init_vnic_req.vnic_major_version), + be16_to_cpu(pkt->cmd.init_vnic_req.vnic_minor_version)); + if (pkt->hdr.pkt_type == TYPE_REQ) { + printk(KERN_INFO + " vnic_instance = %u," + " num_data_paths = %u\n", + pkt->cmd.init_vnic_req.vnic_instance, + pkt->cmd.init_vnic_req.num_data_paths); + printk(KERN_INFO + " num_address_entries = %u\n", + be16_to_cpu(pkt->cmd.init_vnic_req. + num_address_entries)); + } else { + printk(KERN_INFO + " num_lan_switches = %u," + " num_data_paths = %u\n", + pkt->cmd.init_vnic_rsp.num_lan_switches, + pkt->cmd.init_vnic_rsp.num_data_paths); + printk(KERN_INFO + " num_address_entries = %u," + " features_supported = %08x\n", + be16_to_cpu(pkt->cmd.init_vnic_rsp. + num_address_entries), + be32_to_cpu(pkt->cmd.init_vnic_rsp. + features_supported)); + if (pkt->cmd.init_vnic_rsp.num_lan_switches != 0) { + printk(KERN_INFO + "lan_switch[0] lan_switch_num = %u," + " num_enet_ports = %08x\n", + pkt->cmd.init_vnic_rsp. + lan_switch[0].lan_switch_num, + pkt->cmd.init_vnic_rsp. + lan_switch[0].num_enet_ports); + printk(KERN_INFO + " default_vlan = %u," + " hw_mac_address =" + " %02x:%02x:%02x:%02x:%02x:%02x\n", + be16_to_cpu(pkt->cmd.init_vnic_rsp. + lan_switch[0].default_vlan), + pkt->cmd.init_vnic_rsp.lan_switch[0]. + hw_mac_address[0], + pkt->cmd.init_vnic_rsp.lan_switch[0]. + hw_mac_address[1], + pkt->cmd.init_vnic_rsp.lan_switch[0]. + hw_mac_address[2], + pkt->cmd.init_vnic_rsp.lan_switch[0]. + hw_mac_address[3], + pkt->cmd.init_vnic_rsp.lan_switch[0]. + hw_mac_address[4], + pkt->cmd.init_vnic_rsp.lan_switch[0]. + hw_mac_address[5]); + } + } +} + +static void control_log_control_packet(struct vnic_control_packet *pkt) +{ + switch (pkt->hdr.pkt_type) { + case TYPE_INFO: + printk(KERN_INFO "control_packet: pkt_type = TYPE_INFO\n"); + break; + case TYPE_REQ: + printk(KERN_INFO "control_packet: pkt_type = TYPE_REQ\n"); + break; + case TYPE_RSP: + printk(KERN_INFO "control_packet: pkt_type = TYPE_RSP\n"); + break; + case TYPE_ERR: + printk(KERN_INFO "control_packet: pkt_type = TYPE_ERR\n"); + break; + default: + printk(KERN_INFO "control_packet: pkt_type = UNKNOWN\n"); + } + + switch (pkt->hdr.pkt_cmd) { + case CMD_INIT_VNIC: + control_log_init_vnic_pkt(pkt); + break; + case CMD_CONFIG_DATA_PATH: + control_log_data_path_pkt(pkt); + break; + case CMD_EXCHANGE_POOLS: + control_log_exch_pools_pkt(pkt); + break; + case CMD_CONFIG_ADDRESSES: + control_log_config_addrs_pkt(pkt, 0); + break; + case CMD_CONFIG_ADDRESSES2: + control_log_config_addrs_pkt(pkt, 1); + break; + case CMD_CONFIG_LINK: + control_log_config_link_pkt(pkt); + break; + case CMD_REPORT_STATISTICS: + control_log_report_stats_pkt(pkt); + break; + case CMD_CLEAR_STATISTICS: + printk(KERN_INFO + " pkt_cmd = CMD_CLEAR_STATISTICS\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + break; + case CMD_REPORT_STATUS: + control_log_report_status_pkt(pkt); + + break; + case CMD_RESET: + printk(KERN_INFO + " pkt_cmd = CMD_RESET\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + break; + case CMD_HEARTBEAT: + printk(KERN_INFO + " pkt_cmd = CMD_HEARTBEAT\n"); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + printk(KERN_INFO " hb_interval = %d\n", + be32_to_cpu(pkt->cmd.heartbeat_req.hb_interval)); + break; + default: + printk(KERN_INFO + " pkt_cmd = UNKNOWN (%u)\n", + pkt->hdr.pkt_cmd); + printk(KERN_INFO + " pkt_seq_num = %u," + " pkt_retry_count = %u\n", + pkt->hdr.pkt_seq_num, + pkt->hdr.pkt_retry_count); + break; + } +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_control.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_control.h new file mode 100644 index 0000000..77307da --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_control.h @@ -0,0 +1,180 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_CONTROL_H_INCLUDED +#define VNIC_CONTROL_H_INCLUDED + +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS +#include +#include +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ + +#include "vnic_ib.h" +#include "vnic_control_pkt.h" + +enum control_timer_state { + TIMER_IDLE = 0, + TIMER_ACTIVE = 1, + TIMER_EXPIRED = 2 +}; + +enum control_request_state { + REQ_INACTIVE, /* quiet state, all previous operations done + * response is NULL + * last_cmd = CMD_INVALID + * timer_state = IDLE + */ + REQ_POSTED, /* REQ put on send Q + * response is NULL + * last_cmd = command issued + * timer_state = ACTIVE + */ + REQ_SENT, /* Send completed for REQ + * response is NULL + * last_cmd = command issued + * timer_state = ACTIVE + */ + RSP_RECEIVED, /* Received Resp, but no Send completion yet + * response is response buffer received + * last_cmd = command issued + * timer_state = ACTIVE + */ + REQ_COMPLETED, /* all processing for REQ completed, ready to be gotten + * response is response buffer received + * last_cmd = command issued + * timer_state = ACTIVE + */ + REQ_FAILED, /* processing of REQ/RSP failed. + * response is NULL + * last_cmd = CMD_INVALID + * timer_state = IDLE or EXPIRED + * viport has been moved to error state to force + * recovery + */ +}; + +struct control { + struct viport *parent; + struct control_config *config; + struct ib_mr *mr; + struct vnic_ib_conn ib_conn; + struct vnic_control_packet *local_storage; + int send_len; + int recv_len; + u16 maj_ver; + u16 min_ver; + struct vnic_lan_switch_attribs lan_switch; + struct send_io send_io; + struct recv_io *recv_ios; + dma_addr_t send_dma; + dma_addr_t recv_dma; + enum control_timer_state timer_state; + enum control_request_state req_state; + struct timer_list timer; + u8 seq_num; + u8 last_cmd; + struct recv_io *response; + struct recv_io *info; + struct list_head failure_list; + spinlock_t io_lock; + struct completion done; +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS + struct { + cycles_t request_time; /* intermediate value */ + cycles_t response_time; + u32 response_num; + cycles_t response_max; + cycles_t response_min; + u32 timeout_num; + } statistics; +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ +}; + +int control_init(struct control *control, struct viport *viport, + struct control_config *config, struct ib_pd *pd); + +void control_cleanup(struct control *control); + +void control_process_async(struct control *control); + +int control_init_vnic_req(struct control *control); +int control_init_vnic_rsp(struct control *control, u32 *features, + u8 *mac_address, u16 *num_addrs, u16 *vlan); + +int control_config_data_path_req(struct control *control, u64 path_id, + struct vnic_recv_pool_config *host, + struct vnic_recv_pool_config *eioc); +int control_config_data_path_rsp(struct control *control, + struct vnic_recv_pool_config *host, + struct vnic_recv_pool_config *eioc, + struct vnic_recv_pool_config *max_host, + struct vnic_recv_pool_config *max_eioc, + struct vnic_recv_pool_config *min_host, + struct vnic_recv_pool_config *min_eioc); + +int control_exchange_pools_req(struct control *control, + u64 addr, u32 rkey); +int control_exchange_pools_rsp(struct control *control, + u64 *addr, u32 *rkey); + +int control_config_link_req(struct control *control, + u16 flags, u16 mtu); +int control_config_link_rsp(struct control *control, + u16 *flags, u16 *mtu); + +int control_config_addrs_req(struct control *control, + struct vnic_address_op2 *addrs, u16 num); +int control_config_addrs_rsp(struct control *control); + +int control_report_statistics_req(struct control *control); +int control_report_statistics_rsp(struct control *control, + struct vnic_cmd_report_stats_rsp *stats); + +int control_heartbeat_req(struct control *control, u32 hb_interval); +int control_heartbeat_rsp(struct control *control); + +int control_reset_req(struct control *control); +int control_reset_rsp(struct control *control); + +#define control_packet(io) \ + ((struct vnic_control_packet *)(io)->virtual_addr) + +#define control_is_connected(control) \ + (vnic_ib_conn_connected(&((control)->ib_conn))) + +#define control_last_req(control) control_packet(&(control)->send_io) +#define control_features(control) ((control)->features_supported) + +#define control_get_mac_address(control,addr) \ + memcpy(addr, (control)->lan_switch.hw_mac_address, ETH_ALEN) + +#endif /* VNIC_CONTROL_H_INCLUDED */ diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_control_pkt.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_control_pkt.h new file mode 100644 index 0000000..1fc62fb --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_control_pkt.h @@ -0,0 +1,368 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_CONTROL_PKT_H_INCLUDED +#define VNIC_CONTROL_PKT_H_INCLUDED + +#include +#include + +#define VNIC_MAX_NODENAME_LEN 64 + +struct vnic_connection_data { + u64 path_id; + u8 vnic_instance; + u8 path_num; + u8 nodename[VNIC_MAX_NODENAME_LEN + 1]; + u8 reserved; /* for alignment */ + __be32 features_supported; +}; + +struct vnic_control_header { + u8 pkt_type; + u8 pkt_cmd; + u8 pkt_seq_num; + u8 pkt_retry_count; + u32 reserved; /* for 64-bit alignmnet */ +}; + +/* ptk_type values */ +enum { + TYPE_INFO = 0, + TYPE_REQ = 1, + TYPE_RSP = 2, + TYPE_ERR = 3 +}; + +/* ptk_cmd values */ +enum { + CMD_INVALID = 0, + CMD_INIT_VNIC = 1, + CMD_CONFIG_DATA_PATH = 2, + CMD_EXCHANGE_POOLS = 3, + CMD_CONFIG_ADDRESSES = 4, + CMD_CONFIG_LINK = 5, + CMD_REPORT_STATISTICS = 6, + CMD_CLEAR_STATISTICS = 7, + CMD_REPORT_STATUS = 8, + CMD_RESET = 9, + CMD_HEARTBEAT = 10, + CMD_CONFIG_ADDRESSES2 = 11, +}; + +/* pkt_cmd CMD_INIT_VNIC, pkt_type TYPE_REQ data format */ +struct vnic_cmd_init_vnic_req { + __be16 vnic_major_version; + __be16 vnic_minor_version; + u8 vnic_instance; + u8 num_data_paths; + __be16 num_address_entries; +}; + +/* pkt_cmd CMD_INIT_VNIC, pkt_type TYPE_RSP subdata format */ +struct vnic_lan_switch_attribs { + u8 lan_switch_num; + u8 num_enet_ports; + __be16 default_vlan; + u8 hw_mac_address[ETH_ALEN]; +}; + +/* pkt_cmd CMD_INIT_VNIC, pkt_type TYPE_RSP data format */ +struct vnic_cmd_init_vnic_rsp { + __be16 vnic_major_version; + __be16 vnic_minor_version; + u8 num_lan_switches; + u8 num_data_paths; + __be16 num_address_entries; + __be32 features_supported; + struct vnic_lan_switch_attribs lan_switch[1]; +}; + +/* features_supported values */ +enum { + VNIC_FEAT_IPV4_HEADERS = 0x0001, + VNIC_FEAT_IPV6_HEADERS = 0x0002, + VNIC_FEAT_IPV4_CSUM_RX = 0x0004, + VNIC_FEAT_IPV4_CSUM_TX = 0x0008, + VNIC_FEAT_TCP_CSUM_RX = 0x0010, + VNIC_FEAT_TCP_CSUM_TX = 0x0020, + VNIC_FEAT_UDP_CSUM_RX = 0x0040, + VNIC_FEAT_UDP_CSUM_TX = 0x0080, + VNIC_FEAT_TCP_SEGMENT = 0x0100, + VNIC_FEAT_IPV4_IPSEC_OFFLOAD = 0x0200, + VNIC_FEAT_IPV6_IPSEC_OFFLOAD = 0x0400, + VNIC_FEAT_FCS_PROPAGATE = 0x0800, + VNIC_FEAT_PF_KICK = 0x1000, + VNIC_FEAT_PF_FORCE_ROUTE = 0x2000, + VNIC_FEAT_CHASH_OFFLOAD = 0x4000, + /* host send with immediate data */ + VNIC_FEAT_RDMA_IMMED = 0x8000, + /* host ignore inbound PF_VLAN_INSERT flag */ + VNIC_FEAT_IGNORE_VLAN = 0x10000, + /* host supports IB multicast for inbound Ethernet mcast traffic */ + VNIC_FEAT_INBOUND_IB_MC = 0x20000, +}; + +/* pkt_cmd CMD_CONFIG_DATA_PATH subdata format */ +struct vnic_recv_pool_config { + __be32 size_recv_pool_entry; + __be32 num_recv_pool_entries; + __be32 timeout_before_kick; + __be32 num_recv_pool_entries_before_kick; + __be32 num_recv_pool_bytes_before_kick; + __be32 free_recv_pool_entries_per_update; +}; + +/* pkt_cmd CMD_CONFIG_DATA_PATH data format */ +struct vnic_cmd_config_data_path { + u64 path_identifier; + u8 data_path; + u8 reserved[3]; + struct vnic_recv_pool_config host_recv_pool_config; + struct vnic_recv_pool_config eioc_recv_pool_config; +}; + +/* pkt_cmd CMD_EXCHANGE_POOLS data format */ +struct vnic_cmd_exchange_pools { + u8 data_path; + u8 reserved[3]; + __be32 pool_rkey; + __be64 pool_addr; +}; + +/* pkt_cmd CMD_CONFIG_ADDRESSES subdata format */ +struct vnic_address_op { + __be16 index; + u8 operation; + u8 valid; + u8 address[6]; + __be16 vlan; +}; + +/* pkt_cmd CMD_CONFIG_ADDRESSES2 subdata format */ +struct vnic_address_op2 { + __be16 index; + u8 operation; + u8 valid; + u8 address[6]; + __be16 vlan; + u32 reserved; /* for alignment */ + union ib_gid mgid; /* valid in rsp only if both ends support mcast */ +}; + +/* operation values */ +enum { + VNIC_OP_SET_ENTRY = 0x01, + VNIC_OP_GET_ENTRY = 0x02 +}; + +/* pkt_cmd CMD_CONFIG_ADDRESSES data format */ +struct vnic_cmd_config_addresses { + u8 num_address_ops; + u8 lan_switch_num; + struct vnic_address_op list_address_ops[1]; +}; + +/* pkt_cmd CMD_CONFIG_ADDRESSES2 data format */ +struct vnic_cmd_config_addresses2 { + u8 num_address_ops; + u8 lan_switch_num; + u8 reserved1; + u8 reserved2; + u8 reserved3; + struct vnic_address_op2 list_address_ops[1]; +}; + +/* CMD_CONFIG_LINK data format */ +struct vnic_cmd_config_link { + u8 cmd_flags; + u8 lan_switch_num; + __be16 mtu_size; + __be16 default_vlan; + u8 hw_mac_address[6]; + u32 reserved; /* for alignment */ + /* valid in rsp only if both ends support mcast */ + union ib_gid allmulti_mgid; +}; + +/* cmd_flags values */ +enum { + VNIC_FLAG_ENABLE_NIC = 0x01, + VNIC_FLAG_DISABLE_NIC = 0x02, + VNIC_FLAG_ENABLE_MCAST_ALL = 0x04, + VNIC_FLAG_DISABLE_MCAST_ALL = 0x08, + VNIC_FLAG_ENABLE_PROMISC = 0x10, + VNIC_FLAG_DISABLE_PROMISC = 0x20, + VNIC_FLAG_SET_MTU = 0x40 +}; + +/* pkt_cmd CMD_REPORT_STATISTICS, pkt_type TYPE_REQ data format */ +struct vnic_cmd_report_stats_req { + u8 lan_switch_num; +}; + +/* pkt_cmd CMD_REPORT_STATISTICS, pkt_type TYPE_RSP data format */ +struct vnic_cmd_report_stats_rsp { + u8 lan_switch_num; + u8 reserved[7]; /* for 64-bit alignment */ + __be64 if_in_broadcast_pkts; + __be64 if_in_multicast_pkts; + __be64 if_in_octets; + __be64 if_in_ucast_pkts; + __be64 if_in_nucast_pkts; /* if_in_broadcast_pkts + + if_in_multicast_pkts */ + __be64 if_in_underrun; /* (OID_GEN_RCV_NO_BUFFER) */ + __be64 if_in_errors; /* (OID_GEN_RCV_ERROR) */ + __be64 if_out_errors; /* (OID_GEN_XMIT_ERROR) */ + __be64 if_out_octets; + __be64 if_out_ucast_pkts; + __be64 if_out_multicast_pkts; + __be64 if_out_broadcast_pkts; + __be64 if_out_nucast_pkts; /* if_out_broadcast_pkts + + if_out_multicast_pkts */ + __be64 if_out_ok; /* if_out_nucast_pkts + + if_out_ucast_pkts(OID_GEN_XMIT_OK) */ + __be64 if_in_ok; /* if_in_nucast_pkts + + if_in_ucast_pkts(OID_GEN_RCV_OK) */ + __be64 if_out_ucast_bytes; /* (OID_GEN_DIRECTED_BYTES_XMT) */ + __be64 if_out_multicast_bytes; /* (OID_GEN_MULTICAST_BYTES_XMT) */ + __be64 if_out_broadcast_bytes; /* (OID_GEN_BROADCAST_BYTES_XMT) */ + __be64 if_in_ucast_bytes; /* (OID_GEN_DIRECTED_BYTES_RCV) */ + __be64 if_in_multicast_bytes; /* (OID_GEN_MULTICAST_BYTES_RCV) */ + __be64 if_in_broadcast_bytes; /* (OID_GEN_BROADCAST_BYTES_RCV) */ + __be64 ethernet_status; /* OID_GEN_MEDIA_CONNECT_STATUS) */ +}; + +/* pkt_cmd CMD_CLEAR_STATISTICS data format */ +struct vnic_cmd_clear_statistics { + u8 lan_switch_num; +}; + +/* pkt_cmd CMD_REPORT_STATUS data format */ +struct vnic_cmd_report_status { + u8 lan_switch_num; + u8 is_fatal; + u8 reserved[2]; /* for 32-bit alignment */ + __be32 status_number; + __be32 status_info; + u8 file_name[32]; + u8 routine[32]; + __be32 line_num; + __be32 error_parameter; + u8 desc_text[128]; +}; + +/* pkt_cmd CMD_HEARTBEAT data format */ +struct vnic_cmd_heartbeat { + __be32 hb_interval; +}; + +enum { + VNIC_STATUS_LINK_UP = 1, + VNIC_STATUS_LINK_DOWN = 2, + VNIC_STATUS_ENET_AGGREGATION_CHANGE = 3, + VNIC_STATUS_EIOC_SHUTDOWN = 4, + VNIC_STATUS_CONTROL_ERROR = 5, + VNIC_STATUS_EIOC_ERROR = 6 +}; + +#define VNIC_MAX_CONTROLPKTSZ 256 +#define VNIC_MAX_CONTROLDATASZ \ + (VNIC_MAX_CONTROLPKTSZ - sizeof(struct vnic_control_header)) + +struct vnic_control_packet { + struct vnic_control_header hdr; + union { + struct vnic_cmd_init_vnic_req init_vnic_req; + struct vnic_cmd_init_vnic_rsp init_vnic_rsp; + struct vnic_cmd_config_data_path config_data_path_req; + struct vnic_cmd_config_data_path config_data_path_rsp; + struct vnic_cmd_exchange_pools exchange_pools_req; + struct vnic_cmd_exchange_pools exchange_pools_rsp; + struct vnic_cmd_config_addresses config_addresses_req; + struct vnic_cmd_config_addresses2 config_addresses_req2; + struct vnic_cmd_config_addresses config_addresses_rsp; + struct vnic_cmd_config_addresses2 config_addresses_rsp2; + struct vnic_cmd_config_link config_link_req; + struct vnic_cmd_config_link config_link_rsp; + struct vnic_cmd_report_stats_req report_statistics_req; + struct vnic_cmd_report_stats_rsp report_statistics_rsp; + struct vnic_cmd_clear_statistics clear_statistics_req; + struct vnic_cmd_clear_statistics clear_statistics_rsp; + struct vnic_cmd_report_status report_status; + struct vnic_cmd_heartbeat heartbeat_req; + struct vnic_cmd_heartbeat heartbeat_rsp; + + char cmd_data[VNIC_MAX_CONTROLDATASZ]; + } cmd; +}; + +union ib_gid_cpu { + u8 raw[16]; + struct { + u64 subnet_prefix; + u64 interface_id; + } global; +}; + +static inline void bswap_ib_gid(union ib_gid *mgid1, union ib_gid_cpu *mgid2) +{ + /* swap hi & low */ + __be64 low = mgid1->global.subnet_prefix; + mgid2->global.subnet_prefix = be64_to_cpu(mgid1->global.interface_id); + mgid2->global.interface_id = be64_to_cpu(low); +} + +#define VNIC_GID_FMT "%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x" + +#define VNIC_GID_RAW_ARG(gid) be16_to_cpu(*(__be16 *)&(gid)[0]), \ + be16_to_cpu(*(__be16 *)&(gid)[2]), \ + be16_to_cpu(*(__be16 *)&(gid)[4]), \ + be16_to_cpu(*(__be16 *)&(gid)[6]), \ + be16_to_cpu(*(__be16 *)&(gid)[8]), \ + be16_to_cpu(*(__be16 *)&(gid)[10]), \ + be16_to_cpu(*(__be16 *)&(gid)[12]), \ + be16_to_cpu(*(__be16 *)&(gid)[14]) + + +/* These defines are used to figure out how many address entries can be passed + * in config_addresses request. + */ +#define MAX_CONFIG_ADDR_ENTRIES \ + ((VNIC_MAX_CONTROLDATASZ - (sizeof(struct vnic_cmd_config_addresses) \ + - sizeof(struct vnic_address_op)))/sizeof(struct vnic_address_op)) +#define MAX_CONFIG_ADDR_ENTRIES2 \ + ((VNIC_MAX_CONTROLDATASZ - (sizeof(struct vnic_cmd_config_addresses2) \ + - sizeof(struct vnic_address_op2)))/sizeof(struct vnic_address_op2)) + + +#endif /* VNIC_CONTROL_PKT_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:04:46 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:04:46 -0400 Subject: [ofa-general] [PATCH v4 05/14] QLogic VNIC: Implementation of Data path of communication protocol In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210446.11186.15909.stgit@dale> From: Ramachandra K This patch implements the actual data transfer part of the communication protocol with the EVIC/VEx. RDMA of ethernet packets is implemented in here. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_data.c | 1492 +++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_data.h | 206 +++ drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h | 103 ++ 3 files changed, 1801 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_data.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_data.h create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_data.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_data.c new file mode 100644 index 0000000..02d94e7 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_data.c @@ -0,0 +1,1492 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_viport.h" +#include "vnic_main.h" +#include "vnic_data.h" +#include "vnic_trailer.h" +#include "vnic_stats.h" + +static void data_received_kick(struct io *io); +static void data_xmit_complete(struct io *io); + +static void mc_data_recv_routine(struct io *io); +static void mc_data_post_recvs(struct mc_data *mc_data); +static void mc_data_recv_to_skbuff(struct viport *viport, struct sk_buff *skb, + struct viport_trailer *trailer); + +static u32 min_rcv_skb = 60; +module_param(min_rcv_skb, int, 0444); +MODULE_PARM_DESC(min_rcv_skb, "Packets of size (in bytes) less than" + " or equal this value will be copied during receive." + " Default 60"); + +static u32 min_xmt_skb = 60; +module_param(min_xmt_skb, int, 0444); +MODULE_PARM_DESC(min_xmit_skb, "Packets of size (in bytes) less than" + " or equal to this value will be copied during transmit." + "Default 60"); + +int data_init(struct data *data, struct viport *viport, + struct data_config *config, struct ib_pd *pd) +{ + DATA_FUNCTION("data_init()\n"); + + data->parent = viport; + data->config = config; + data->ib_conn.viport = viport; + data->ib_conn.ib_config = &config->ib_config; + data->ib_conn.state = IB_CONN_UNINITTED; + data->ib_conn.callback_thread = NULL; + data->ib_conn.callback_thread_end = 0; + + if ((min_xmt_skb < 60) || (min_xmt_skb > 9000)) { + DATA_ERROR("min_xmt_skb (%d) must be between 60 and 9000\n", + min_xmt_skb); + goto failure; + } + if (vnic_ib_conn_init(&data->ib_conn, viport, pd, + &config->ib_config)) { + DATA_ERROR("Data IB connection initialization failed\n"); + goto failure; + } + data->mr = ib_get_dma_mr(pd, + IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_READ | + IB_ACCESS_REMOTE_WRITE); + if (IS_ERR(data->mr)) { + DATA_ERROR("failed to register memory for" + " data connection\n"); + goto destroy_conn; + } + + data->ib_conn.cm_id = ib_create_cm_id(viport->config->ibdev, + vnic_ib_cm_handler, + &data->ib_conn); + + if (IS_ERR(data->ib_conn.cm_id)) { + DATA_ERROR("creating data CM ID failed\n"); + goto dereg_mr; + } + + return 0; + +dereg_mr: + ib_dereg_mr(data->mr); +destroy_conn: + vnic_completion_cleanup(&data->ib_conn); + ib_destroy_qp(data->ib_conn.qp); + ib_destroy_cq(data->ib_conn.cq); +failure: + return -1; +} + +static void data_post_recvs(struct data *data) +{ + unsigned long flags; + int i = 0; + + DATA_FUNCTION("data_post_recvs()\n"); + spin_lock_irqsave(&data->recv_ios_lock, flags); + while (!list_empty(&data->recv_ios)) { + struct io *io = list_entry(data->recv_ios.next, + struct io, list_ptrs); + struct recv_io *recv_io = (struct recv_io *)io; + + list_del(&recv_io->io.list_ptrs); + spin_unlock_irqrestore(&data->recv_ios_lock, flags); + if (vnic_ib_post_recv(&data->ib_conn, &recv_io->io)) { + viport_failure(data->parent); + return; + } + i++; + spin_lock_irqsave(&data->recv_ios_lock, flags); + } + spin_unlock_irqrestore(&data->recv_ios_lock, flags); + DATA_INFO("data posted %d %p\n", i, &data->recv_ios); +} + +static void data_init_pool_work_reqs(struct data *data, + struct recv_io *recv_io) +{ + struct recv_pool *recv_pool = &data->recv_pool; + struct xmit_pool *xmit_pool = &data->xmit_pool; + struct rdma_io *rdma_io; + struct rdma_dest *rdma_dest; + dma_addr_t xmit_dma; + u8 *xmit_data; + unsigned int i; + + INIT_LIST_HEAD(&data->recv_ios); + spin_lock_init(&data->recv_ios_lock); + spin_lock_init(&data->xmit_buf_lock); + for (i = 0; i < data->config->num_recvs; i++) { + recv_io[i].io.viport = data->parent; + recv_io[i].io.routine = data_received_kick; + recv_io[i].list.addr = data->region_data_dma; + recv_io[i].list.length = 4; + recv_io[i].list.lkey = data->mr->lkey; + + recv_io[i].io.rwr.wr_id = (u64)&recv_io[i].io; + recv_io[i].io.rwr.sg_list = &recv_io[i].list; + recv_io[i].io.rwr.num_sge = 1; + + list_add(&recv_io[i].io.list_ptrs, &data->recv_ios); + } + + INIT_LIST_HEAD(&recv_pool->avail_recv_bufs); + for (i = 0; i < recv_pool->pool_sz; i++) { + rdma_dest = &recv_pool->recv_bufs[i]; + list_add(&rdma_dest->list_ptrs, + &recv_pool->avail_recv_bufs); + } + + xmit_dma = xmit_pool->xmitdata_dma; + xmit_data = xmit_pool->xmit_data; + + for (i = 0; i < xmit_pool->num_xmit_bufs; i++) { + rdma_io = &xmit_pool->xmit_bufs[i]; + rdma_io->index = i; + rdma_io->io.viport = data->parent; + rdma_io->io.routine = data_xmit_complete; + + rdma_io->list[0].lkey = data->mr->lkey; + rdma_io->list[1].lkey = data->mr->lkey; + rdma_io->io.swr.wr_id = (u64)rdma_io; + rdma_io->io.swr.sg_list = rdma_io->list; + rdma_io->io.swr.num_sge = 2; + rdma_io->io.swr.opcode = IB_WR_RDMA_WRITE; + rdma_io->io.swr.send_flags = IB_SEND_SIGNALED; + rdma_io->io.type = RDMA; + + rdma_io->data = xmit_data; + rdma_io->data_dma = xmit_dma; + + xmit_data += ALIGN(min_xmt_skb, VIPORT_TRAILER_ALIGNMENT); + xmit_dma += ALIGN(min_xmt_skb, VIPORT_TRAILER_ALIGNMENT); + rdma_io->trailer = (struct viport_trailer *)xmit_data; + rdma_io->trailer_dma = xmit_dma; + xmit_data += sizeof(struct viport_trailer); + xmit_dma += sizeof(struct viport_trailer); + } + + xmit_pool->rdma_rkey = data->mr->rkey; + xmit_pool->rdma_addr = xmit_pool->buf_pool_dma; +} + +static void data_init_free_bufs_swrs(struct data *data) +{ + struct rdma_io *rdma_io; + struct send_io *send_io; + + rdma_io = &data->free_bufs_io; + rdma_io->io.viport = data->parent; + rdma_io->io.routine = NULL; + + rdma_io->list[0].lkey = data->mr->lkey; + + rdma_io->io.swr.wr_id = (u64)rdma_io; + rdma_io->io.swr.sg_list = rdma_io->list; + rdma_io->io.swr.num_sge = 1; + rdma_io->io.swr.opcode = IB_WR_RDMA_WRITE; + rdma_io->io.swr.send_flags = IB_SEND_SIGNALED; + rdma_io->io.type = RDMA; + + send_io = &data->kick_io; + send_io->io.viport = data->parent; + send_io->io.routine = NULL; + + send_io->list.addr = data->region_data_dma; + send_io->list.length = 0; + send_io->list.lkey = data->mr->lkey; + + send_io->io.swr.wr_id = (u64)send_io; + send_io->io.swr.sg_list = &send_io->list; + send_io->io.swr.num_sge = 1; + send_io->io.swr.opcode = IB_WR_SEND; + send_io->io.swr.send_flags = IB_SEND_SIGNALED; + send_io->io.type = SEND; +} + +static int data_init_buf_pools(struct data *data) +{ + struct recv_pool *recv_pool = &data->recv_pool; + struct xmit_pool *xmit_pool = &data->xmit_pool; + struct viport *viport = data->parent; + + recv_pool->buf_pool_len = + sizeof(struct buff_pool_entry) * recv_pool->eioc_pool_sz; + + recv_pool->buf_pool = kzalloc(recv_pool->buf_pool_len, GFP_KERNEL); + + if (!recv_pool->buf_pool) { + DATA_ERROR("failed allocating %d bytes" + " for recv pool bufpool\n", + recv_pool->buf_pool_len); + goto failure; + } + + recv_pool->buf_pool_dma = + ib_dma_map_single(viport->config->ibdev, + recv_pool->buf_pool, recv_pool->buf_pool_len, + DMA_TO_DEVICE); + + if (ib_dma_mapping_error(viport->config->ibdev, recv_pool->buf_pool_dma)) { + DATA_ERROR("xmit buf_pool dma map error\n"); + goto free_recv_pool; + } + + xmit_pool->buf_pool_len = + sizeof(struct buff_pool_entry) * xmit_pool->pool_sz; + xmit_pool->buf_pool = kzalloc(xmit_pool->buf_pool_len, GFP_KERNEL); + + if (!xmit_pool->buf_pool) { + DATA_ERROR("failed allocating %d bytes" + " for xmit pool bufpool\n", + xmit_pool->buf_pool_len); + goto unmap_recv_pool; + } + + xmit_pool->buf_pool_dma = + ib_dma_map_single(viport->config->ibdev, + xmit_pool->buf_pool, xmit_pool->buf_pool_len, + DMA_FROM_DEVICE); + + if (ib_dma_mapping_error(viport->config->ibdev, xmit_pool->buf_pool_dma)) { + DATA_ERROR("xmit buf_pool dma map error\n"); + goto free_xmit_pool; + } + + xmit_pool->xmit_data = kzalloc(xmit_pool->xmitdata_len, GFP_KERNEL); + + if (!xmit_pool->xmit_data) { + DATA_ERROR("failed allocating %d bytes for xmit data\n", + xmit_pool->xmitdata_len); + goto unmap_xmit_pool; + } + + xmit_pool->xmitdata_dma = + ib_dma_map_single(viport->config->ibdev, + xmit_pool->xmit_data, xmit_pool->xmitdata_len, + DMA_TO_DEVICE); + + if (ib_dma_mapping_error(viport->config->ibdev, xmit_pool->xmitdata_dma)) { + DATA_ERROR("xmit data dma map error\n"); + goto free_xmit_data; + } + + return 0; + +free_xmit_data: + kfree(xmit_pool->xmit_data); +unmap_xmit_pool: + ib_dma_unmap_single(data->parent->config->ibdev, + xmit_pool->buf_pool_dma, + xmit_pool->buf_pool_len, DMA_FROM_DEVICE); +free_xmit_pool: + kfree(xmit_pool->buf_pool); +unmap_recv_pool: + ib_dma_unmap_single(data->parent->config->ibdev, + recv_pool->buf_pool_dma, + recv_pool->buf_pool_len, DMA_TO_DEVICE); +free_recv_pool: + kfree(recv_pool->buf_pool); +failure: + return -1; +} + +static void data_init_xmit_pool(struct data *data) +{ + struct xmit_pool *xmit_pool = &data->xmit_pool; + + xmit_pool->pool_sz = + be32_to_cpu(data->eioc_pool_parms.num_recv_pool_entries); + xmit_pool->buffer_sz = + be32_to_cpu(data->eioc_pool_parms.size_recv_pool_entry); + + xmit_pool->notify_count = 0; + xmit_pool->notify_bundle = data->config->notify_bundle; + xmit_pool->next_xmit_pool = 0; + xmit_pool->num_xmit_bufs = xmit_pool->notify_bundle * 2; + xmit_pool->next_xmit_buf = 0; + xmit_pool->last_comp_buf = xmit_pool->num_xmit_bufs - 1; + /* This assumes that data_init_recv_pool has been called + * before. + */ + data->max_mtu = MAX_PAYLOAD(min((data)->recv_pool.buffer_sz, + (data)->xmit_pool.buffer_sz)) - VLAN_ETH_HLEN; + + xmit_pool->kick_count = 0; + xmit_pool->kick_byte_count = 0; + + xmit_pool->send_kicks = + be32_to_cpu(data-> + eioc_pool_parms.num_recv_pool_entries_before_kick) + || be32_to_cpu(data-> + eioc_pool_parms.num_recv_pool_bytes_before_kick); + xmit_pool->kick_bundle = + be32_to_cpu(data-> + eioc_pool_parms.num_recv_pool_entries_before_kick); + xmit_pool->kick_byte_bundle = + be32_to_cpu(data-> + eioc_pool_parms.num_recv_pool_bytes_before_kick); + + xmit_pool->need_buffers = 1; + + xmit_pool->xmitdata_len = + BUFFER_SIZE(min_xmt_skb) * xmit_pool->num_xmit_bufs; +} + +static void data_init_recv_pool(struct data *data) +{ + struct recv_pool *recv_pool = &data->recv_pool; + + recv_pool->pool_sz = data->config->host_recv_pool_entries; + recv_pool->eioc_pool_sz = + be32_to_cpu(data->host_pool_parms.num_recv_pool_entries); + if (recv_pool->pool_sz > recv_pool->eioc_pool_sz) + recv_pool->pool_sz = + be32_to_cpu(data->host_pool_parms.num_recv_pool_entries); + + recv_pool->buffer_sz = + be32_to_cpu(data->host_pool_parms.size_recv_pool_entry); + + recv_pool->sz_free_bundle = + be32_to_cpu(data-> + host_pool_parms.free_recv_pool_entries_per_update); + recv_pool->num_free_bufs = 0; + recv_pool->num_posted_bufs = 0; + + recv_pool->next_full_buf = 0; + recv_pool->next_free_buf = 0; + recv_pool->kick_on_free = 0; +} + +int data_connect(struct data *data) +{ + struct xmit_pool *xmit_pool = &data->xmit_pool; + struct recv_pool *recv_pool = &data->recv_pool; + struct recv_io *recv_io; + unsigned int sz; + struct viport *viport = data->parent; + + DATA_FUNCTION("data_connect()\n"); + + /* Do not interchange the order of the functions + * called below as this will affect the MAX MTU + * calculation + */ + + data_init_recv_pool(data); + data_init_xmit_pool(data); + + sz = sizeof(struct rdma_dest) * recv_pool->pool_sz + + sizeof(struct recv_io) * data->config->num_recvs + + sizeof(struct rdma_io) * xmit_pool->num_xmit_bufs; + + data->local_storage = vmalloc(sz); + + if (!data->local_storage) { + DATA_ERROR("failed allocating %d bytes" + " local storage\n", sz); + goto out; + } + + memset(data->local_storage, 0, sz); + + recv_pool->recv_bufs = (struct rdma_dest *)data->local_storage; + sz = sizeof(struct rdma_dest) * recv_pool->pool_sz; + + recv_io = (struct recv_io *)(data->local_storage + sz); + sz += sizeof(struct recv_io) * data->config->num_recvs; + + xmit_pool->xmit_bufs = (struct rdma_io *)(data->local_storage + sz); + data->region_data = kzalloc(4, GFP_KERNEL); + + if (!data->region_data) { + DATA_ERROR("failed to alloc memory for region data\n"); + goto free_local_storage; + } + + data->region_data_dma = + ib_dma_map_single(viport->config->ibdev, + data->region_data, 4, DMA_BIDIRECTIONAL); + + if (ib_dma_mapping_error(viport->config->ibdev, data->region_data_dma)) { + DATA_ERROR("region data dma map error\n"); + goto free_region_data; + } + + if (data_init_buf_pools(data)) + goto unmap_region_data; + + data_init_free_bufs_swrs(data); + data_init_pool_work_reqs(data, recv_io); + + data_post_recvs(data); + + if (vnic_ib_cm_connect(&data->ib_conn)) + goto unmap_region_data; + + return 0; + +unmap_region_data: + ib_dma_unmap_single(data->parent->config->ibdev, + data->region_data_dma, 4, DMA_BIDIRECTIONAL); +free_region_data: + kfree(data->region_data); +free_local_storage: + vfree(data->local_storage); +out: + return -1; +} + +static void data_add_free_buffer(struct data *data, int index, + struct rdma_dest *rdma_dest) +{ + struct recv_pool *pool = &data->recv_pool; + struct buff_pool_entry *bpe; + dma_addr_t vaddr_dma; + + DATA_FUNCTION("data_add_free_buffer()\n"); + rdma_dest->trailer->connection_hash_and_valid = 0; + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); + + bpe = &pool->buf_pool[index]; + bpe->rkey = cpu_to_be32(data->mr->rkey); + vaddr_dma = ib_dma_map_single(data->parent->config->ibdev, + rdma_dest->data, pool->buffer_sz, + DMA_FROM_DEVICE); + if (ib_dma_mapping_error(data->parent->config->ibdev, vaddr_dma)) { + DATA_ERROR("rdma_dest->data dma map error\n"); + goto failure; + } + bpe->remote_addr = cpu_to_be64(vaddr_dma); + bpe->valid = (u32) (rdma_dest - &pool->recv_bufs[0]) + 1; + ++pool->num_free_bufs; +failure: + ib_dma_sync_single_for_device(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); +} + +/* NOTE: this routine is not reentrant */ +static void data_alloc_buffers(struct data *data, int initial_allocation) +{ + struct recv_pool *pool = &data->recv_pool; + struct rdma_dest *rdma_dest; + struct sk_buff *skb; + int index; + + DATA_FUNCTION("data_alloc_buffers()\n"); + index = ADD(pool->next_free_buf, pool->num_free_bufs, + pool->eioc_pool_sz); + + while (!list_empty(&pool->avail_recv_bufs)) { + rdma_dest = + list_entry(pool->avail_recv_bufs.next, + struct rdma_dest, list_ptrs); + if (!rdma_dest->skb) { + if (initial_allocation) + skb = alloc_skb(pool->buffer_sz + 2, + GFP_KERNEL); + else + skb = dev_alloc_skb(pool->buffer_sz + 2); + if (!skb) + break; + skb_reserve(skb, 2); + skb_put(skb, pool->buffer_sz); + rdma_dest->skb = skb; + rdma_dest->data = skb->data; + rdma_dest->trailer = + (struct viport_trailer *)(rdma_dest->data + + pool->buffer_sz - + sizeof(struct + viport_trailer)); + } + rdma_dest->trailer->connection_hash_and_valid = 0; + + list_del_init(&rdma_dest->list_ptrs); + + data_add_free_buffer(data, index, rdma_dest); + index = NEXT(index, pool->eioc_pool_sz); + } +} + +static void data_send_kick_message(struct data *data) +{ + struct xmit_pool *pool = &data->xmit_pool; + DATA_FUNCTION("data_send_kick_message()\n"); + /* stop timer for bundle_timeout */ + if (data->kick_timer_on) { + del_timer(&data->kick_timer); + data->kick_timer_on = 0; + } + pool->kick_count = 0; + pool->kick_byte_count = 0; + + /* TODO: keep track of when kick is outstanding, and + * don't reuse until complete + */ + if (vnic_ib_post_send(&data->ib_conn, &data->free_bufs_io.io)) { + DATA_ERROR("failed to post send\n"); + viport_failure(data->parent); + } +} + +static void data_send_free_recv_buffers(struct data *data) +{ + struct recv_pool *pool = &data->recv_pool; + struct ib_send_wr *swr = &data->free_bufs_io.io.swr; + + int bufs_sent = 0; + u64 rdma_addr; + u32 offset; + u32 sz; + unsigned int num_to_send, next_increment; + + DATA_FUNCTION("data_send_free_recv_buffers()\n"); + + for (num_to_send = pool->sz_free_bundle; + num_to_send <= pool->num_free_bufs; + num_to_send += pool->sz_free_bundle) { + /* handle multiple bundles as one when possible. */ + next_increment = num_to_send + pool->sz_free_bundle; + if ((next_increment <= pool->num_free_bufs) + && (pool->next_free_buf + next_increment <= + pool->eioc_pool_sz)) + continue; + + offset = pool->next_free_buf * + sizeof(struct buff_pool_entry); + sz = num_to_send * sizeof(struct buff_pool_entry); + rdma_addr = pool->eioc_rdma_addr + offset; + swr->sg_list->length = sz; + swr->sg_list->addr = pool->buf_pool_dma + offset; + swr->wr.rdma.remote_addr = rdma_addr; + + if (vnic_ib_post_send(&data->ib_conn, + &data->free_bufs_io.io)) { + DATA_ERROR("failed to post send\n"); + viport_failure(data->parent); + return; + } + INC(pool->next_free_buf, num_to_send, pool->eioc_pool_sz); + pool->num_free_bufs -= num_to_send; + pool->num_posted_bufs += num_to_send; + bufs_sent = 1; + } + + if (bufs_sent) { + if (pool->kick_on_free) + data_send_kick_message(data); + } + if (pool->num_posted_bufs == 0) { + struct vnic *vnic = data->parent->vnic; + unsigned long flags; + + spin_lock_irqsave(&vnic->current_path_lock, flags); + if (vnic->current_path == &vnic->primary_path) { + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + DATA_ERROR("%s: primary path: " + "unable to allocate receive buffers\n", + vnic->config->name); + } else { + if (vnic->current_path == &vnic->secondary_path) { + spin_unlock_irqrestore(&vnic->current_path_lock, + flags); + DATA_ERROR("%s: secondary path: " + "unable to allocate receive buffers\n", + vnic->config->name); + } else + spin_unlock_irqrestore(&vnic->current_path_lock, + flags); + } + data->ib_conn.state = IB_CONN_ERRORED; + viport_failure(data->parent); + } +} + +void data_connected(struct data *data) +{ + DATA_FUNCTION("data_connected()\n"); + data->free_bufs_io.io.swr.wr.rdma.rkey = + data->recv_pool.eioc_rdma_rkey; + data_alloc_buffers(data, 1); + data_send_free_recv_buffers(data); + data->connected = 1; +} + +void data_disconnect(struct data *data) +{ + struct xmit_pool *xmit_pool = &data->xmit_pool; + struct recv_pool *recv_pool = &data->recv_pool; + unsigned int i; + + DATA_FUNCTION("data_disconnect()\n"); + + data->connected = 0; + if (data->kick_timer_on) { + del_timer_sync(&data->kick_timer); + data->kick_timer_on = 0; + } + + if (ib_send_cm_dreq(data->ib_conn.cm_id, NULL, 0)) + DATA_ERROR("data CM DREQ sending failed\n"); + data->ib_conn.state = IB_CONN_DISCONNECTED; + + vnic_completion_cleanup(&data->ib_conn); + + for (i = 0; i < xmit_pool->num_xmit_bufs; i++) { + if (xmit_pool->xmit_bufs[i].skb) + dev_kfree_skb(xmit_pool->xmit_bufs[i].skb); + xmit_pool->xmit_bufs[i].skb = NULL; + + } + for (i = 0; i < recv_pool->pool_sz; i++) { + if (data->recv_pool.recv_bufs[i].skb) + dev_kfree_skb(recv_pool->recv_bufs[i].skb); + recv_pool->recv_bufs[i].skb = NULL; + } + vfree(data->local_storage); + if (data->region_data) { + ib_dma_unmap_single(data->parent->config->ibdev, + data->region_data_dma, 4, + DMA_BIDIRECTIONAL); + kfree(data->region_data); + } + + if (recv_pool->buf_pool) { + ib_dma_unmap_single(data->parent->config->ibdev, + recv_pool->buf_pool_dma, + recv_pool->buf_pool_len, DMA_TO_DEVICE); + kfree(recv_pool->buf_pool); + } + + if (xmit_pool->buf_pool) { + ib_dma_unmap_single(data->parent->config->ibdev, + xmit_pool->buf_pool_dma, + xmit_pool->buf_pool_len, DMA_FROM_DEVICE); + kfree(xmit_pool->buf_pool); + } + + if (xmit_pool->xmit_data) { + ib_dma_unmap_single(data->parent->config->ibdev, + xmit_pool->xmitdata_dma, + xmit_pool->xmitdata_len, DMA_TO_DEVICE); + kfree(xmit_pool->xmit_data); + } +} + +void data_cleanup(struct data *data) +{ + ib_destroy_cm_id(data->ib_conn.cm_id); + + /* Completion callback cleanup called again. + * This is to cleanup the threads in case there is an + * error before state LINK_DATACONNECT due to which + * data_disconnect is not called. + */ + vnic_completion_cleanup(&data->ib_conn); + ib_destroy_qp(data->ib_conn.qp); + ib_destroy_cq(data->ib_conn.cq); + ib_dereg_mr(data->mr); + +} + +static int data_alloc_xmit_buffer(struct data *data, struct sk_buff *skb, + struct buff_pool_entry **pp_bpe, + struct rdma_io **pp_rdma_io, + int *last) +{ + struct xmit_pool *pool = &data->xmit_pool; + unsigned long flags; + int ret; + + DATA_FUNCTION("data_alloc_xmit_buffer()\n"); + + spin_lock_irqsave(&data->xmit_buf_lock, flags); + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); + *last = 0; + *pp_rdma_io = &pool->xmit_bufs[pool->next_xmit_buf]; + *pp_bpe = &pool->buf_pool[pool->next_xmit_pool]; + + if ((*pp_bpe)->valid && pool->next_xmit_buf != + pool->last_comp_buf) { + INC(pool->next_xmit_buf, 1, pool->num_xmit_bufs); + INC(pool->next_xmit_pool, 1, pool->pool_sz); + if (!pool->buf_pool[pool->next_xmit_pool].valid) { + DATA_INFO("just used the last EIOU" + " receive buffer\n"); + *last = 1; + pool->need_buffers = 1; + vnic_stop_xmit(data->parent->vnic, + data->parent->parent); + data_kickreq_stats(data); + } else if (pool->next_xmit_buf == pool->last_comp_buf) { + DATA_INFO("just used our last xmit buffer\n"); + pool->need_buffers = 1; + vnic_stop_xmit(data->parent->vnic, + data->parent->parent); + } + (*pp_rdma_io)->skb = skb; + (*pp_bpe)->valid = 0; + ret = 0; + } else { + data_no_xmitbuf_stats(data); + DATA_ERROR("Out of xmit buffers\n"); + vnic_stop_xmit(data->parent->vnic, + data->parent->parent); + ret = -1; + } + + ib_dma_sync_single_for_device(data->parent->config->ibdev, + pool->buf_pool_dma, + pool->buf_pool_len, DMA_TO_DEVICE); + spin_unlock_irqrestore(&data->xmit_buf_lock, flags); + return ret; +} + +static void data_rdma_packet(struct data *data, struct buff_pool_entry *bpe, + struct rdma_io *rdma_io) +{ + struct ib_send_wr *swr; + struct sk_buff *skb; + dma_addr_t trailer_data_dma; + dma_addr_t skb_data_dma; + struct xmit_pool *xmit_pool = &data->xmit_pool; + struct viport *viport = data->parent; + u8 *d; + int len; + int fill_len; + + DATA_FUNCTION("data_rdma_packet()\n"); + swr = &rdma_io->io.swr; + skb = rdma_io->skb; + len = ALIGN(rdma_io->len, VIPORT_TRAILER_ALIGNMENT); + fill_len = len - skb->len; + + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + xmit_pool->xmitdata_dma, + xmit_pool->xmitdata_len, DMA_TO_DEVICE); + + d = (u8 *) rdma_io->trailer - fill_len; + trailer_data_dma = rdma_io->trailer_dma - fill_len; + memset(d, 0, fill_len); + + swr->sg_list[0].length = skb->len; + if (skb->len <= min_xmt_skb) { + memcpy(rdma_io->data, skb->data, skb->len); + swr->sg_list[0].lkey = data->mr->lkey; + swr->sg_list[0].addr = rdma_io->data_dma; + dev_kfree_skb_any(skb); + rdma_io->skb = NULL; + } else { + swr->sg_list[0].lkey = data->mr->lkey; + + skb_data_dma = ib_dma_map_single(viport->config->ibdev, + skb->data, skb->len, + DMA_TO_DEVICE); + + if (ib_dma_mapping_error(viport->config->ibdev, skb_data_dma)) { + DATA_ERROR("skb data dma map error\n"); + goto failure; + } + + rdma_io->skb_data_dma = skb_data_dma; + + swr->sg_list[0].addr = skb_data_dma; + skb_orphan(skb); + } + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + xmit_pool->buf_pool_dma, + xmit_pool->buf_pool_len, DMA_TO_DEVICE); + + swr->sg_list[1].addr = trailer_data_dma; + swr->sg_list[1].length = fill_len + sizeof(struct viport_trailer); + swr->sg_list[0].lkey = data->mr->lkey; + swr->wr.rdma.remote_addr = be64_to_cpu(bpe->remote_addr); + swr->wr.rdma.remote_addr += data->xmit_pool.buffer_sz; + swr->wr.rdma.remote_addr -= (sizeof(struct viport_trailer) + len); + swr->wr.rdma.rkey = be32_to_cpu(bpe->rkey); + + ib_dma_sync_single_for_device(data->parent->config->ibdev, + xmit_pool->buf_pool_dma, + xmit_pool->buf_pool_len, DMA_TO_DEVICE); + + /* If VNIC_FEAT_RDMA_IMMED is supported then change the work request + * opcode to IB_WR_RDMA_WRITE_WITH_IMM + */ + + if (data->parent->features_supported & VNIC_FEAT_RDMA_IMMED) { + swr->ex.imm_data = 0; + swr->opcode = IB_WR_RDMA_WRITE_WITH_IMM; + } + + data->xmit_pool.notify_count++; + if (data->xmit_pool.notify_count >= data->xmit_pool.notify_bundle) { + data->xmit_pool.notify_count = 0; + swr->send_flags = IB_SEND_SIGNALED; + } else { + swr->send_flags = 0; + } + ib_dma_sync_single_for_device(data->parent->config->ibdev, + xmit_pool->xmitdata_dma, + xmit_pool->xmitdata_len, DMA_TO_DEVICE); + if (vnic_ib_post_send(&data->ib_conn, &rdma_io->io)) { + DATA_ERROR("failed to post send for data RDMA write\n"); + viport_failure(data->parent); + goto failure; + } + + data_xmits_stats(data); +failure: + ib_dma_sync_single_for_device(data->parent->config->ibdev, + xmit_pool->xmitdata_dma, + xmit_pool->xmitdata_len, DMA_TO_DEVICE); +} + +static void data_kick_timeout_handler(unsigned long arg) +{ + struct data *data = (struct data *)arg; + + DATA_FUNCTION("data_kick_timeout_handler()\n"); + data->kick_timer_on = 0; + data_send_kick_message(data); +} + +int data_xmit_packet(struct data *data, struct sk_buff *skb) +{ + struct xmit_pool *pool = &data->xmit_pool; + struct rdma_io *rdma_io; + struct buff_pool_entry *bpe; + struct viport_trailer *trailer; + unsigned int sz = skb->len; + int last; + + DATA_FUNCTION("data_xmit_packet()\n"); + if (sz > pool->buffer_sz) { + DATA_ERROR("outbound packet too large, size = %d\n", sz); + return -1; + } + + if (data_alloc_xmit_buffer(data, skb, &bpe, &rdma_io, &last)) { + DATA_ERROR("error in allocating data xmit buffer\n"); + return -1; + } + + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + pool->xmitdata_dma, pool->xmitdata_len, + DMA_TO_DEVICE); + trailer = rdma_io->trailer; + + memset(trailer, 0, sizeof *trailer); + memcpy(trailer->dest_mac_addr, skb->data, ETH_ALEN); + + if (skb->sk) + trailer->connection_hash_and_valid = 0x40 | + ((be16_to_cpu(inet_sk(skb->sk)->sport) + + be16_to_cpu(inet_sk(skb->sk)->dport)) & 0x3f); + + trailer->connection_hash_and_valid |= CHV_VALID; + + if ((sz > 16) && (*(__be16 *) (skb->data + 12) == + __constant_cpu_to_be16(ETH_P_8021Q))) { + trailer->vlan = *(__be16 *) (skb->data + 14); + memmove(skb->data + 4, skb->data, 12); + skb_pull(skb, 4); + sz -= 4; + trailer->pkt_flags |= PF_VLAN_INSERT; + } + if (last) + trailer->pkt_flags |= PF_KICK; + if (sz < ETH_ZLEN) { + /* EIOU requires all packets to be + * of ethernet minimum packet size. + */ + trailer->data_length = __constant_cpu_to_be16(ETH_ZLEN); + rdma_io->len = ETH_ZLEN; + } else { + trailer->data_length = cpu_to_be16(sz); + rdma_io->len = sz; + } + + if (skb->ip_summed == CHECKSUM_PARTIAL) { + trailer->tx_chksum_flags = TX_CHKSUM_FLAGS_CHECKSUM_V4 + | TX_CHKSUM_FLAGS_IP_CHECKSUM + | TX_CHKSUM_FLAGS_TCP_CHECKSUM + | TX_CHKSUM_FLAGS_UDP_CHECKSUM; + } + + ib_dma_sync_single_for_device(data->parent->config->ibdev, + pool->xmitdata_dma, pool->xmitdata_len, + DMA_TO_DEVICE); + data_rdma_packet(data, bpe, rdma_io); + + if (pool->send_kicks) { + /* EIOC needs kicks to inform it of sent packets */ + pool->kick_count++; + pool->kick_byte_count += sz; + if ((pool->kick_count >= pool->kick_bundle) + || (pool->kick_byte_count >= pool->kick_byte_bundle)) { + data_send_kick_message(data); + } else if (pool->kick_count == 1) { + init_timer(&data->kick_timer); + /* timeout_before_kick is in usec */ + data->kick_timer.expires = + msecs_to_jiffies(be32_to_cpu(data-> + eioc_pool_parms.timeout_before_kick) * 1000) + + jiffies; + data->kick_timer.data = (unsigned long)data; + data->kick_timer.function = data_kick_timeout_handler; + add_timer(&data->kick_timer); + data->kick_timer_on = 1; + } + } + return 0; +} + +static void data_check_xmit_buffers(struct data *data) +{ + struct xmit_pool *pool = &data->xmit_pool; + unsigned long flags; + + DATA_FUNCTION("data_check_xmit_buffers()\n"); + spin_lock_irqsave(&data->xmit_buf_lock, flags); + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); + + if (data->xmit_pool.need_buffers + && pool->buf_pool[pool->next_xmit_pool].valid + && pool->next_xmit_buf != pool->last_comp_buf) { + data->xmit_pool.need_buffers = 0; + vnic_restart_xmit(data->parent->vnic, + data->parent->parent); + DATA_INFO("there are free xmit buffers\n"); + } + ib_dma_sync_single_for_device(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); + + spin_unlock_irqrestore(&data->xmit_buf_lock, flags); +} + +static struct sk_buff *data_recv_to_skbuff(struct data *data, + struct rdma_dest *rdma_dest) +{ + struct viport_trailer *trailer; + struct sk_buff *skb = NULL; + int start; + unsigned int len; + u8 rx_chksum_flags; + + DATA_FUNCTION("data_recv_to_skbuff()\n"); + trailer = rdma_dest->trailer; + start = data_offset(data, trailer); + len = data_len(data, trailer); + + if (len <= min_rcv_skb) + skb = dev_alloc_skb(len + VLAN_HLEN + 2); + /* leave room for VLAN header and alignment */ + if (skb) { + skb_reserve(skb, VLAN_HLEN + 2); + memcpy(skb->data, rdma_dest->data + start, len); + skb_put(skb, len); + } else { + skb = rdma_dest->skb; + rdma_dest->skb = NULL; + rdma_dest->trailer = NULL; + rdma_dest->data = NULL; + skb_pull(skb, start); + skb_trim(skb, len); + } + + rx_chksum_flags = trailer->rx_chksum_flags; + DATA_INFO("rx_chksum_flags = %d, LOOP = %c, IP = %c," + " TCP = %c, UDP = %c\n", + rx_chksum_flags, + (rx_chksum_flags & RX_CHKSUM_FLAGS_LOOPBACK) ? 'Y' : 'N', + (rx_chksum_flags & RX_CHKSUM_FLAGS_IP_CHECKSUM_SUCCEEDED) ? 'Y' + : (rx_chksum_flags & RX_CHKSUM_FLAGS_IP_CHECKSUM_FAILED) ? 'N' : + '-', + (rx_chksum_flags & RX_CHKSUM_FLAGS_TCP_CHECKSUM_SUCCEEDED) ? 'Y' + : (rx_chksum_flags & RX_CHKSUM_FLAGS_TCP_CHECKSUM_FAILED) ? 'N' : + '-', + (rx_chksum_flags & RX_CHKSUM_FLAGS_UDP_CHECKSUM_SUCCEEDED) ? 'Y' + : (rx_chksum_flags & RX_CHKSUM_FLAGS_UDP_CHECKSUM_FAILED) ? 'N' : + '-'); + + if ((rx_chksum_flags & RX_CHKSUM_FLAGS_LOOPBACK) + || ((rx_chksum_flags & RX_CHKSUM_FLAGS_IP_CHECKSUM_SUCCEEDED) + && ((rx_chksum_flags & RX_CHKSUM_FLAGS_TCP_CHECKSUM_SUCCEEDED) + || (rx_chksum_flags & + RX_CHKSUM_FLAGS_UDP_CHECKSUM_SUCCEEDED)))) + skb->ip_summed = CHECKSUM_UNNECESSARY; + else + skb->ip_summed = CHECKSUM_NONE; + + if ((trailer->pkt_flags & PF_VLAN_INSERT) && + !(data->parent->features_supported & VNIC_FEAT_IGNORE_VLAN)) { + u8 *rv; + + rv = skb_push(skb, 4); + memmove(rv, rv + 4, 12); + *(__be16 *) (rv + 12) = __constant_cpu_to_be16(ETH_P_8021Q); + if (trailer->pkt_flags & PF_PVID_OVERRIDDEN) + *(__be16 *) (rv + 14) = trailer->vlan & + __constant_cpu_to_be16(0xF000); + else + *(__be16 *) (rv + 14) = trailer->vlan; + } + + return skb; +} + +static int data_incoming_recv(struct data *data) +{ + struct recv_pool *pool = &data->recv_pool; + struct rdma_dest *rdma_dest; + struct viport_trailer *trailer; + struct buff_pool_entry *bpe; + struct sk_buff *skb; + dma_addr_t vaddr_dma; + + DATA_FUNCTION("data_incoming_recv()\n"); + if (pool->next_full_buf == pool->next_free_buf) + return -1; + bpe = &pool->buf_pool[pool->next_full_buf]; + vaddr_dma = be64_to_cpu(bpe->remote_addr); + rdma_dest = &pool->recv_bufs[bpe->valid - 1]; + trailer = rdma_dest->trailer; + + if (!trailer + || !(trailer->connection_hash_and_valid & CHV_VALID)) + return -1; + + /* received a packet */ + if (trailer->pkt_flags & PF_KICK) + pool->kick_on_free = 1; + + skb = data_recv_to_skbuff(data, rdma_dest); + + if (skb) { + vnic_recv_packet(data->parent->vnic, + data->parent->parent, skb); + list_add(&rdma_dest->list_ptrs, &pool->avail_recv_bufs); + } + + ib_dma_unmap_single(data->parent->config->ibdev, + vaddr_dma, pool->buffer_sz, + DMA_FROM_DEVICE); + ib_dma_sync_single_for_cpu(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); + + bpe->valid = 0; + ib_dma_sync_single_for_device(data->parent->config->ibdev, + pool->buf_pool_dma, pool->buf_pool_len, + DMA_TO_DEVICE); + + INC(pool->next_full_buf, 1, pool->eioc_pool_sz); + pool->num_posted_bufs--; + data_recvs_stats(data); + return 0; +} + +static void data_received_kick(struct io *io) +{ + struct data *data = &io->viport->data; + unsigned long flags; + + DATA_FUNCTION("data_received_kick()\n"); + data_note_kickrcv_time(); + spin_lock_irqsave(&data->recv_ios_lock, flags); + list_add(&io->list_ptrs, &data->recv_ios); + spin_unlock_irqrestore(&data->recv_ios_lock, flags); + data_post_recvs(data); + data_rcvkicks_stats(data); + data_check_xmit_buffers(data); + + while (!data_incoming_recv(data)); + + if (data->connected) { + data_alloc_buffers(data, 0); + data_send_free_recv_buffers(data); + } +} + +static void data_xmit_complete(struct io *io) +{ + struct rdma_io *rdma_io = (struct rdma_io *)io; + struct data *data = &io->viport->data; + struct xmit_pool *pool = &data->xmit_pool; + struct sk_buff *skb; + + DATA_FUNCTION("data_xmit_complete()\n"); + + if (rdma_io->skb) + ib_dma_unmap_single(data->parent->config->ibdev, + rdma_io->skb_data_dma, rdma_io->skb->len, + DMA_TO_DEVICE); + + while (pool->last_comp_buf != rdma_io->index) { + INC(pool->last_comp_buf, 1, pool->num_xmit_bufs); + skb = pool->xmit_bufs[pool->last_comp_buf].skb; + if (skb) + dev_kfree_skb_any(skb); + pool->xmit_bufs[pool->last_comp_buf].skb = NULL; + } + + data_check_xmit_buffers(data); +} + +static int mc_data_alloc_skb(struct ud_recv_io *recv_io, u32 len, + int initial_allocation) +{ + struct sk_buff *skb; + struct mc_data *mc_data = &recv_io->io.viport->mc_data; + + DATA_FUNCTION("mc_data_alloc_skb\n"); + if (initial_allocation) + skb = alloc_skb(len, GFP_KERNEL); + else + skb = alloc_skb(len, GFP_ATOMIC); + if (!skb) { + DATA_ERROR("failed to alloc MULTICAST skb\n"); + return -1; + } + skb_put(skb, len); + recv_io->skb = skb; + + recv_io->skb_data_dma = ib_dma_map_single( + recv_io->io.viport->config->ibdev, + skb->data, skb->len, + DMA_FROM_DEVICE); + + if (ib_dma_mapping_error(recv_io->io.viport->config->ibdev, + recv_io->skb_data_dma)) { + DATA_ERROR("skb data dma map error\n"); + dev_kfree_skb(skb); + return -1; + } + + recv_io->list[0].addr = recv_io->skb_data_dma; + recv_io->list[0].length = sizeof(struct ib_grh); + recv_io->list[0].lkey = mc_data->mr->lkey; + + recv_io->list[1].addr = recv_io->skb_data_dma + sizeof(struct ib_grh); + recv_io->list[1].length = len - sizeof(struct ib_grh); + recv_io->list[1].lkey = mc_data->mr->lkey; + + recv_io->io.rwr.wr_id = (u64)&recv_io->io; + recv_io->io.rwr.sg_list = recv_io->list; + recv_io->io.rwr.num_sge = 2; + recv_io->io.rwr.next = NULL; + + return 0; +} + +static int mc_data_alloc_buffers(struct mc_data *mc_data) +{ + unsigned int i, num; + struct ud_recv_io *bufs = NULL, *recv_io; + + DATA_FUNCTION("mc_data_alloc_buffers\n"); + if (!mc_data->skb_len) { + unsigned int len; + /* align multicast msg buffer on viport_trailer boundary */ + len = (MCAST_MSG_SIZE + VIPORT_TRAILER_ALIGNMENT - 1) & + (~((unsigned int)VIPORT_TRAILER_ALIGNMENT - 1)); + /* + * Add size of grh and trailer - + * note, we don't need a + 4 for vlan because we have room in + * netbuf for grh & trailer and we'll strip them both, so there + * will be room enough to handle the 4 byte insertion for vlan. + */ + len += sizeof(struct ib_grh) + + sizeof(struct viport_trailer); + mc_data->skb_len = len; + DATA_INFO("mc_data->skb_len %d (sizes:%d %d)\n", + len, (int)sizeof(struct ib_grh), + (int)sizeof(struct viport_trailer)); + } + mc_data->recv_len = sizeof(struct ud_recv_io) * mc_data->num_recvs; + bufs = kmalloc(mc_data->recv_len, GFP_KERNEL); + if (!bufs) { + DATA_ERROR("failed to allocate MULTICAST buffers size:%d\n", + mc_data->recv_len); + return -1; + } + DATA_INFO("allocated num_recvs:%d recv_len:%d \n", + mc_data->num_recvs, mc_data->recv_len); + for (num = 0; num < mc_data->num_recvs; num++) { + recv_io = &bufs[num]; + recv_io->len = mc_data->skb_len; + recv_io->io.type = RECV_UD; + recv_io->io.viport = mc_data->parent; + recv_io->io.routine = mc_data_recv_routine; + + if (mc_data_alloc_skb(recv_io, mc_data->skb_len, 1)) { + for (i = 0; i < num; i++) { + recv_io = &bufs[i]; + ib_dma_unmap_single(recv_io->io.viport->config->ibdev, + recv_io->skb_data_dma, + recv_io->skb->len, + DMA_FROM_DEVICE); + dev_kfree_skb(recv_io->skb); + } + kfree(bufs); + return -1; + } + list_add_tail(&recv_io->io.list_ptrs, + &mc_data->avail_recv_ios_list); + } + mc_data->recv_ios = bufs; + return 0; +} + +void vnic_mc_data_cleanup(struct mc_data *mc_data) +{ + unsigned int num; + + DATA_FUNCTION("vnic_mc_data_cleanup()\n"); + vnic_completion_cleanup(&mc_data->ib_conn); + if (!IS_ERR(mc_data->ib_conn.qp)) { + ib_destroy_qp(mc_data->ib_conn.qp); + mc_data->ib_conn.qp = (struct ib_qp *)ERR_PTR(-EINVAL); + } + if (!IS_ERR(mc_data->ib_conn.cq)) { + ib_destroy_cq(mc_data->ib_conn.cq); + mc_data->ib_conn.cq = (struct ib_cq *)ERR_PTR(-EINVAL); + } + if (mc_data->recv_ios) { + for (num = 0; num < mc_data->num_recvs; num++) { + if (mc_data->recv_ios[num].skb) + dev_kfree_skb(mc_data->recv_ios[num].skb); + mc_data->recv_ios[num].skb = NULL; + } + kfree(mc_data->recv_ios); + mc_data->recv_ios = (struct ud_recv_io *)NULL; + } + if (mc_data->mr) { + ib_dereg_mr(mc_data->mr); + mc_data->mr = (struct ib_mr *)NULL; + } + DATA_FUNCTION("vnic_mc_data_cleanup done\n"); + +} + +int mc_data_init(struct mc_data *mc_data, struct viport *viport, + struct data_config *config, struct ib_pd *pd) +{ + DATA_FUNCTION("mc_data_init()\n"); + + mc_data->num_recvs = viport->data.config->num_recvs; + + INIT_LIST_HEAD(&mc_data->avail_recv_ios_list); + spin_lock_init(&mc_data->recv_lock); + + mc_data->parent = viport; + mc_data->config = config; + + mc_data->ib_conn.cm_id = NULL; + mc_data->ib_conn.viport = viport; + mc_data->ib_conn.ib_config = &config->ib_config; + mc_data->ib_conn.state = IB_CONN_UNINITTED; + mc_data->ib_conn.callback_thread = NULL; + mc_data->ib_conn.callback_thread_end = 0; + + if (vnic_ib_mc_init(mc_data, viport, pd, + &config->ib_config)) { + DATA_ERROR("vnic_ib_mc_init failed\n"); + goto failure; + } + mc_data->mr = ib_get_dma_mr(pd, + IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_WRITE); + if (IS_ERR(mc_data->mr)) { + DATA_ERROR("failed to register memory for" + " mc_data connection\n"); + goto destroy_conn; + } + + if (mc_data_alloc_buffers(mc_data)) + goto dereg_mr; + + mc_data_post_recvs(mc_data); + if (vnic_ib_mc_mod_qp_to_rts(mc_data->ib_conn.qp)) + goto dereg_mr; + + return 0; + +dereg_mr: + ib_dereg_mr(mc_data->mr); + mc_data->mr = (struct ib_mr *)NULL; +destroy_conn: + vnic_completion_cleanup(&mc_data->ib_conn); + ib_destroy_qp(mc_data->ib_conn.qp); + mc_data->ib_conn.qp = (struct ib_qp *)ERR_PTR(-EINVAL); + ib_destroy_cq(mc_data->ib_conn.cq); + mc_data->ib_conn.cq = (struct ib_cq *)ERR_PTR(-EINVAL); +failure: + return -1; +} + +static void mc_data_post_recvs(struct mc_data *mc_data) +{ + unsigned long flags; + int i = 0; + DATA_FUNCTION("mc_data_post_recvs\n"); + spin_lock_irqsave(&mc_data->recv_lock, flags); + while (!list_empty(&mc_data->avail_recv_ios_list)) { + struct io *io = list_entry(mc_data->avail_recv_ios_list.next, + struct io, list_ptrs); + struct ud_recv_io *recv_io = + container_of(io, struct ud_recv_io, io); + list_del(&recv_io->io.list_ptrs); + spin_unlock_irqrestore(&mc_data->recv_lock, flags); + if (vnic_ib_mc_post_recv(mc_data, &recv_io->io)) { + viport_failure(mc_data->parent); + return; + } + spin_lock_irqsave(&mc_data->recv_lock, flags); + i++; + } + DATA_INFO("mcdata posted %d %p\n", i, &mc_data->avail_recv_ios_list); + spin_unlock_irqrestore(&mc_data->recv_lock, flags); +} + +static void mc_data_recv_routine(struct io *io) +{ + struct sk_buff *skb; + struct ib_grh *grh; + struct viport_trailer *trailer; + struct mc_data *mc_data; + unsigned long flags; + struct ud_recv_io *recv_io = container_of(io, struct ud_recv_io, io); + union ib_gid_cpu sgid; + + DATA_FUNCTION("mc_data_recv_routine\n"); + skb = recv_io->skb; + grh = (struct ib_grh *)skb->data; + mc_data = &recv_io->io.viport->mc_data; + + ib_dma_unmap_single(recv_io->io.viport->config->ibdev, + recv_io->skb_data_dma, recv_io->skb->len, + DMA_FROM_DEVICE); + + /* first - check if we've got our own mc packet */ + /* convert sgid from host to cpu form before comparing */ + bswap_ib_gid(&grh->sgid, &sgid); + if (cpu_to_be64(sgid.global.interface_id) == + io->viport->config->path_info.path.sgid.global.interface_id) { + DATA_ERROR("dropping - our mc packet\n"); + dev_kfree_skb(skb); + } else { + /* GRH is at head and trailer at end. Remove GRH from head. */ + trailer = (struct viport_trailer *) + (skb->data + recv_io->len - + sizeof(struct viport_trailer)); + skb_pull(skb, sizeof(struct ib_grh)); + if (trailer->connection_hash_and_valid & CHV_VALID) { + mc_data_recv_to_skbuff(io->viport, skb, trailer); + vnic_recv_packet(io->viport->vnic, io->viport->parent, + skb); + vnic_multicast_recv_pkt_stats(io->viport->vnic); + } else { + DATA_ERROR("dropping - no CHV_VALID in HashAndValid\n"); + dev_kfree_skb(skb); + } + } + recv_io->skb = NULL; + if (mc_data_alloc_skb(recv_io, mc_data->skb_len, 0)) + return; + + spin_lock_irqsave(&mc_data->recv_lock, flags); + list_add_tail(&recv_io->io.list_ptrs, &mc_data->avail_recv_ios_list); + spin_unlock_irqrestore(&mc_data->recv_lock, flags); + mc_data_post_recvs(mc_data); + return; +} + +static void mc_data_recv_to_skbuff(struct viport *viport, struct sk_buff *skb, + struct viport_trailer *trailer) +{ + u8 rx_chksum_flags = trailer->rx_chksum_flags; + + /* drop alignment bytes at start */ + skb_pull(skb, trailer->data_alignment_offset); + /* drop excess from end */ + skb_trim(skb, __be16_to_cpu(trailer->data_length)); + + if ((rx_chksum_flags & RX_CHKSUM_FLAGS_LOOPBACK) + || ((rx_chksum_flags & RX_CHKSUM_FLAGS_IP_CHECKSUM_SUCCEEDED) + && ((rx_chksum_flags & RX_CHKSUM_FLAGS_TCP_CHECKSUM_SUCCEEDED) + || (rx_chksum_flags & + RX_CHKSUM_FLAGS_UDP_CHECKSUM_SUCCEEDED)))) + skb->ip_summed = CHECKSUM_UNNECESSARY; + else + skb->ip_summed = CHECKSUM_NONE; + + if ((trailer->pkt_flags & PF_VLAN_INSERT) && + !(viport->features_supported & VNIC_FEAT_IGNORE_VLAN)) { + u8 *rv; + + /* insert VLAN id between source & length */ + DATA_INFO("VLAN adjustment\n"); + rv = skb_push(skb, 4); + memmove(rv, rv + 4, 12); + *(__be16 *) (rv + 12) = __constant_cpu_to_be16(ETH_P_8021Q); + if (trailer->pkt_flags & PF_PVID_OVERRIDDEN) + /* + * Indicates VLAN is 0 but we keep the protocol id. + */ + *(__be16 *) (rv + 14) = trailer->vlan & + __constant_cpu_to_be16(0xF000); + else + *(__be16 *) (rv + 14) = trailer->vlan; + DATA_INFO("vlan:%x\n", *(int *)(rv+14)); + } + + return; +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_data.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_data.h new file mode 100644 index 0000000..365251d --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_data.h @@ -0,0 +1,206 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_DATA_H_INCLUDED +#define VNIC_DATA_H_INCLUDED + +#include + +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS +#include +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ + +#include "vnic_ib.h" +#include "vnic_control_pkt.h" +#include "vnic_trailer.h" + +struct rdma_dest { + struct list_head list_ptrs; + struct sk_buff *skb; + u8 *data; + struct viport_trailer *trailer __attribute__((aligned(32))); +}; + +struct buff_pool_entry { + __be64 remote_addr; + __be32 rkey; + u32 valid; +}; + +struct recv_pool { + u32 buffer_sz; + u32 pool_sz; + u32 eioc_pool_sz; + u32 eioc_rdma_rkey; + u64 eioc_rdma_addr; + u32 next_full_buf; + u32 next_free_buf; + u32 num_free_bufs; + u32 num_posted_bufs; + u32 sz_free_bundle; + int kick_on_free; + struct buff_pool_entry *buf_pool; + dma_addr_t buf_pool_dma; + int buf_pool_len; + struct rdma_dest *recv_bufs; + struct list_head avail_recv_bufs; +}; + +struct xmit_pool { + u32 buffer_sz; + u32 pool_sz; + u32 notify_count; + u32 notify_bundle; + u32 next_xmit_buf; + u32 last_comp_buf; + u32 num_xmit_bufs; + u32 next_xmit_pool; + u32 kick_count; + u32 kick_byte_count; + u32 kick_bundle; + u32 kick_byte_bundle; + int need_buffers; + int send_kicks; + uint32_t rdma_rkey; + u64 rdma_addr; + struct buff_pool_entry *buf_pool; + dma_addr_t buf_pool_dma; + int buf_pool_len; + struct rdma_io *xmit_bufs; + u8 *xmit_data; + dma_addr_t xmitdata_dma; + int xmitdata_len; +}; + +struct data { + struct viport *parent; + struct data_config *config; + struct ib_mr *mr; + struct vnic_ib_conn ib_conn; + u8 *local_storage; + struct vnic_recv_pool_config host_pool_parms; + struct vnic_recv_pool_config eioc_pool_parms; + struct recv_pool recv_pool; + struct xmit_pool xmit_pool; + u8 *region_data; + dma_addr_t region_data_dma; + struct rdma_io free_bufs_io; + struct send_io kick_io; + struct list_head recv_ios; + spinlock_t recv_ios_lock; + spinlock_t xmit_buf_lock; + int kick_timer_on; + int connected; + u16 max_mtu; + struct timer_list kick_timer; + struct completion done; +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS + struct { + u32 xmit_num; + u32 recv_num; + u32 free_buf_sends; + u32 free_buf_num; + u32 free_buf_min; + u32 kick_recvs; + u32 kick_reqs; + u32 no_xmit_bufs; + cycles_t no_xmit_buf_time; + } statistics; +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ +}; + +struct mc_data { + struct viport *parent; + struct data_config *config; + struct ib_mr *mr; + struct vnic_ib_conn ib_conn; + + u32 num_recvs; + u32 skb_len; + spinlock_t recv_lock; + int recv_len; + struct ud_recv_io *recv_ios; + struct list_head avail_recv_ios_list; +}; + +int data_init(struct data *data, struct viport *viport, + struct data_config *config, struct ib_pd *pd); + +int data_connect(struct data *data); +void data_connected(struct data *data); +void data_disconnect(struct data *data); + +int data_xmit_packet(struct data *data, struct sk_buff *skb); + +void data_cleanup(struct data *data); + +#define data_is_connected(data) \ + (vnic_ib_conn_connected(&((data)->ib_conn))) +#define data_path_id(data) ((data)->config->path_id) +#define data_eioc_pool(data) (&(data)->eioc_pool_parms) +#define data_host_pool(data) (&(data)->host_pool_parms) +#define data_eioc_pool_min(data) (&(data)->config->eioc_min) +#define data_host_pool_min(data) (&(data)->config->host_min) +#define data_eioc_pool_max(data) (&(data)->config->eioc_max) +#define data_host_pool_max(data) (&(data)->config->host_max) +#define data_local_pool_addr(data) ((data)->xmit_pool.rdma_addr) +#define data_local_pool_rkey(data) ((data)->xmit_pool.rdma_rkey) +#define data_remote_pool_addr(data) (&(data)->recv_pool.eioc_rdma_addr) +#define data_remote_pool_rkey(data) (&(data)->recv_pool.eioc_rdma_rkey) + +#define data_max_mtu(data) ((data)->max_mtu) + + +#define data_len(data, trailer) be16_to_cpu(trailer->data_length) +#define data_offset(data, trailer) \ + ((data)->recv_pool.buffer_sz - sizeof(struct viport_trailer) \ + - ALIGN(data_len((data), (trailer)), VIPORT_TRAILER_ALIGNMENT) \ + + (trailer->data_alignment_offset)) + +/* the following macros manipulate ring buffer indexes. + * the ring buffer size must be a power of 2. + */ +#define ADD(index, increment, size) (((index) + (increment))&((size) - 1)) +#define NEXT(index, size) ADD(index, 1, size) +#define INC(index, increment, size) (index) = ADD(index, increment, size) + +/* this is max multicast msg embedded will send */ +#define MCAST_MSG_SIZE \ + (2048 - sizeof(struct ib_grh) - sizeof(struct viport_trailer)) + +int mc_data_init(struct mc_data *mc_data, struct viport *viport, + struct data_config *config, + struct ib_pd *pd); + +void vnic_mc_data_cleanup(struct mc_data *mc_data); + +#endif /* VNIC_DATA_H_INCLUDED */ diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h new file mode 100644 index 0000000..dd8a073 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_trailer.h @@ -0,0 +1,103 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_TRAILER_H_INCLUDED +#define VNIC_TRAILER_H_INCLUDED + +/* pkt_flags values */ +enum { + PF_CHASH_VALID = 0x01, + PF_IPSEC_VALID = 0x02, + PF_TCP_SEGMENT = 0x04, + PF_KICK = 0x08, + PF_VLAN_INSERT = 0x10, + PF_PVID_OVERRIDDEN = 0x20, + PF_FCS_INCLUDED = 0x40, + PF_FORCE_ROUTE = 0x80 +}; + +/* tx_chksum_flags values */ +enum { + TX_CHKSUM_FLAGS_CHECKSUM_V4 = 0x01, + TX_CHKSUM_FLAGS_CHECKSUM_V6 = 0x02, + TX_CHKSUM_FLAGS_TCP_CHECKSUM = 0x04, + TX_CHKSUM_FLAGS_UDP_CHECKSUM = 0x08, + TX_CHKSUM_FLAGS_IP_CHECKSUM = 0x10 +}; + +/* rx_chksum_flags values */ +enum { + RX_CHKSUM_FLAGS_TCP_CHECKSUM_FAILED = 0x01, + RX_CHKSUM_FLAGS_UDP_CHECKSUM_FAILED = 0x02, + RX_CHKSUM_FLAGS_IP_CHECKSUM_FAILED = 0x04, + RX_CHKSUM_FLAGS_TCP_CHECKSUM_SUCCEEDED = 0x08, + RX_CHKSUM_FLAGS_UDP_CHECKSUM_SUCCEEDED = 0x10, + RX_CHKSUM_FLAGS_IP_CHECKSUM_SUCCEEDED = 0x20, + RX_CHKSUM_FLAGS_LOOPBACK = 0x40, + RX_CHKSUM_FLAGS_RESERVED = 0x80 +}; + +/* connection_hash_and_valid values */ +enum { + CHV_VALID = 0x80, + CHV_HASH_MASH = 0x7f +}; + +struct viport_trailer { + s8 data_alignment_offset; + u8 rndis_header_length; /* reserved for use by edp */ + __be16 data_length; + u8 pkt_flags; + u8 tx_chksum_flags; + u8 rx_chksum_flags; + u8 ip_sec_flags; + u32 tcp_seq_no; + u32 ip_sec_offload_handle; + u32 ip_sec_next_offload_handle; + u8 dest_mac_addr[6]; + __be16 vlan; + u16 time_stamp; + u8 origin; + u8 connection_hash_and_valid; +}; + +#define VIPORT_TRAILER_ALIGNMENT 32 + +#define BUFFER_SIZE(len) \ + (sizeof(struct viport_trailer) + \ + ALIGN((len), VIPORT_TRAILER_ALIGNMENT)) + +#define MAX_PAYLOAD(len) \ + ALIGN_DOWN((len) - sizeof(struct viport_trailer), \ + VIPORT_TRAILER_ALIGNMENT) + +#endif /* VNIC_TRAILER_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:05:16 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:05:16 -0400 Subject: [ofa-general] [PATCH v4 06/14] QLogic VNIC: IB core stack interaction In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210516.11186.36378.stgit@dale> From: Ramachandra K The patch implements the interaction of the QLogic VNIC driver with the underlying core infiniband stack. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c | 1057 ++++++++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h | 207 +++++ 2 files changed, 1264 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c new file mode 100644 index 0000000..1957e90 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_ib.c @@ -0,0 +1,1057 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_data.h" +#include "vnic_config.h" +#include "vnic_ib.h" +#include "vnic_viport.h" +#include "vnic_sys.h" +#include "vnic_main.h" +#include "vnic_stats.h" + +static int vnic_ib_inited; +static void vnic_add_one(struct ib_device *device); +static void vnic_remove_one(struct ib_device *device); +static int vnic_defer_completion(void *ptr); + +static int vnic_ib_mc_init_qp(struct mc_data *mc_data, + struct vnic_ib_config *config, + struct ib_pd *pd, + struct viport_config *viport_config); + +static struct ib_client vnic_client = { + .name = "vnic", + .add = vnic_add_one, + .remove = vnic_remove_one +}; + +struct ib_sa_client vnic_sa_client; + +int vnic_ib_init(void) +{ + int ret = -1; + + IB_FUNCTION("vnic_ib_init()\n"); + + /* class has to be registered before + * calling ib_register_client() because, that call + * will trigger vnic_add_port() which will register + * class_device for the port with the parent class + * as vnic_class + */ + ret = class_register(&vnic_class); + if (ret) { + printk(KERN_ERR PFX "couldn't register class" + " infiniband_qlgc_vnic; error %d", ret); + goto out; + } + + ib_sa_register_client(&vnic_sa_client); + ret = ib_register_client(&vnic_client); + if (ret) { + printk(KERN_ERR PFX "couldn't register IB client;" + " error %d", ret); + goto err_ib_reg; + } + + interface_dev.dev.class = &vnic_class; + interface_dev.dev.release = vnic_release_dev; + snprintf(interface_dev.dev.bus_id, + BUS_ID_SIZE, "interfaces"); + init_completion(&interface_dev.released); + ret = device_register(&interface_dev.dev); + if (ret) { + printk(KERN_ERR PFX "couldn't register class interfaces;" + " error %d", ret); + goto err_class_dev; + } + + ret = device_create_file(&interface_dev.dev, + &dev_attr_delete_vnic); + if (ret) { + printk(KERN_ERR PFX "couldn't create class file" + " 'delete_vnic'; error %d", ret); + goto err_class_file; + } + + ret = device_create_file(&interface_dev.dev, + &dev_attr_create_vnic); + if (ret) { + printk(KERN_ERR PFX "couldn't create class file" + " 'create_vnic'; error %d", ret); + goto err_create_vnic; + } + + vnic_ib_inited = 1; + + return ret; +err_create_vnic: + device_remove_file(&interface_dev.dev, &dev_attr_delete_vnic); +err_class_file: + device_unregister(&interface_dev.dev); +err_class_dev: + ib_unregister_client(&vnic_client); +err_ib_reg: + ib_sa_unregister_client(&vnic_sa_client); + class_unregister(&vnic_class); +out: + return ret; +} + +static struct vnic_ib_port *vnic_add_port(struct vnic_ib_device *device, + u8 port_num) +{ + struct vnic_ib_port *port; + + port = kzalloc(sizeof *port, GFP_KERNEL); + if (!port) + return NULL; + + init_completion(&port->pdev_info.released); + port->dev = device; + port->port_num = port_num; + + port->pdev_info.dev.class = &vnic_class; + port->pdev_info.dev.parent = NULL; + port->pdev_info.dev.release = vnic_release_dev; + snprintf(port->pdev_info.dev.bus_id, BUS_ID_SIZE, + "vnic-%s-%d", device->dev->name, port_num); + + if (device_register(&port->pdev_info.dev)) + goto free_port; + + if (device_create_file(&port->pdev_info.dev, + &dev_attr_create_primary)) + goto err_class; + if (device_create_file(&port->pdev_info.dev, + &dev_attr_create_secondary)) + goto err_class; + + return port; +err_class: + device_unregister(&port->pdev_info.dev); +free_port: + kfree(port); + + return NULL; +} + +static void vnic_add_one(struct ib_device *device) +{ + struct vnic_ib_device *vnic_dev; + struct vnic_ib_port *port; + int s, e, p; + + vnic_dev = kmalloc(sizeof *vnic_dev, GFP_KERNEL); + if (!vnic_dev) + return; + + vnic_dev->dev = device; + INIT_LIST_HEAD(&vnic_dev->port_list); + + if (device->node_type == RDMA_NODE_IB_SWITCH) { + s = 0; + e = 0; + + } else { + s = 1; + e = device->phys_port_cnt; + + } + + for (p = s; p <= e; p++) { + port = vnic_add_port(vnic_dev, p); + if (port) + list_add_tail(&port->list, &vnic_dev->port_list); + } + + ib_set_client_data(device, &vnic_client, vnic_dev); + +} + +static void vnic_remove_one(struct ib_device *device) +{ + struct vnic_ib_device *vnic_dev; + struct vnic_ib_port *port, *tmp_port; + + vnic_dev = ib_get_client_data(device, &vnic_client); + list_for_each_entry_safe(port, tmp_port, + &vnic_dev->port_list, list) { + device_unregister(&port->pdev_info.dev); + /* + * wait for sysfs entries to go away, so that no new vnics + * are created + */ + wait_for_completion(&port->pdev_info.released); + kfree(port); + + } + kfree(vnic_dev); + + /* TODO Only those vnic interfaces associated with + * the HCA whose remove event is called should be freed + * Currently all the vnic interfaces are freed + */ + + while (!list_empty(&vnic_list)) { + struct vnic *vnic = + list_entry(vnic_list.next, struct vnic, list_ptrs); + vnic_free(vnic); + } + + vnic_npevent_cleanup(); + viport_cleanup(); + +} + +void vnic_ib_cleanup(void) +{ + IB_FUNCTION("vnic_ib_cleanup()\n"); + + if (!vnic_ib_inited) + return; + + device_remove_file(&interface_dev.dev, &dev_attr_delete_vnic); + device_remove_file(&interface_dev.dev, &dev_attr_create_vnic); + + device_unregister(&interface_dev.dev); + wait_for_completion(&interface_dev.released); + + ib_unregister_client(&vnic_client); + ib_sa_unregister_client(&vnic_sa_client); + class_unregister(&vnic_class); +} + +static void vnic_path_rec_completion(int status, + struct ib_sa_path_rec *pathrec, + void *context) +{ + struct vnic_ib_path_info *p = context; + p->status = status; + if (!status) + p->path = *pathrec; + + complete(&p->done); +} + +int vnic_ib_get_path(struct netpath *netpath, struct vnic *vnic) +{ + struct viport_config *config = netpath->viport->config; + int ret = 0; + + init_completion(&config->path_info.done); + IB_INFO("Using SA path rec get time out value of %d\n", + config->sa_path_rec_get_timeout); + config->path_info.path_query_id = + ib_sa_path_rec_get(&vnic_sa_client, + config->ibdev, + config->port, + &config->path_info.path, + IB_SA_PATH_REC_DGID | + IB_SA_PATH_REC_SGID | + IB_SA_PATH_REC_NUMB_PATH | + IB_SA_PATH_REC_PKEY, + config->sa_path_rec_get_timeout, + GFP_KERNEL, + vnic_path_rec_completion, + &config->path_info, + &config->path_info.path_query); + + if (config->path_info.path_query_id < 0) { + IB_ERROR("SA path record query failed; error %d\n", + config->path_info.path_query_id); + ret = config->path_info.path_query_id; + goto out; + } + + wait_for_completion(&config->path_info.done); + + if (config->path_info.status < 0) { + printk(KERN_WARNING PFX "connection not available to dgid " + "%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x", + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[0]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[2]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[4]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[6]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[8]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[10]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[12]), + (int)be16_to_cpu(*(__be16 *) &config->path_info.path. + dgid.raw[14])); + + if (config->path_info.status == -ETIMEDOUT) + printk(KERN_INFO " path query timed out\n"); + else if (config->path_info.status == -EIO) + printk(KERN_INFO " path query sending error\n"); + else + printk(KERN_INFO " error %d\n", + config->path_info.status); + + ret = config->path_info.status; + } +out: + if (ret) + netpath_timer(netpath, vnic->config->no_path_timeout); + + return ret; +} + +static inline void vnic_ib_handle_completions(struct ib_wc *wc, + struct vnic_ib_conn *ib_conn, + u32 *comp_num, + cycles_t *comp_time) +{ + struct io *io; + + io = (struct io *)(wc->wr_id); + vnic_ib_comp_stats(ib_conn, comp_num); + if (wc->status) { + IB_INFO("completion error wc.status %d" + " wc.opcode %d vendor err 0x%x\n", + wc->status, wc->opcode, wc->vendor_err); + } else if (io) { + vnic_ib_io_stats(io, ib_conn, *comp_time); + if (io->type == RECV_UD) { + struct ud_recv_io *recv_io = + container_of(io, struct ud_recv_io, io); + recv_io->len = wc->byte_len; + } + if (io->routine) + (*io->routine) (io); + } +} + +static void ib_qp_event(struct ib_event *event, void *context) +{ + IB_ERROR("QP event %d\n", event->event); +} + +static void vnic_ib_completion(struct ib_cq *cq, void *ptr) +{ + struct vnic_ib_conn *ib_conn = ptr; + unsigned long flags; + int compl_received; + struct ib_wc wc; + cycles_t comp_time; + u32 comp_num = 0; + + /* for multicast, cm_id is NULL, so skip that test */ + if (ib_conn->cm_id && + (ib_conn->state != IB_CONN_CONNECTED)) + return; + + /* Check if completion processing is taking place in thread + * If not then process completions in this handler, + * else set compl_received if not set, to indicate that + * there are more completions to process in thread. + */ + + spin_lock_irqsave(&ib_conn->compl_received_lock, flags); + compl_received = ib_conn->compl_received; + spin_unlock_irqrestore(&ib_conn->compl_received_lock, flags); + + if (ib_conn->in_thread || compl_received) { + if (!compl_received) { + spin_lock_irqsave(&ib_conn->compl_received_lock, flags); + ib_conn->compl_received = 1; + spin_unlock_irqrestore(&ib_conn->compl_received_lock, + flags); + } + wake_up(&(ib_conn->callback_wait_queue)); + } else { + vnic_ib_note_comptime_stats(&comp_time); + vnic_ib_callback_stats(ib_conn); + ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); + while (ib_poll_cq(cq, 1, &wc) > 0) { + vnic_ib_handle_completions(&wc, ib_conn, &comp_num, + &comp_time); + if (ib_conn->cm_id && + ib_conn->state != IB_CONN_CONNECTED) + break; + + /* If we get more completions than the completion limit + * defer completion to the thread + */ + if ((!ib_conn->in_thread) && + (comp_num >= ib_conn->ib_config->completion_limit)) { + ib_conn->in_thread = 1; + spin_lock_irqsave( + &ib_conn->compl_received_lock, flags); + ib_conn->compl_received = 1; + spin_unlock_irqrestore( + &ib_conn->compl_received_lock, flags); + wake_up(&(ib_conn->callback_wait_queue)); + break; + } + + } + vnic_ib_maxio_stats(ib_conn, comp_num); + } +} + +static int vnic_ib_mod_qp_to_rts(struct ib_cm_id *cm_id, + struct vnic_ib_conn *ib_conn) +{ + int attr_mask = 0; + int ret; + struct ib_qp_attr *qp_attr = NULL; + + qp_attr = kmalloc(sizeof *qp_attr, GFP_KERNEL); + if (!qp_attr) + return -ENOMEM; + + qp_attr->qp_state = IB_QPS_RTR; + + ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask); + if (ret) + goto out; + + ret = ib_modify_qp(ib_conn->qp, qp_attr, attr_mask); + if (ret) + goto out; + + IB_INFO("QP RTR\n"); + + qp_attr->qp_state = IB_QPS_RTS; + + ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask); + if (ret) + goto out; + + ret = ib_modify_qp(ib_conn->qp, qp_attr, attr_mask); + if (ret) + goto out; + + IB_INFO("QP RTS\n"); + + ret = ib_send_cm_rtu(cm_id, NULL, 0); + if (ret) + goto out; +out: + kfree(qp_attr); + return ret; +} + +int vnic_ib_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) +{ + struct vnic_ib_conn *ib_conn = cm_id->context; + struct viport *viport = ib_conn->viport; + int err = 0; + + switch (event->event) { + case IB_CM_REQ_ERROR: + IB_ERROR("sending CM REQ failed\n"); + err = 1; + viport->retry = 1; + break; + case IB_CM_REP_RECEIVED: + IB_INFO("CM REP recvd\n"); + if (vnic_ib_mod_qp_to_rts(cm_id, ib_conn)) + err = 1; + else { + ib_conn->state = IB_CONN_CONNECTED; + vnic_ib_connected_time_stats(ib_conn); + IB_INFO("RTU SENT\n"); + } + break; + case IB_CM_REJ_RECEIVED: + printk(KERN_ERR PFX " CM rejected control connection\n"); + if (event->param.rej_rcvd.reason == + IB_CM_REJ_INVALID_SERVICE_ID) + printk(KERN_ERR "reason: invalid service ID. " + "IOCGUID value specified may be incorrect\n"); + else + printk(KERN_ERR "reason code : 0x%x\n", + event->param.rej_rcvd.reason); + + err = 1; + viport->retry = 1; + break; + case IB_CM_MRA_RECEIVED: + IB_INFO("CM MRA received\n"); + break; + + case IB_CM_DREP_RECEIVED: + IB_INFO("CM DREP recvd\n"); + ib_conn->state = IB_CONN_DISCONNECTED; + break; + + case IB_CM_TIMEWAIT_EXIT: + IB_ERROR("CM timewait exit\n"); + err = 1; + break; + + default: + IB_INFO("unhandled CM event %d\n", event->event); + break; + + } + + if (err) { + ib_conn->state = IB_CONN_DISCONNECTED; + viport_failure(viport); + } + + viport_kick(viport); + return 0; +} + + +int vnic_ib_cm_connect(struct vnic_ib_conn *ib_conn) +{ + struct ib_cm_req_param *req = NULL; + struct viport *viport; + int ret = -1; + + if (!vnic_ib_conn_initted(ib_conn)) { + IB_ERROR("IB Connection out of state for CM connect (%d)\n", + ib_conn->state); + return -EINVAL; + } + + vnic_ib_conntime_stats(ib_conn); + req = kzalloc(sizeof *req, GFP_KERNEL); + if (!req) + return -ENOMEM; + + viport = ib_conn->viport; + + req->primary_path = &viport->config->path_info.path; + req->alternate_path = NULL; + req->qp_num = ib_conn->qp->qp_num; + req->qp_type = ib_conn->qp->qp_type; + req->service_id = ib_conn->ib_config->service_id; + req->private_data = &ib_conn->ib_config->conn_data; + req->private_data_len = sizeof(struct vnic_connection_data); + req->flow_control = 1; + + get_random_bytes(&req->starting_psn, 4); + req->starting_psn &= 0xffffff; + + /* + * Both responder_resources and initiator_depth are set to zero + * as we do not need RDMA read. + * + * They also must be set to zero, otherwise data connections + * are rejected by VEx. + */ + req->responder_resources = 0; + req->initiator_depth = 0; + req->remote_cm_response_timeout = 20; + req->local_cm_response_timeout = 20; + req->retry_count = ib_conn->ib_config->retry_count; + req->rnr_retry_count = ib_conn->ib_config->rnr_retry_count; + req->max_cm_retries = 15; + + ib_conn->state = IB_CONN_CONNECTING; + + ret = ib_send_cm_req(ib_conn->cm_id, req); + + kfree(req); + + if (ret) { + IB_ERROR("CM REQ sending failed; error %d \n", ret); + ib_conn->state = IB_CONN_DISCONNECTED; + } + + return ret; +} + +static int vnic_ib_init_qp(struct vnic_ib_conn *ib_conn, + struct vnic_ib_config *config, + struct ib_pd *pd, + struct viport_config *viport_config) +{ + struct ib_qp_init_attr *init_attr; + struct ib_qp_attr *attr; + int ret; + + init_attr = kzalloc(sizeof *init_attr, GFP_KERNEL); + if (!init_attr) + return -ENOMEM; + + init_attr->event_handler = ib_qp_event; + init_attr->cap.max_send_wr = config->num_sends; + init_attr->cap.max_recv_wr = config->num_recvs; + init_attr->cap.max_recv_sge = config->recv_scatter; + init_attr->cap.max_send_sge = config->send_gather; + init_attr->sq_sig_type = IB_SIGNAL_ALL_WR; + init_attr->qp_type = IB_QPT_RC; + init_attr->send_cq = ib_conn->cq; + init_attr->recv_cq = ib_conn->cq; + + ib_conn->qp = ib_create_qp(pd, init_attr); + + if (IS_ERR(ib_conn->qp)) { + ret = -1; + IB_ERROR("could not create QP\n"); + goto free_init_attr; + } + + attr = kmalloc(sizeof *attr, GFP_KERNEL); + if (!attr) { + ret = -ENOMEM; + goto destroy_qp; + } + + ret = ib_find_pkey(viport_config->ibdev, viport_config->port, + be16_to_cpu(viport_config->path_info.path.pkey), + &attr->pkey_index); + if (ret) { + printk(KERN_WARNING PFX "ib_find_pkey() failed; " + "error %d\n", ret); + goto freeattr; + } + + attr->qp_state = IB_QPS_INIT; + attr->qp_access_flags = IB_ACCESS_REMOTE_WRITE; + attr->port_num = viport_config->port; + + ret = ib_modify_qp(ib_conn->qp, attr, + IB_QP_STATE | + IB_QP_PKEY_INDEX | + IB_QP_ACCESS_FLAGS | IB_QP_PORT); + if (ret) { + printk(KERN_WARNING PFX "could not modify QP; error %d \n", + ret); + goto freeattr; + } + + kfree(attr); + kfree(init_attr); + return ret; + +freeattr: + kfree(attr); +destroy_qp: + ib_destroy_qp(ib_conn->qp); +free_init_attr: + kfree(init_attr); + return ret; +} + +int vnic_ib_conn_init(struct vnic_ib_conn *ib_conn, struct viport *viport, + struct ib_pd *pd, struct vnic_ib_config *config) +{ + struct viport_config *viport_config = viport->config; + int ret = -1; + unsigned int cq_size = config->num_sends + config->num_recvs; + + + if (!vnic_ib_conn_uninitted(ib_conn)) { + IB_ERROR("IB Connection out of state for init (%d)\n", + ib_conn->state); + return -EINVAL; + } + + ib_conn->cq = ib_create_cq(viport_config->ibdev, vnic_ib_completion, +#ifdef BUILD_FOR_OFED_1_2 + NULL, ib_conn, cq_size); +#else + NULL, ib_conn, cq_size, 0); +#endif + if (IS_ERR(ib_conn->cq)) { + IB_ERROR("could not create CQ\n"); + goto out; + } + + IB_INFO("cq created %p %d\n", ib_conn->cq, cq_size); + ib_req_notify_cq(ib_conn->cq, IB_CQ_NEXT_COMP); + init_waitqueue_head(&(ib_conn->callback_wait_queue)); + init_completion(&(ib_conn->callback_thread_exit)); + + spin_lock_init(&ib_conn->compl_received_lock); + + ib_conn->callback_thread = kthread_run(vnic_defer_completion, ib_conn, + "qlgc_vnic_def_compl"); + if (IS_ERR(ib_conn->callback_thread)) { + IB_ERROR("Could not create vnic_callback_thread;" + " error %d\n", (int) PTR_ERR(ib_conn->callback_thread)); + ib_conn->callback_thread = NULL; + goto destroy_cq; + } + + ret = vnic_ib_init_qp(ib_conn, config, pd, viport_config); + + if (ret) + goto destroy_thread; + + spin_lock_init(&ib_conn->conn_lock); + ib_conn->state = IB_CONN_INITTED; + + return ret; + +destroy_thread: + vnic_completion_cleanup(ib_conn); +destroy_cq: + ib_destroy_cq(ib_conn->cq); +out: + return ret; +} + +int vnic_ib_post_recv(struct vnic_ib_conn *ib_conn, struct io *io) +{ + cycles_t post_time; + struct ib_recv_wr *bad_wr; + int ret = -1; + unsigned long flags; + + IB_FUNCTION("vnic_ib_post_recv()\n"); + + spin_lock_irqsave(&ib_conn->conn_lock, flags); + + if (!vnic_ib_conn_initted(ib_conn) && + !vnic_ib_conn_connected(ib_conn)) { + ret = -EINVAL; + goto out; + } + + vnic_ib_pre_rcvpost_stats(ib_conn, io, &post_time); + io->type = RECV; + ret = ib_post_recv(ib_conn->qp, &io->rwr, &bad_wr); + if (ret) { + IB_ERROR("error in posting rcv wr; error %d\n", ret); + ib_conn->state = IB_CONN_ERRORED; + goto out; + } + + vnic_ib_post_rcvpost_stats(ib_conn, post_time); +out: + spin_unlock_irqrestore(&ib_conn->conn_lock, flags); + return ret; + +} + +int vnic_ib_post_send(struct vnic_ib_conn *ib_conn, struct io *io) +{ + cycles_t post_time; + unsigned long flags; + struct ib_send_wr *bad_wr; + int ret = -1; + + IB_FUNCTION("vnic_ib_post_send()\n"); + + spin_lock_irqsave(&ib_conn->conn_lock, flags); + if (!vnic_ib_conn_connected(ib_conn)) { + IB_ERROR("IB Connection out of state for" + " posting sends (%d)\n", ib_conn->state); + goto out; + } + + vnic_ib_pre_sendpost_stats(io, &post_time); + if (io->swr.opcode == IB_WR_RDMA_WRITE) + io->type = RDMA; + else + io->type = SEND; + + ret = ib_post_send(ib_conn->qp, &io->swr, &bad_wr); + if (ret) { + IB_ERROR("error in posting send wr; error %d\n", ret); + ib_conn->state = IB_CONN_ERRORED; + goto out; + } + + vnic_ib_post_sendpost_stats(ib_conn, io, post_time); +out: + spin_unlock_irqrestore(&ib_conn->conn_lock, flags); + return ret; +} + +static int vnic_defer_completion(void *ptr) +{ + struct vnic_ib_conn *ib_conn = ptr; + struct ib_wc wc; + struct ib_cq *cq = ib_conn->cq; + cycles_t comp_time; + u32 comp_num = 0; + unsigned long flags; + + while (!ib_conn->callback_thread_end) { + wait_event_interruptible(ib_conn->callback_wait_queue, + ib_conn->compl_received || + ib_conn->callback_thread_end); + ib_conn->in_thread = 1; + spin_lock_irqsave(&ib_conn->compl_received_lock, flags); + ib_conn->compl_received = 0; + spin_unlock_irqrestore(&ib_conn->compl_received_lock, flags); + if (ib_conn->cm_id && + ib_conn->state != IB_CONN_CONNECTED) + goto out_thread; + + vnic_ib_note_comptime_stats(&comp_time); + vnic_ib_callback_stats(ib_conn); + ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); + while (ib_poll_cq(cq, 1, &wc) > 0) { + vnic_ib_handle_completions(&wc, ib_conn, &comp_num, + &comp_time); + if (ib_conn->cm_id && + ib_conn->state != IB_CONN_CONNECTED) + break; + } + vnic_ib_maxio_stats(ib_conn, comp_num); +out_thread: + ib_conn->in_thread = 0; + } + complete_and_exit(&(ib_conn->callback_thread_exit), 0); + return 0; +} + +void vnic_completion_cleanup(struct vnic_ib_conn *ib_conn) +{ + if (ib_conn->callback_thread) { + ib_conn->callback_thread_end = 1; + wake_up(&(ib_conn->callback_wait_queue)); + wait_for_completion(&(ib_conn->callback_thread_exit)); + ib_conn->callback_thread = NULL; + } +} + +int vnic_ib_mc_init(struct mc_data *mc_data, struct viport *viport, + struct ib_pd *pd, struct vnic_ib_config *config) +{ + struct viport_config *viport_config = viport->config; + int ret = -1; + unsigned int cq_size = config->num_recvs; /* recvs only */ + + IB_FUNCTION("vnic_ib_mc_init\n"); + + mc_data->ib_conn.cq = ib_create_cq(viport_config->ibdev, vnic_ib_completion, +#ifdef BUILD_FOR_OFED_1_2 + NULL, &mc_data->ib_conn, cq_size); +#else + NULL, &mc_data->ib_conn, cq_size, 0); +#endif + if (IS_ERR(mc_data->ib_conn.cq)) { + IB_ERROR("ib_create_cq failed\n"); + goto out; + } + IB_INFO("mc cq created %p %d\n", mc_data->ib_conn.cq, cq_size); + + ret = ib_req_notify_cq(mc_data->ib_conn.cq, IB_CQ_NEXT_COMP); + if (ret) { + IB_ERROR("ib_req_notify_cq failed %x \n", ret); + goto destroy_cq; + } + + init_waitqueue_head(&(mc_data->ib_conn.callback_wait_queue)); + init_completion(&(mc_data->ib_conn.callback_thread_exit)); + + spin_lock_init(&mc_data->ib_conn.compl_received_lock); + mc_data->ib_conn.callback_thread = kthread_run(vnic_defer_completion, + &mc_data->ib_conn, + "qlgc_vnic_mc_def_compl"); + if (IS_ERR(mc_data->ib_conn.callback_thread)) { + IB_ERROR("Could not create vnic_callback_thread for MULTICAST;" + " error %d\n", + (int) PTR_ERR(mc_data->ib_conn.callback_thread)); + mc_data->ib_conn.callback_thread = NULL; + goto destroy_cq; + } + IB_INFO("callback_thread created\n"); + + ret = vnic_ib_mc_init_qp(mc_data, config, pd, viport_config); + if (ret) + goto destroy_thread; + + spin_lock_init(&mc_data->ib_conn.conn_lock); + mc_data->ib_conn.state = IB_CONN_INITTED; /* stays in this state */ + + return ret; + +destroy_thread: + vnic_completion_cleanup(&mc_data->ib_conn); +destroy_cq: + ib_destroy_cq(mc_data->ib_conn.cq); + mc_data->ib_conn.cq = (struct ib_cq *)ERR_PTR(-EINVAL); +out: + return ret; +} + +static int vnic_ib_mc_init_qp(struct mc_data *mc_data, + struct vnic_ib_config *config, + struct ib_pd *pd, + struct viport_config *viport_config) +{ + struct ib_qp_init_attr *init_attr; + struct ib_qp_attr *qp_attr; + int ret; + + IB_FUNCTION("vnic_ib_mc_init_qp\n"); + + if (!mc_data->ib_conn.cq) { + IB_ERROR("cq is null\n"); + return -ENOMEM; + } + + init_attr = kzalloc(sizeof *init_attr, GFP_KERNEL); + if (!init_attr) { + IB_ERROR("failed to alloc init_attr\n"); + return -ENOMEM; + } + + init_attr->cap.max_recv_wr = config->num_recvs; + init_attr->cap.max_send_wr = 1; + init_attr->cap.max_recv_sge = 2; + init_attr->cap.max_send_sge = 1; + + /* Completion for all work requests. */ + init_attr->sq_sig_type = IB_SIGNAL_ALL_WR; + + init_attr->qp_type = IB_QPT_UD; + + init_attr->send_cq = mc_data->ib_conn.cq; + init_attr->recv_cq = mc_data->ib_conn.cq; + + IB_INFO("creating qp %d \n", config->num_recvs); + + mc_data->ib_conn.qp = ib_create_qp(pd, init_attr); + + if (IS_ERR(mc_data->ib_conn.qp)) { + ret = -1; + IB_ERROR("could not create QP\n"); + goto free_init_attr; + } + + qp_attr = kzalloc(sizeof *qp_attr, GFP_KERNEL); + if (!qp_attr) { + ret = -ENOMEM; + goto destroy_qp; + } + + qp_attr->qp_state = IB_QPS_INIT; + qp_attr->port_num = viport_config->port; + qp_attr->qkey = IOC_NUMBER(be64_to_cpu(viport_config->ioc_guid)); + qp_attr->pkey_index = 0; + /* cannot set access flags for UD qp + qp_attr->qp_access_flags = IB_ACCESS_REMOTE_WRITE; */ + + IB_INFO("port_num:%d qkey:%d pkey:%d\n", qp_attr->port_num, + qp_attr->qkey, qp_attr->pkey_index); + ret = ib_modify_qp(mc_data->ib_conn.qp, qp_attr, + IB_QP_STATE | + IB_QP_PKEY_INDEX | + IB_QP_QKEY | + + /* cannot set this for UD + IB_QP_ACCESS_FLAGS | */ + + IB_QP_PORT); + if (ret) { + IB_ERROR("ib_modify_qp to INIT failed %d \n", ret); + goto free_qp_attr; + } + + kfree(qp_attr); + kfree(init_attr); + return ret; + +free_qp_attr: + kfree(qp_attr); +destroy_qp: + ib_destroy_qp(mc_data->ib_conn.qp); + mc_data->ib_conn.qp = ERR_PTR(-EINVAL); +free_init_attr: + kfree(init_attr); + return ret; +} + +int vnic_ib_mc_mod_qp_to_rts(struct ib_qp *qp) +{ + int ret; + struct ib_qp_attr *qp_attr = NULL; + + IB_FUNCTION("vnic_ib_mc_mod_qp_to_rts\n"); + qp_attr = kmalloc(sizeof *qp_attr, GFP_KERNEL); + if (!qp_attr) + return -ENOMEM; + + memset(qp_attr, 0, sizeof *qp_attr); + qp_attr->qp_state = IB_QPS_RTR; + + ret = ib_modify_qp(qp, qp_attr, IB_QP_STATE); + if (ret) { + IB_ERROR("ib_modify_qp to RTR failed %d\n", ret); + goto out; + } + IB_INFO("MC QP RTR\n"); + + memset(qp_attr, 0, sizeof *qp_attr); + qp_attr->qp_state = IB_QPS_RTS; + qp_attr->sq_psn = 0; + + ret = ib_modify_qp(qp, qp_attr, IB_QP_STATE | IB_QP_SQ_PSN); + if (ret) { + IB_ERROR("ib_modify_qp to RTS failed %d\n", ret); + goto out; + } + IB_INFO("MC QP RTS\n"); + + return 0; + +out: + kfree(qp_attr); + return -1; +} + +int vnic_ib_mc_post_recv(struct mc_data *mc_data, struct io *io) +{ + cycles_t post_time; + struct ib_recv_wr *bad_wr; + int ret = -1; + + IB_FUNCTION("vnic_ib_mc_post_recv()\n"); + + vnic_ib_pre_rcvpost_stats(&mc_data->ib_conn, io, &post_time); + io->type = RECV_UD; + ret = ib_post_recv(mc_data->ib_conn.qp, &io->rwr, &bad_wr); + if (ret) { + IB_ERROR("error in posting rcv wr; error %d\n", ret); + goto out; + } + vnic_ib_post_rcvpost_stats(&mc_data->ib_conn, post_time); + +out: + return ret; +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h new file mode 100644 index 0000000..6303ae6 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_ib.h @@ -0,0 +1,207 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_IB_H_INCLUDED +#define VNIC_IB_H_INCLUDED + +#include +#include +#include +#include +#include +#include + +#include "vnic_sys.h" +#include "vnic_netpath.h" +#define PFX "qlgc_vnic: " + +struct io; +typedef void (comp_routine_t) (struct io *io); + +enum vnic_ib_conn_state { + IB_CONN_UNINITTED = 0, + IB_CONN_INITTED = 1, + IB_CONN_CONNECTING = 2, + IB_CONN_CONNECTED = 3, + IB_CONN_DISCONNECTED = 4, + IB_CONN_ERRORED = 5 +}; + +struct vnic_ib_conn { + struct viport *viport; + struct vnic_ib_config *ib_config; + spinlock_t conn_lock; + enum vnic_ib_conn_state state; + struct ib_qp *qp; + struct ib_cq *cq; + struct ib_cm_id *cm_id; + int callback_thread_end; + struct task_struct *callback_thread; + wait_queue_head_t callback_wait_queue; + u32 in_thread; + u32 compl_received; + struct completion callback_thread_exit; + spinlock_t compl_received_lock; +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS + struct { + cycles_t connection_time; + cycles_t rdma_post_time; + u32 rdma_post_ios; + cycles_t rdma_comp_time; + u32 rdma_comp_ios; + cycles_t send_post_time; + u32 send_post_ios; + cycles_t send_comp_time; + u32 send_comp_ios; + cycles_t recv_post_time; + u32 recv_post_ios; + cycles_t recv_comp_time; + u32 recv_comp_ios; + u32 num_ios; + u32 num_callbacks; + u32 max_ios; + } statistics; +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ +}; + +struct vnic_ib_path_info { + struct ib_sa_path_rec path; + struct ib_sa_query *path_query; + int path_query_id; + int status; + struct completion done; +}; + +struct vnic_ib_device { + struct ib_device *dev; + struct list_head port_list; +}; + +struct vnic_ib_port { + struct vnic_ib_device *dev; + u8 port_num; + struct dev_info pdev_info; + struct list_head list; +}; + +struct io { + struct list_head list_ptrs; + struct viport *viport; + comp_routine_t *routine; + struct ib_recv_wr rwr; + struct ib_send_wr swr; +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS + cycles_t time; +#endif /* CONFIG_INFINIBAND_QLGC_VNIC_STATS */ + enum {RECV, RDMA, SEND, RECV_UD} type; +}; + +struct rdma_io { + struct io io; + struct ib_sge list[2]; + u16 index; + u16 len; + u8 *data; + dma_addr_t data_dma; + struct sk_buff *skb; + dma_addr_t skb_data_dma; + struct viport_trailer *trailer; + dma_addr_t trailer_dma; +}; + +struct send_io { + struct io io; + struct ib_sge list; + u8 *virtual_addr; +}; + +struct recv_io { + struct io io; + struct ib_sge list; + u8 *virtual_addr; +}; + +struct ud_recv_io { + struct io io; + u16 len; + dma_addr_t skb_data_dma; + struct ib_sge list[2]; /* one for grh and other for rest of pkt. */ + struct sk_buff *skb; +}; + +int vnic_ib_init(void); +void vnic_ib_cleanup(void); +void vnic_completion_cleanup(struct vnic_ib_conn *ib_conn); + +struct vnic; +int vnic_ib_get_path(struct netpath *netpath, struct vnic *vnic); +int vnic_ib_conn_init(struct vnic_ib_conn *ib_conn, struct viport *viport, + struct ib_pd *pd, struct vnic_ib_config *config); + +int vnic_ib_post_recv(struct vnic_ib_conn *ib_conn, struct io *io); +int vnic_ib_post_send(struct vnic_ib_conn *ib_conn, struct io *io); +int vnic_ib_cm_connect(struct vnic_ib_conn *ib_conn); +int vnic_ib_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event); + +#define vnic_ib_conn_uninitted(ib_conn) \ + ((ib_conn)->state == IB_CONN_UNINITTED) +#define vnic_ib_conn_initted(ib_conn) \ + ((ib_conn)->state == IB_CONN_INITTED) +#define vnic_ib_conn_connecting(ib_conn) \ + ((ib_conn)->state == IB_CONN_CONNECTING) +#define vnic_ib_conn_connected(ib_conn) \ + ((ib_conn)->state == IB_CONN_CONNECTED) +#define vnic_ib_conn_disconnected(ib_conn) \ + ((ib_conn)->state == IB_CONN_DISCONNECTED) + +#define MCAST_GROUP_INVALID 0x00 /* viport failed to join or left mc group */ +#define MCAST_GROUP_JOINING 0x01 /* wait for completion */ +#define MCAST_GROUP_JOINED 0x02 /* join process completed successfully */ + +/* vnic_sa_client is used to register with sa once. It is needed to join and + * leave multicast groups. + */ +extern struct ib_sa_client vnic_sa_client; + +/* The following functions are using initialize and handle multicast + * components. + */ +struct mc_data; /* forward declaration */ +/* Initialize all necessary mc components */ +int vnic_ib_mc_init(struct mc_data *mc_data, struct viport *viport, + struct ib_pd *pd, struct vnic_ib_config *config); +/* Put multicast qp in RTS */ +int vnic_ib_mc_mod_qp_to_rts(struct ib_qp *qp); +/* Post multicast receive buffers */ +int vnic_ib_mc_post_recv(struct mc_data *mc_data, struct io *io); + +#endif /* VNIC_IB_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:05:46 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:05:46 -0400 Subject: [ofa-general] [PATCH v4 07/14] QLogic VNIC: Handling configurable parameters of the driver In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210546.11186.63864.stgit@dale> From: Poornima Kamath This patch adds the files that handle various configurable parameters of the VNIC driver ---- configuration of virtual NIC, control, data connections to the EVIC and general IB connection parameters. Signed-off-by: Poornima Kamath Signed-off-by: Ramachandra K Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_config.c | 379 ++++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_config.h | 240 +++++++++++++++ 2 files changed, 619 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_config.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_config.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_config.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_config.c new file mode 100644 index 0000000..cf120a6 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_config.c @@ -0,0 +1,379 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_config.h" +#include "vnic_trailer.h" +#include "vnic_main.h" + +u16 vnic_max_mtu = MAX_MTU; + +static u32 default_no_path_timeout = DEFAULT_NO_PATH_TIMEOUT; +static u32 sa_path_rec_get_timeout = SA_PATH_REC_GET_TIMEOUT; +static u32 default_primary_reconnect_timeout = + DEFAULT_PRIMARY_RECONNECT_TIMEOUT; +static u32 default_primary_switch_timeout = DEFAULT_PRIMARY_SWITCH_TIMEOUT; +static int default_prefer_primary = DEFAULT_PREFER_PRIMARY; + +static int use_rx_csum = VNIC_USE_RX_CSUM; +static int use_tx_csum = VNIC_USE_TX_CSUM; + +static u32 control_response_timeout = CONTROL_RSP_TIMEOUT; +static u32 completion_limit = DEFAULT_COMPLETION_LIMIT; + +module_param(vnic_max_mtu, ushort, 0444); +MODULE_PARM_DESC(vnic_max_mtu, "Maximum MTU size (1500-9500). Default is 9500"); + +module_param(default_prefer_primary, bool, 0444); +MODULE_PARM_DESC(default_prefer_primary, "Determines if primary path is" + " preferred (1) or not (0). Defaults to 0"); +module_param(use_rx_csum, bool, 0444); +MODULE_PARM_DESC(use_rx_csum, "Determines if RX checksum is done on VEx (1)" + " or not (0). Defaults to 1"); +module_param(use_tx_csum, bool, 0444); +MODULE_PARM_DESC(use_tx_csum, "Determines if TX checksum is done on VEx (1)" + " or not (0). Defaults to 1"); +module_param(default_no_path_timeout, uint, 0444); +MODULE_PARM_DESC(default_no_path_timeout, "Time to wait in milliseconds" + " before reconnecting to VEx after connection loss"); +module_param(default_primary_reconnect_timeout, uint, 0444); +MODULE_PARM_DESC(default_primary_reconnect_timeout, "Time to wait in" + " milliseconds before reconnecting the" + " primary path to VEx"); +module_param(default_primary_switch_timeout, uint, 0444); +MODULE_PARM_DESC(default_primary_switch_timeout, "Time to wait before" + " switching back to primary path if" + " primary path is preferred"); +module_param(sa_path_rec_get_timeout, uint, 0444); +MODULE_PARM_DESC(sa_path_rec_get_timeout, "Time out value in milliseconds" + " for SA path record get queries"); + +module_param(control_response_timeout, uint, 0444); +MODULE_PARM_DESC(control_response_timeout, "Time out value in milliseconds" + " to wait for response to control requests"); + +module_param(completion_limit, uint, 0444); +MODULE_PARM_DESC(completion_limit, "Maximum completions to process" + " in a single completion callback invocation. Default is 100" + " Minimum value is 10"); + +static void config_control_defaults(struct control_config *control_config, + struct vnic_path_param *params) +{ + int len; + char *dot; + u64 sid; + + sid = (SST_AGN << 56) | (SST_OUI << 32) | (CONTROL_PATH_ID << 8) + | IOC_NUMBER(be64_to_cpu(params->ioc_guid)); + + control_config->ib_config.service_id = cpu_to_be64(sid); + control_config->ib_config.conn_data.path_id = 0; + control_config->ib_config.conn_data.vnic_instance = params->instance; + control_config->ib_config.conn_data.path_num = 0; + control_config->ib_config.conn_data.features_supported = + __constant_cpu_to_be32((u32) (VNIC_FEAT_IGNORE_VLAN | + VNIC_FEAT_RDMA_IMMED)); + dot = strchr(init_utsname()->nodename, '.'); + + if (dot) + len = dot - init_utsname()->nodename; + else + len = strlen(init_utsname()->nodename); + + if (len > VNIC_MAX_NODENAME_LEN) + len = VNIC_MAX_NODENAME_LEN; + + memcpy(control_config->ib_config.conn_data.nodename, + init_utsname()->nodename, len); + + if (params->ib_multicast == 1) + control_config->ib_multicast = 1; + else if (params->ib_multicast == 0) + control_config->ib_multicast = 0; + else { + /* parameter is not set - enable it by default */ + control_config->ib_multicast = 1; + CONFIG_ERROR("IOCGUID=%llx INSTANCE=%d IB_MULTICAST defaulted" + " to TRUE\n", + be64_to_cpu(params->ioc_guid), + (char)params->instance); + } + + if (control_config->ib_multicast) + control_config->ib_config.conn_data.features_supported |= + __constant_cpu_to_be32(VNIC_FEAT_INBOUND_IB_MC); + + control_config->ib_config.retry_count = RETRY_COUNT; + control_config->ib_config.rnr_retry_count = RETRY_COUNT; + control_config->ib_config.min_rnr_timer = MIN_RNR_TIMER; + + /* These values are not configurable*/ + control_config->ib_config.num_recvs = 5; + control_config->ib_config.num_sends = 1; + control_config->ib_config.recv_scatter = 1; + control_config->ib_config.send_gather = 1; + control_config->ib_config.completion_limit = completion_limit; + + control_config->num_recvs = control_config->ib_config.num_recvs; + + control_config->vnic_instance = params->instance; + control_config->max_address_entries = MAX_ADDRESS_ENTRIES; + control_config->min_address_entries = MIN_ADDRESS_ENTRIES; + control_config->rsp_timeout = msecs_to_jiffies(control_response_timeout); +} + +static void config_data_defaults(struct data_config *data_config, + struct vnic_path_param *params) +{ + u64 sid; + + sid = (SST_AGN << 56) | (SST_OUI << 32) | (DATA_PATH_ID << 8) + | IOC_NUMBER(be64_to_cpu(params->ioc_guid)); + + data_config->ib_config.service_id = cpu_to_be64(sid); + data_config->ib_config.conn_data.path_id = jiffies; /* random */ + data_config->ib_config.conn_data.vnic_instance = params->instance; + data_config->ib_config.conn_data.path_num = 0; + + data_config->ib_config.retry_count = RETRY_COUNT; + data_config->ib_config.rnr_retry_count = RETRY_COUNT; + data_config->ib_config.min_rnr_timer = MIN_RNR_TIMER; + + /* + * NOTE: the num_recvs size assumes that the EIOC could + * RDMA enough packets to fill all of the host recv + * pool entries, plus send a kick message after each + * packet, plus RDMA new buffers for the size of + * the EIOC recv buffer pool, plus send kick messages + * after each min_host_update_sz of new buffers all + * before the host can even pull off the first completed + * receive off the completion queue, and repost the + * receive. NOT LIKELY! + */ + data_config->ib_config.num_recvs = HOST_RECV_POOL_ENTRIES + + (MAX_EIOC_POOL_SZ / MIN_HOST_UPDATE_SZ); + + data_config->ib_config.num_sends = (2 * NOTIFY_BUNDLE_SZ) + + (HOST_RECV_POOL_ENTRIES / MIN_EIOC_UPDATE_SZ) + 1; + + data_config->ib_config.recv_scatter = 1; /* not configurable */ + data_config->ib_config.send_gather = 2; /* not configurable */ + data_config->ib_config.completion_limit = completion_limit; + + data_config->num_recvs = data_config->ib_config.num_recvs; + data_config->path_id = data_config->ib_config.conn_data.path_id; + + + data_config->host_recv_pool_entries = HOST_RECV_POOL_ENTRIES; + + data_config->host_min.size_recv_pool_entry = + cpu_to_be32(BUFFER_SIZE(VLAN_ETH_HLEN + MIN_MTU)); + data_config->host_max.size_recv_pool_entry = + cpu_to_be32(BUFFER_SIZE(VLAN_ETH_HLEN + vnic_max_mtu)); + data_config->eioc_min.size_recv_pool_entry = + cpu_to_be32(BUFFER_SIZE(VLAN_ETH_HLEN + MIN_MTU)); + data_config->eioc_max.size_recv_pool_entry = + __constant_cpu_to_be32(MAX_PARAM_VALUE); + + data_config->host_min.num_recv_pool_entries = + __constant_cpu_to_be32(MIN_HOST_POOL_SZ); + data_config->host_max.num_recv_pool_entries = + __constant_cpu_to_be32(MAX_PARAM_VALUE); + data_config->eioc_min.num_recv_pool_entries = + __constant_cpu_to_be32(MIN_EIOC_POOL_SZ); + data_config->eioc_max.num_recv_pool_entries = + __constant_cpu_to_be32(MAX_EIOC_POOL_SZ); + + data_config->host_min.timeout_before_kick = + __constant_cpu_to_be32(MIN_HOST_KICK_TIMEOUT); + data_config->host_max.timeout_before_kick = + __constant_cpu_to_be32(MAX_HOST_KICK_TIMEOUT); + data_config->eioc_min.timeout_before_kick = 0; + data_config->eioc_max.timeout_before_kick = + __constant_cpu_to_be32(MAX_PARAM_VALUE); + + data_config->host_min.num_recv_pool_entries_before_kick = + __constant_cpu_to_be32(MIN_HOST_KICK_ENTRIES); + data_config->host_max.num_recv_pool_entries_before_kick = + __constant_cpu_to_be32(MAX_HOST_KICK_ENTRIES); + data_config->eioc_min.num_recv_pool_entries_before_kick = 0; + data_config->eioc_max.num_recv_pool_entries_before_kick = + __constant_cpu_to_be32(MAX_PARAM_VALUE); + + data_config->host_min.num_recv_pool_bytes_before_kick = + __constant_cpu_to_be32(MIN_HOST_KICK_BYTES); + data_config->host_max.num_recv_pool_bytes_before_kick = + __constant_cpu_to_be32(MAX_HOST_KICK_BYTES); + data_config->eioc_min.num_recv_pool_bytes_before_kick = 0; + data_config->eioc_max.num_recv_pool_bytes_before_kick = + __constant_cpu_to_be32(MAX_PARAM_VALUE); + + data_config->host_min.free_recv_pool_entries_per_update = + __constant_cpu_to_be32(MIN_HOST_UPDATE_SZ); + data_config->host_max.free_recv_pool_entries_per_update = + __constant_cpu_to_be32(MAX_HOST_UPDATE_SZ); + data_config->eioc_min.free_recv_pool_entries_per_update = + __constant_cpu_to_be32(MIN_EIOC_UPDATE_SZ); + data_config->eioc_max.free_recv_pool_entries_per_update = + __constant_cpu_to_be32(MAX_EIOC_UPDATE_SZ); + + data_config->notify_bundle = NOTIFY_BUNDLE_SZ; +} + +static void config_path_info_defaults(struct viport_config *config, + struct vnic_path_param *params) +{ + int i; + ib_query_gid(config->ibdev, config->port, 0, + &config->path_info.path.sgid); + for (i = 0; i < 16; i++) + config->path_info.path.dgid.raw[i] = params->dgid[i]; + + config->path_info.path.pkey = params->pkey; + config->path_info.path.numb_path = 1; + config->sa_path_rec_get_timeout = sa_path_rec_get_timeout; + +} + +static void config_viport_defaults(struct viport_config *config, + struct vnic_path_param *params) +{ + config->ibdev = params->ibdev; + config->port = params->port; + config->ioc_guid = params->ioc_guid; + config->stats_interval = msecs_to_jiffies(VIPORT_STATS_INTERVAL); + config->hb_interval = msecs_to_jiffies(VIPORT_HEARTBEAT_INTERVAL); + config->hb_timeout = VIPORT_HEARTBEAT_TIMEOUT * 1000; + /*hb_timeout needs to be in usec*/ + strcpy(config->ioc_string, params->ioc_string); + config_path_info_defaults(config, params); + + config_control_defaults(&config->control_config, params); + config_data_defaults(&config->data_config, params); +} + +static void config_vnic_defaults(struct vnic_config *config) +{ + config->no_path_timeout = msecs_to_jiffies(default_no_path_timeout); + config->primary_connect_timeout = + msecs_to_jiffies(DEFAULT_PRIMARY_CONNECT_TIMEOUT); + config->primary_reconnect_timeout = + msecs_to_jiffies(default_primary_reconnect_timeout); + config->primary_switch_timeout = + msecs_to_jiffies(default_primary_switch_timeout); + config->prefer_primary = default_prefer_primary; + config->use_rx_csum = use_rx_csum; + config->use_tx_csum = use_tx_csum; +} + +struct viport_config *config_alloc_viport(struct vnic_path_param *params) +{ + struct viport_config *config; + + config = kzalloc(sizeof *config, GFP_KERNEL); + if (!config) { + CONFIG_ERROR("could not allocate memory for" + " struct viport_config\n"); + return NULL; + } + + config_viport_defaults(config, params); + + return config; +} + +struct vnic_config *config_alloc_vnic(void) +{ + struct vnic_config *config; + + config = kzalloc(sizeof *config, GFP_KERNEL); + if (!config) { + CONFIG_ERROR("couldn't allocate memory for" + " struct vnic_config\n"); + + return NULL; + } + + config_vnic_defaults(config); + return config; +} + +char *config_viport_name(struct viport_config *config) +{ + /* function only called by one thread, can return a static string */ + static char str[64]; + + sprintf(str, "GUID %llx instance %d", + be64_to_cpu(config->ioc_guid), + config->control_config.vnic_instance); + return str; +} + +int config_start(void) +{ + vnic_max_mtu = min_t(u16, vnic_max_mtu, MAX_MTU); + vnic_max_mtu = max_t(u16, vnic_max_mtu, MIN_MTU); + + sa_path_rec_get_timeout = min_t(u32, sa_path_rec_get_timeout, + MAX_SA_TIMEOUT); + sa_path_rec_get_timeout = max_t(u32, sa_path_rec_get_timeout, + MIN_SA_TIMEOUT); + + control_response_timeout = min_t(u32, control_response_timeout, + MAX_CONTROL_RSP_TIMEOUT); + + control_response_timeout = max_t(u32, control_response_timeout, + MIN_CONTROL_RSP_TIMEOUT); + + completion_limit = max_t(u32, completion_limit, + MIN_COMPLETION_LIMIT); + + if (!default_no_path_timeout) + default_no_path_timeout = DEFAULT_NO_PATH_TIMEOUT; + + if (!default_primary_reconnect_timeout) + default_primary_reconnect_timeout = + DEFAULT_PRIMARY_RECONNECT_TIMEOUT; + + if (!default_primary_switch_timeout) + default_primary_switch_timeout = + DEFAULT_PRIMARY_SWITCH_TIMEOUT; + + return 0; + +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_config.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_config.h new file mode 100644 index 0000000..d8e0e0f --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_config.h @@ -0,0 +1,240 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_CONFIG_H_INCLUDED +#define VNIC_CONFIG_H_INCLUDED + +#include +#include +#include + +#include "vnic_control.h" +#include "vnic_ib.h" + +#define SST_AGN 0x10ULL +#define SST_OUI 0x00066AULL + +enum { + CONTROL_PATH_ID = 0x0, + DATA_PATH_ID = 0x1 +}; + +#define IOC_NUMBER(GUID) (((GUID) >> 32) & 0xFF) + +enum { + VNIC_CLASS_SUBCLASS = 0x2000066A, + VNIC_PROTOCOL = 0, + VNIC_PROT_VERSION = 1 +}; + +enum { + MIN_MTU = 1500, /* minimum negotiated MTU size */ + MAX_MTU = 9500 /* jumbo frame */ +}; + +/* + * TODO: tune the pool parameter values + */ +enum { + MIN_ADDRESS_ENTRIES = 16, + MAX_ADDRESS_ENTRIES = 64 +}; + +enum { + HOST_RECV_POOL_ENTRIES = 512, + MIN_HOST_POOL_SZ = 64, + MIN_EIOC_POOL_SZ = 64, + MAX_EIOC_POOL_SZ = 256, + MIN_HOST_UPDATE_SZ = 8, + MAX_HOST_UPDATE_SZ = 32, + MIN_EIOC_UPDATE_SZ = 8, + MAX_EIOC_UPDATE_SZ = 32, + NOTIFY_BUNDLE_SZ = 32 +}; + +enum { + MIN_HOST_KICK_TIMEOUT = 10, /* in usec */ + MAX_HOST_KICK_TIMEOUT = 100 /* in usec */ +}; + +enum { + MIN_HOST_KICK_ENTRIES = 1, + MAX_HOST_KICK_ENTRIES = 128 +}; + +enum { + MIN_HOST_KICK_BYTES = 0, + MAX_HOST_KICK_BYTES = 5000 +}; + +enum { + DEFAULT_NO_PATH_TIMEOUT = 10000, + DEFAULT_PRIMARY_CONNECT_TIMEOUT = 10000, + DEFAULT_PRIMARY_RECONNECT_TIMEOUT = 10000, + DEFAULT_PRIMARY_SWITCH_TIMEOUT = 10000 +}; + +enum { + VIPORT_STATS_INTERVAL = 500, /* .5 sec */ + VIPORT_HEARTBEAT_INTERVAL = 1000, /* 1 second */ + VIPORT_HEARTBEAT_TIMEOUT = 64000 /* 64 sec */ +}; + +enum { + /* 5 sec increased for EVIC support for large number of + * host connections + */ + CONTROL_RSP_TIMEOUT = 5000, + MIN_CONTROL_RSP_TIMEOUT = 1000, /* 1 sec */ + MAX_CONTROL_RSP_TIMEOUT = 60000 /* 60 sec */ +}; + +/* Maximum number of completions to be processed + * during a single completion callback invocation + */ +enum { + DEFAULT_COMPLETION_LIMIT = 100, + MIN_COMPLETION_LIMIT = 10 +}; + +/* infiniband connection parameters */ +enum { + RETRY_COUNT = 3, + MIN_RNR_TIMER = 22, /* 20 ms */ + DEFAULT_PKEY = 0 /* pkey table index */ +}; + +enum { + SA_PATH_REC_GET_TIMEOUT = 1000, /* 1000 ms */ + MIN_SA_TIMEOUT = 100, /* 100 ms */ + MAX_SA_TIMEOUT = 20000 /* 20s */ +}; + +#define MAX_PARAM_VALUE 0x40000000 +#define VNIC_USE_RX_CSUM 1 +#define VNIC_USE_TX_CSUM 1 +#define DEFAULT_PREFER_PRIMARY 0 + +/* As per IBTA specification, IOCString Maximum length can be 512 bits. */ +#define MAX_IOC_STRING_LEN (512/8) + +struct vnic_path_param { + __be64 ioc_guid; + u8 ioc_string[MAX_IOC_STRING_LEN+1]; + u8 port; + u8 instance; + struct ib_device *ibdev; + struct vnic_ib_port *ibport; + u8 dgid[16]; + __be16 pkey; + int heartbeat; + int ib_multicast; + int opt_mask; +}; + +struct vnic_ib_config { + __be64 service_id; + struct vnic_connection_data conn_data; + u32 retry_count; + u32 rnr_retry_count; + u8 min_rnr_timer; + u32 num_sends; + u32 num_recvs; + u32 recv_scatter; /* 1 */ + u32 send_gather; /* 1 or 2 */ + u32 completion_limit; +}; + +struct control_config { + struct vnic_ib_config ib_config; + u32 num_recvs; + u8 vnic_instance; + u16 max_address_entries; + u16 min_address_entries; + u32 rsp_timeout; + u32 ib_multicast; +}; + +struct data_config { + struct vnic_ib_config ib_config; + u64 path_id; + u32 num_recvs; + u32 host_recv_pool_entries; + struct vnic_recv_pool_config host_min; + struct vnic_recv_pool_config host_max; + struct vnic_recv_pool_config eioc_min; + struct vnic_recv_pool_config eioc_max; + u32 notify_bundle; +}; + +struct viport_config { + struct viport *viport; + struct control_config control_config; + struct data_config data_config; + struct vnic_ib_path_info path_info; + u32 sa_path_rec_get_timeout; + struct ib_device *ibdev; + u32 port; + unsigned long stats_interval; + u32 hb_interval; + u32 hb_timeout; + __be64 ioc_guid; + u8 ioc_string[MAX_IOC_STRING_LEN+1]; + size_t path_idx; +}; + +/* + * primary_connect_timeout - if the secondary connects first, + * how long do we give the primary? + * primary_reconnect_timeout - same as above, but used when recovering + * from the case where both paths fail + * primary_switch_timeout - how long do we wait before switching to the + * primary when it comes back? + */ +struct vnic_config { + struct vnic *vnic; + char name[IFNAMSIZ]; + unsigned long no_path_timeout; + u32 primary_connect_timeout; + u32 primary_reconnect_timeout; + u32 primary_switch_timeout; + int prefer_primary; + int use_rx_csum; + int use_tx_csum; +}; + +int config_start(void); +struct viport_config *config_alloc_viport(struct vnic_path_param *params); +struct vnic_config *config_alloc_vnic(void); +char *config_viport_name(struct viport_config *config); + +#endif /* VNIC_CONFIG_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:06:17 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:06:17 -0400 Subject: [ofa-general] [PATCH v4 08/14] QLogic VNIC: sysfs interface implementation for the driver In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210616.11186.10855.stgit@dale> From: Amar Mudrankit The sysfs interface for the QLogic VNIC driver is implemented through this patch. Signed-off-by: Amar Mudrankit Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath --- drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c | 1256 +++++++++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h | 52 + 2 files changed, 1308 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c new file mode 100644 index 0000000..7dbc955 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.c @@ -0,0 +1,1256 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include + +#include "vnic_util.h" +#include "vnic_config.h" +#include "vnic_ib.h" +#include "vnic_viport.h" +#include "vnic_main.h" +#include "vnic_stats.h" + +/* + * target eiocs are added by writing each individual connection parameter + * into respective sysfs file under interface level or path level. + */ +enum { + VNIC_OPT_ERR = 0, + VNIC_OPT_IOC_GUID = 1 << 0, + VNIC_OPT_DGID = 1 << 1, + VNIC_OPT_PKEY = 1 << 2, + VNIC_OPT_NAME = 1 << 3, + VNIC_OPT_INSTANCE = 1 << 4, + VNIC_OPT_RXCSUM = 1 << 5, + VNIC_OPT_TXCSUM = 1 << 6, + VNIC_OPT_HEARTBEAT = 1 << 7, + VNIC_OPT_IOC_STRING = 1 << 8, + VNIC_OPT_IB_MULTICAST = 1 << 9, + VNIC_OPT_ALL = (VNIC_OPT_IOC_GUID | + VNIC_OPT_DGID | VNIC_OPT_NAME | VNIC_OPT_PKEY), +}; + +static match_table_t vnic_opt_tokens = { + {VNIC_OPT_IOC_GUID, "ioc_guid=%s"}, + {VNIC_OPT_DGID, "dgid=%s"}, + {VNIC_OPT_PKEY, "pkey=%x"}, + {VNIC_OPT_NAME, "name=%s"}, + {VNIC_OPT_INSTANCE, "instance=%d"}, + {VNIC_OPT_RXCSUM, "rx_csum=%s"}, + {VNIC_OPT_TXCSUM, "tx_csum=%s"}, + {VNIC_OPT_HEARTBEAT, "heartbeat=%d"}, + {VNIC_OPT_IOC_STRING, "ioc_string=\"%s"}, + {VNIC_OPT_IB_MULTICAST, "ib_multicast=%s"}, + {VNIC_OPT_ERR, NULL} +}; + +void vnic_release_dev(struct device *dev) +{ + struct dev_info *dev_info = + container_of(dev, struct dev_info, dev); + + complete(&dev_info->released); +} + +struct class vnic_class = { + .name = "infiniband_qlgc_vnic", + .dev_release = vnic_release_dev +}; + +struct dev_info interface_dev; + +static int vnic_all_connection_params(struct vnic_path_param *path) +{ + int i, ret = -EINVAL; + + if ((path->opt_mask & VNIC_OPT_ALL) == VNIC_OPT_ALL) + ret = 0; + else + for (i = 0; i < ARRAY_SIZE(vnic_opt_tokens); ++i) + if ((vnic_opt_tokens[i].token & VNIC_OPT_ALL) && + !(vnic_opt_tokens[i].token & path->opt_mask)) + printk(KERN_WARNING PFX + "target creation request is " + "missing parameter '%s'\n", + vnic_opt_tokens[i].pattern); + + return ret; +} + +static ssize_t show_vnic_state(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + switch (vnic->state) { + case VNIC_UNINITIALIZED: + return sprintf(buf, "VNIC_UNINITIALIZED\n"); + case VNIC_REGISTERED: + return sprintf(buf, "VNIC_REGISTERED\n"); + default: + return sprintf(buf, "INVALID STATE\n"); + } +} + +static DEVICE_ATTR(vnic_state, S_IRUGO, show_vnic_state, NULL); + +static ssize_t show_rx_csum(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + + if (vnic->config->use_rx_csum) + return sprintf(buf, "true\n"); + else + return sprintf(buf, "false\n"); +} + +static ssize_t store_rx_csum(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + + if (!strncmp(buf, "true", 4)) + vnic->params.rx_csum = 1; + else if (!strncmp(buf, "false", 5)) + vnic->params.rx_csum = 0; + else { + printk(KERN_WARNING PFX "bad rx_csum parameter." + " must be 'true' or 'false'\n"); + return -EINVAL; + } + + return count; +} + +static DEVICE_ATTR(rx_csum, S_IRUGO | S_IWUSR, show_rx_csum, store_rx_csum); + +static ssize_t show_tx_csum(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + + if (vnic->config->use_tx_csum) + return sprintf(buf, "true\n"); + else + return sprintf(buf, "false\n"); +} + +static ssize_t store_tx_csum(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + + if (!strncmp(buf, "true", 4)) + vnic->params.tx_csum = 1; + else if (!strncmp(buf, "false", 5)) + vnic->params.tx_csum = 0; + else { + printk(KERN_WARNING PFX "bad tx_csum parameter." + " must be 'true' or 'false'\n"); + return -EINVAL; + } + + return count; +} + +static DEVICE_ATTR(tx_csum, S_IRUGO | S_IWUSR, show_tx_csum, store_tx_csum); + +static ssize_t show_current_path(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + unsigned long flags; + size_t length; + + spin_lock_irqsave(&vnic->current_path_lock, flags); + if (vnic->current_path == &vnic->primary_path) + length = sprintf(buf, "primary_path\n"); + else if (vnic->current_path == &vnic->secondary_path) + length = sprintf(buf, "secondary path\n"); + else + length = sprintf(buf, "none\n"); + spin_unlock_irqrestore(&vnic->current_path_lock, flags); + return length; +} + +static DEVICE_ATTR(current_path, S_IRUGO, show_current_path, NULL); + +static ssize_t show_multicast(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + + if (!vnic->primary_path.viport) + return sprintf(buf, "\n"); + + if (vnic->primary_path.viport->control.config->ib_multicast && + vnic->secondary_path.viport->control.config->ib_multicast) + return sprintf(buf, "true\n"); + else + return sprintf(buf, "false\n"); +} + +static ssize_t store_multicast(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, dev_info); + + if (!strncmp(buf, "true", 4)) { + vnic->params.primary_path.ib_multicast = 1; + vnic->params.secondary_path.ib_multicast = 1; + } else if (!strncmp(buf, "false", 5)) { + vnic->params.primary_path.ib_multicast = 0; + vnic->params.secondary_path.ib_multicast = 0; + } else { + printk(KERN_WARNING PFX "bad ib_multicast parameter." + " must be 'true' or 'false'\n"); + return -EINVAL; + } + + return count; +} +static DEVICE_ATTR(multicast, S_IRUGO | S_IWUSR, show_multicast, store_multicast); + +static struct attribute *vnic_dev_attrs[] = { + &dev_attr_vnic_state.attr, + &dev_attr_rx_csum.attr, + &dev_attr_tx_csum.attr, + &dev_attr_current_path.attr, + &dev_attr_multicast.attr, + NULL +}; + +struct attribute_group vnic_dev_attr_group = { + .attrs = vnic_dev_attrs, +}; + +static inline void print_dgid(u8 *dgid) +{ + int i; + + for (i = 0; i < 16; i += 2) + printk("%04x", be16_to_cpu(*(__be16 *)&dgid[i])); +} + +static inline int is_dgid_zero(u8 *dgid) +{ + int i; + + for (i = 0; i < 16; i++) { + if (dgid[i] != 0) + return 1; + } + return 0; +} + +static int vnic_create_netpath(struct netpath *npdest, + struct vnic_path_param *p_params) +{ + struct viport_config *viport_config; + struct viport *viport; + struct vnic *vnic; + struct list_head *ptr; + int ret = 0; + + list_for_each(ptr, &vnic_list) { + vnic = list_entry(ptr, struct vnic, list_ptrs); + if (vnic->primary_path.viport) { + viport_config = vnic->primary_path.viport->config; + if ((viport_config->ioc_guid == p_params->ioc_guid) + && (viport_config->control_config.vnic_instance + == p_params->instance) + && (be64_to_cpu(p_params->ioc_guid))) { + SYS_ERROR("GUID %llx," + " INSTANCE %d already in use\n", + be64_to_cpu(p_params->ioc_guid), + p_params->instance); + ret = -EINVAL; + goto out; + } + } + + if (vnic->secondary_path.viport) { + viport_config = vnic->secondary_path.viport->config; + if ((viport_config->ioc_guid == p_params->ioc_guid) + && (viport_config->control_config.vnic_instance + == p_params->instance) + && (be64_to_cpu(p_params->ioc_guid))) { + SYS_ERROR("GUID %llx," + " INSTANCE %d already in use\n", + be64_to_cpu(p_params->ioc_guid), + p_params->instance); + ret = -EINVAL; + goto out; + } + } + } + + if (npdest->viport) { + SYS_ERROR("create_netpath: path already exists\n"); + ret = -EINVAL; + goto out; + } + + viport_config = config_alloc_viport(p_params); + if (!viport_config) { + SYS_ERROR("create_netpath: failed creating viport config\n"); + ret = -1; + goto out; + } + + /*User specified heartbeat value is in 1/100s of a sec*/ + if (p_params->heartbeat != -1) { + viport_config->hb_interval = + msecs_to_jiffies(p_params->heartbeat * 10); + viport_config->hb_timeout = + (p_params->heartbeat << 6) * 10000; /* usec */ + } + + viport_config->path_idx = 0; + + viport = viport_allocate(viport_config); + if (!viport) { + SYS_ERROR("create_netpath: failed creating viport\n"); + kfree(viport_config); + ret = -1; + goto out; + } + + npdest->viport = viport; + viport->parent = npdest; + viport->vnic = npdest->parent; + + if (is_dgid_zero(p_params->dgid) && p_params->ioc_guid != 0 + && p_params->pkey != 0) { + viport_kick(viport); + vnic_disconnected(npdest->parent, npdest); + } else { + printk(KERN_WARNING "Specified parameters IOCGUID=%llx, " + "P_Key=%x, DGID=", be64_to_cpu(p_params->ioc_guid), + p_params->pkey); + print_dgid(p_params->dgid); + printk(" insufficient for establishing %s path for interface " + "%s. Hence, path will not be established.\n", + (npdest->second_bias ? "secondary" : "primary"), + npdest->parent->config->name); + } +out: + return ret; +} + +static int vnic_setup_path_files(struct netpath *path, char *name) +{ + init_completion(&path->dev_info.released); + + path->dev_info.dev.class = NULL; + path->dev_info.dev.parent = &path->parent->dev_info.dev; + path->dev_info.dev.release = vnic_release_dev; + snprintf(path->dev_info.dev.bus_id, BUS_ID_SIZE, name); + + if (device_register(&path->dev_info.dev)) { + SYS_ERROR("error in registering path class dev\n"); + goto out; + } + + if (sysfs_create_group(&path->dev_info.dev.kobj, + &vnic_path_attr_group)) { + SYS_ERROR("error in creating vnic path group attrs"); + goto err_path; + } + + return 0; + +err_path: + device_unregister(&path->dev_info.dev); + wait_for_completion(&path->dev_info.released); +out: + return -1; +} + +static inline void vnic_path_param_init(struct vnic_path_param *path) +{ + *(path->ioc_string) = '\0'; + path->heartbeat = -1; + path->instance = 0; + path->ib_multicast = -1; + path->opt_mask = 0; +} + +static inline void vnic_param_init(struct vnic_param *params) +{ + params->rx_csum = -1; + params->tx_csum = -1; + vnic_path_param_init(¶ms->primary_path); + vnic_path_param_init(¶ms->secondary_path); +} + +static ssize_t vnic_create(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct vnic_config *vnic_config; + struct vnic *vnic; + struct list_head *ptr; + + SYS_INFO("vnic_create: name = %s\n", buf); + list_for_each(ptr, &vnic_list) { + vnic = list_entry(ptr, struct vnic, list_ptrs); + if (!strcmp(vnic->config->name, buf)) { + SYS_ERROR("vnic %s already exists\n", buf); + return -EEXIST; + } + } + + vnic_config = config_alloc_vnic(); + if (!vnic_config) { + SYS_ERROR("create_vnic: failed creating vnic config\n"); + goto out; + } + strcpy(vnic_config->name, buf); + + vnic = vnic_allocate(vnic_config); + if (!vnic) { + SYS_ERROR("create_vnic: failed allocating vnic\n"); + goto free_vnic_config; + } + + init_completion(&vnic->dev_info.released); + vnic->dev_info.dev.class = NULL; + vnic->dev_info.dev.parent = &interface_dev.dev; + vnic->dev_info.dev.release = vnic_release_dev; + snprintf(vnic->dev_info.dev.bus_id, BUS_ID_SIZE, buf); + + if (device_register(&vnic->dev_info.dev)) { + SYS_ERROR("create_vnic: error in registering" + " vnic class dev\n"); + goto free_vnic; + } + + if (sysfs_create_group(&vnic->dev_info.dev.kobj, + &vnic_dev_attr_group)) { + SYS_ERROR("create_vnic: error in creating" + "vnic attr group\n"); + goto err_attr; + } + + if (vnic_setup_stats_files(vnic)) + goto err_stats; + + if (vnic_setup_path_files(&vnic->primary_path, "primary_path")) + goto err_primary_path; + + if (vnic_setup_path_files(&vnic->secondary_path, "secondary_path")) + goto err_secondary_path; + + vnic_param_init(&vnic->params); + return count; + +err_secondary_path: + vnic_cleanup_path_files(&vnic->primary_path); +err_primary_path: + vnic_cleanup_stats_files(vnic); +err_stats: + sysfs_remove_group(&vnic->dev_info.dev.kobj, + &vnic_dev_attr_group); +err_attr: + device_unregister(&vnic->dev_info.dev); + wait_for_completion(&vnic->dev_info.released); +free_vnic: + list_del(&vnic->list_ptrs); + kfree(vnic); +free_vnic_config: + kfree(vnic_config); +out: + return -EINVAL; +} + +DEVICE_ATTR(create_vnic, S_IWUSR, NULL, vnic_create); + +static ssize_t vnic_delete(struct device *dev, struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct vnic *vnic; + struct list_head *ptr; + int ret = -EINVAL; + + if (count > IFNAMSIZ) { + printk(KERN_WARNING PFX "invalid vnic interface name\n"); + return ret; + } + + SYS_INFO("vnic_delete: name = %s\n", buf); + list_for_each(ptr, &vnic_list) { + vnic = list_entry(ptr, struct vnic, list_ptrs); + if (!strcmp(vnic->config->name, buf)) { + vnic_free(vnic); + return count; + } + } + + printk(KERN_WARNING PFX "vnic interface '%s' does not exist\n", buf); + return ret; +} + +DEVICE_ATTR(delete_vnic, S_IWUSR, NULL, vnic_delete); + +static ssize_t show_viport_state(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "INVALID STATE\n"); + + switch (path->viport->state) { + case VIPORT_DISCONNECTED: + return sprintf(buf, "VIPORT_DISCONNECTED\n"); + case VIPORT_CONNECTED: + return sprintf(buf, "VIPORT_CONNECTED\n"); + default: + return sprintf(buf, "INVALID STATE\n"); + } + +} + +static DEVICE_ATTR(viport_state, S_IRUGO, show_viport_state, NULL); + +static ssize_t show_link_state(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "INVALID STATE\n"); + + switch (path->viport->link_state) { + case LINK_UNINITIALIZED: + return sprintf(buf, "LINK_UNINITIALIZED\n"); + case LINK_INITIALIZE: + return sprintf(buf, "LINK_INITIALIZE\n"); + case LINK_INITIALIZECONTROL: + return sprintf(buf, "LINK_INITIALIZECONTROL\n"); + case LINK_INITIALIZEDATA: + return sprintf(buf, "LINK_INITIALIZEDATA\n"); + case LINK_CONTROLCONNECT: + return sprintf(buf, "LINK_CONTROLCONNECT\n"); + case LINK_CONTROLCONNECTWAIT: + return sprintf(buf, "LINK_CONTROLCONNECTWAIT\n"); + case LINK_INITVNICREQ: + return sprintf(buf, "LINK_INITVNICREQ\n"); + case LINK_INITVNICRSP: + return sprintf(buf, "LINK_INITVNICRSP\n"); + case LINK_BEGINDATAPATH: + return sprintf(buf, "LINK_BEGINDATAPATH\n"); + case LINK_CONFIGDATAPATHREQ: + return sprintf(buf, "LINK_CONFIGDATAPATHREQ\n"); + case LINK_CONFIGDATAPATHRSP: + return sprintf(buf, "LINK_CONFIGDATAPATHRSP\n"); + case LINK_DATACONNECT: + return sprintf(buf, "LINK_DATACONNECT\n"); + case LINK_DATACONNECTWAIT: + return sprintf(buf, "LINK_DATACONNECTWAIT\n"); + case LINK_XCHGPOOLREQ: + return sprintf(buf, "LINK_XCHGPOOLREQ\n"); + case LINK_XCHGPOOLRSP: + return sprintf(buf, "LINK_XCHGPOOLRSP\n"); + case LINK_INITIALIZED: + return sprintf(buf, "LINK_INITIALIZED\n"); + case LINK_IDLE: + return sprintf(buf, "LINK_IDLE\n"); + case LINK_IDLING: + return sprintf(buf, "LINK_IDLING\n"); + case LINK_CONFIGLINKREQ: + return sprintf(buf, "LINK_CONFIGLINKREQ\n"); + case LINK_CONFIGLINKRSP: + return sprintf(buf, "LINK_CONFIGLINKRSP\n"); + case LINK_CONFIGADDRSREQ: + return sprintf(buf, "LINK_CONFIGADDRSREQ\n"); + case LINK_CONFIGADDRSRSP: + return sprintf(buf, "LINK_CONFIGADDRSRSP\n"); + case LINK_REPORTSTATREQ: + return sprintf(buf, "LINK_REPORTSTATREQ\n"); + case LINK_REPORTSTATRSP: + return sprintf(buf, "LINK_REPORTSTATRSP\n"); + case LINK_HEARTBEATREQ: + return sprintf(buf, "LINK_HEARTBEATREQ\n"); + case LINK_HEARTBEATRSP: + return sprintf(buf, "LINK_HEARTBEATRSP\n"); + case LINK_RESET: + return sprintf(buf, "LINK_RESET\n"); + case LINK_RESETRSP: + return sprintf(buf, "LINK_RESETRSP\n"); + case LINK_RESETCONTROL: + return sprintf(buf, "LINK_RESETCONTROL\n"); + case LINK_RESETCONTROLRSP: + return sprintf(buf, "LINK_RESETCONTROLRSP\n"); + case LINK_DATADISCONNECT: + return sprintf(buf, "LINK_DATADISCONNECT\n"); + case LINK_CONTROLDISCONNECT: + return sprintf(buf, "LINK_CONTROLDISCONNECT\n"); + case LINK_CLEANUPDATA: + return sprintf(buf, "LINK_CLEANUPDATA\n"); + case LINK_CLEANUPCONTROL: + return sprintf(buf, "LINK_CLEANUPCONTROL\n"); + case LINK_DISCONNECTED: + return sprintf(buf, "LINK_DISCONNECTED\n"); + case LINK_RETRYWAIT: + return sprintf(buf, "LINK_RETRYWAIT\n"); + default: + return sprintf(buf, "INVALID STATE\n"); + } +} + +static DEVICE_ATTR(link_state, S_IRUGO, show_link_state, NULL); + +static ssize_t show_heartbeat(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + /* hb_inteval is in jiffies, convert it back to + * 1/100ths of a second + */ + if (!path->viport) + return sprintf(buf, "0\n"); + else + return sprintf(buf, "%d\n", + (jiffies_to_msecs(path->viport->config->hb_interval)/10)); +} + +static ssize_t store_heartbeat(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + int token; + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + char *dup = kstrdup(buf, GFP_KERNEL); + substring_t heartbeat = { dup, dup + strlen(dup) }; + + if (match_int(&heartbeat, &token)) { + printk(KERN_WARNING PFX "bad heartbeat parameter '%s'\n", buf); + return -EINVAL; + } + if (token > 6000 || token <= 0) { + printk(KERN_WARNING PFX "heartbeat parameter must be" + " > 0 and <= 6000\n"); + return -EINVAL; + } + + if (path->second_bias) + path->parent->params.secondary_path.heartbeat = token; + else + path->parent->params.primary_path.heartbeat = token; + + kfree(dup); + return count; +} + +static DEVICE_ATTR(heartbeat, S_IRUGO | S_IWUSR, show_heartbeat, store_heartbeat); + +static ssize_t show_ioc_guid(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "0\n"); + else + return sprintf(buf, "%llx\n", + __be64_to_cpu(path->viport->config->ioc_guid)); +} + +static ssize_t store_ioc_guid(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + char *dup = kstrdup(buf, GFP_KERNEL); + + __be64 iocguid = cpu_to_be64(simple_strtoull(dup, NULL, 16)); + if (path->second_bias) { + path->parent->params.secondary_path.ioc_guid = iocguid; + path->parent->params.secondary_path.opt_mask |= VNIC_OPT_IOC_GUID; + } else { + path->parent->params.primary_path.ioc_guid = iocguid; + path->parent->params.primary_path.opt_mask |= VNIC_OPT_IOC_GUID; + } + + kfree(dup); + return count; +} + +static DEVICE_ATTR(ioc_guid, S_IRUGO | S_IWUSR, show_ioc_guid, store_ioc_guid); + +static inline void get_dgid_string(u8 *dgid, char *buf) +{ + int i; + char holder[5]; + + for (i = 0; i < 16; i += 2) { + sprintf(holder, "%04x", be16_to_cpu(*(__be16 *)&dgid[i])); + strcat(buf, holder); + } + + strcat(buf, "\n"); +} + +static ssize_t show_dgid(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "0\n"); + else { + get_dgid_string(path->viport->config->path_info.path.dgid.raw, + buf); + return strlen(buf); + } +} + +static ssize_t store_dgid(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + char *dup = kstrdup(buf, GFP_KERNEL); + char dgid[3]; + int i; + + if (strlen(dup) != 32) { + printk(KERN_WARNING PFX "bad dest GID parameter '%s'\n", dup); + kfree(dup); + return -EINVAL; + } + + for (i = 0; i < 16; ++i) { + strlcpy(dgid, dup + i * 2, 3); + if (path->second_bias) { + path->parent->params.secondary_path.dgid[i] = + simple_strtoul(dgid, NULL, 16); + path->parent->params.secondary_path.opt_mask |= + VNIC_OPT_DGID; + } else { + path->parent->params.primary_path.dgid[i] = + simple_strtoul(dgid, NULL, 16); + path->parent->params.primary_path.opt_mask |= + VNIC_OPT_DGID; + } + + } + kfree(dup); + return count; +} + +static DEVICE_ATTR(dgid, S_IRUGO | S_IWUSR, show_dgid, store_dgid); + +static ssize_t show_pkey(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "0\n"); + else + return sprintf(buf, "%x\n", + path->viport->config->path_info.path.pkey); +} + +static ssize_t store_pkey(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + int token; + char *dup = kstrdup(buf, GFP_KERNEL); + substring_t pkey = { dup, dup + strlen(dup) }; + + if (match_hex(&pkey, &token)) { + printk(KERN_WARNING PFX "bad P_key parameter '%s'\n", buf); + return -EINVAL; + } + + if (path->second_bias) { + path->parent->params.secondary_path.pkey = cpu_to_be16(token); + path->parent->params.secondary_path.opt_mask |= VNIC_OPT_PKEY; + } else { + path->parent->params.primary_path.pkey = cpu_to_be16(token); + path->parent->params.primary_path.opt_mask |= VNIC_OPT_PKEY; + } + kfree(dup); + return count; +} + +static DEVICE_ATTR(pkey, S_IRUGO | S_IWUSR, show_pkey, store_pkey); + +static ssize_t show_hca_info(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "\n"); + else + return sprintf(buf, "vnic-%s-%d\n", + path->viport->config->ibdev->name, + path->viport->config->port); +} + +static DEVICE_ATTR(hca_info, S_IRUGO, show_hca_info, NULL); + +static ssize_t show_ioc_string(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "\n"); + else + return sprintf(buf, "%s\n", path->viport->config->ioc_string); +} + +static ssize_t store_ioc_string(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (strlen(buf) > MAX_IOC_STRING_LEN) { + printk(KERN_WARNING PFX "ioc string parameter too long\n"); + return -EINVAL; + } + if (path->second_bias) + strncpy(path->parent->params.secondary_path.ioc_string, buf, + MAX_IOC_STRING_LEN); + else + strncpy(path->parent->params.primary_path.ioc_string, buf, + MAX_IOC_STRING_LEN); + + return count; +} + +static DEVICE_ATTR(ioc_string, S_IRUGO | S_IWUSR, show_ioc_string, store_ioc_string); + +static ssize_t show_multicast_state(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "\n"); + + if (!(path->viport->features_supported & VNIC_FEAT_INBOUND_IB_MC)) + return sprintf(buf, "feature not enabled\n"); + + switch (path->viport->mc_info.state) { + case MCAST_STATE_INVALID: + return sprintf(buf, "state=Invalid\n"); + case MCAST_STATE_JOINING: + return sprintf(buf, "state=Joining MGID:" VNIC_GID_FMT "\n", + VNIC_GID_RAW_ARG(path->viport->mc_info.mgid.raw)); + case MCAST_STATE_ATTACHING: + return sprintf(buf, "state=Attaching MGID:" VNIC_GID_FMT + " MLID:%X\n", + VNIC_GID_RAW_ARG(path->viport->mc_info.mgid.raw), + path->viport->mc_info.mlid); + case MCAST_STATE_JOINED_ATTACHED: + return sprintf(buf, + "state=Joined & Attached MGID:" VNIC_GID_FMT + " MLID:%X\n", + VNIC_GID_RAW_ARG(path->viport->mc_info.mgid.raw), + path->viport->mc_info.mlid); + case MCAST_STATE_DETACHING: + return sprintf(buf, "state=Detaching MGID: " VNIC_GID_FMT "\n", + VNIC_GID_RAW_ARG(path->viport->mc_info.mgid.raw)); + case MCAST_STATE_RETRIED: + return sprintf(buf, "state=Retries Exceeded\n"); + } + return sprintf(buf, "invalid state\n"); +} + +static DEVICE_ATTR(multicast_state, S_IRUGO, show_multicast_state, NULL); + +static ssize_t show_instance(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + + if (!path->viport) + return sprintf(buf, "-1\n"); + else + return sprintf(buf, "%d\n", + path->viport->control.config->vnic_instance); +} + +static ssize_t store_instance(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + int token; + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct netpath *path = container_of(info, struct netpath, dev_info); + char *dup = kstrdup(buf, GFP_KERNEL); + substring_t instance = { dup, dup + strlen(dup) }; + + if (match_int(&instance, &token)) { + printk(KERN_WARNING PFX "bad instance parameter '%s'\n", buf); + return -EINVAL; + } + + if (token > 255 || token < 0) { + printk(KERN_WARNING PFX "instance parameter must be" + " >= 0 and <= 255\n"); + return -EINVAL; + } + + if (path->second_bias) + path->parent->params.secondary_path.instance = token; + else + path->parent->params.primary_path.instance = token; + + kfree(dup); + return count; +} + +static DEVICE_ATTR(instance, S_IRUGO | S_IWUSR, show_instance, store_instance); + +static struct attribute *vnic_path_attrs[] = { + &dev_attr_viport_state.attr, + &dev_attr_link_state.attr, + &dev_attr_heartbeat.attr, + &dev_attr_ioc_guid.attr, + &dev_attr_dgid.attr, + &dev_attr_pkey.attr, + &dev_attr_hca_info.attr, + &dev_attr_ioc_string.attr, + &dev_attr_multicast_state.attr, + &dev_attr_instance.attr, + NULL +}; + +struct attribute_group vnic_path_attr_group = { + .attrs = vnic_path_attrs, +}; + +void vnic_cleanup_path_files(struct netpath *path) +{ + sysfs_remove_group(&path->dev_info.dev.kobj, + &vnic_path_attr_group); + device_unregister(&path->dev_info.dev); + wait_for_completion(&path->dev_info.released); +} + +static inline void update_dgids(u8 *old, u8 *new, char *vnic_name, + char *path_name) +{ + int i; + + if (!memcmp(old, new, 16)) + return; + + printk(KERN_INFO PFX "Changing dgid from 0x"); + print_dgid(old); + printk(" to 0x"); + print_dgid(new); + printk(" for %s path of %s\n", path_name, vnic_name); + for (i = 0; i < 16; i++) + old[i] = new[i]; +} + +static inline void update_ioc_guids(struct vnic_path_param *params, + struct netpath *path, + char *vnic_name, char *path_name) +{ + u64 sid; + + if (path->viport->config->ioc_guid == params->ioc_guid) + return; + + printk(KERN_INFO PFX "Changing IOC GUID from 0x%llx to 0x%llx " + "for %s path of %s\n", + __be64_to_cpu(path->viport->config->ioc_guid), + __be64_to_cpu(params->ioc_guid), path_name, vnic_name); + + path->viport->config->ioc_guid = params->ioc_guid; + + sid = (SST_AGN << 56) | (SST_OUI << 32) | (CONTROL_PATH_ID << 8) + | IOC_NUMBER(be64_to_cpu(params->ioc_guid)); + + path->viport->config->control_config.ib_config.service_id = + cpu_to_be64(sid); + + sid = (SST_AGN << 56) | (SST_OUI << 32) | (DATA_PATH_ID << 8) + | IOC_NUMBER(be64_to_cpu(params->ioc_guid)); + + path->viport->config->data_config.ib_config.service_id = + cpu_to_be64(sid); +} + +static inline void update_pkeys(__be16 *old, __be16 *new, char *vnic_name, + char *path_name) +{ + if (*old == *new) + return; + + printk(KERN_INFO PFX "Changing P_Key from 0x%x to 0x%x " + "for %s path of %s\n", *old, *new, + path_name, vnic_name); + *old = *new; +} + +static void update_ioc_strings(struct vnic_path_param *params, + struct netpath *path, char *path_name) +{ + if (!strcmp(params->ioc_string, path->viport->config->ioc_string)) + return; + + printk(KERN_INFO PFX "Changing ioc_string to %s for %s path of %s\n", + params->ioc_string, path_name, + path->parent->config->name); + + strcpy(path->viport->config->ioc_string, params->ioc_string); +} + +static void update_path_parameters(struct vnic_path_param *params, + struct netpath *path) +{ + update_dgids(path->viport->config->path_info.path.dgid.raw, + params->dgid, path->parent->config->name, + (path->second_bias ? "secondary" : "primary")); + + update_ioc_guids(params, path, path->parent->config->name, + (path->second_bias ? "secondary" : "primary")); + + update_pkeys(&path->viport->config->path_info.path.pkey, + ¶ms->pkey, path->parent->config->name, + (path->second_bias ? "secondary" : "primary")); + + update_ioc_strings(params, path, + (path->second_bias ? "secondary" : "primary")); +} + +static ssize_t update_params_and_connect(struct vnic_path_param *params, + struct netpath *path, size_t count) +{ + if (is_dgid_zero(params->dgid) && params->ioc_guid != 0 && + params->pkey != 0) { + + if (!memcmp(path->viport->config->path_info.path.dgid.raw, + params->dgid, 16) && + params->ioc_guid == path->viport->config->ioc_guid && + params->pkey == path->viport->config->path_info.path.pkey) { + + printk(KERN_WARNING PFX "All of the dgid, ioc_guid and " + "pkeys are same as the existing" + " one. Not updating values.\n"); + return -EINVAL; + } else { + if (path->viport->state == VIPORT_CONNECTED) { + printk(KERN_WARNING PFX "%s path of %s " + "interface is already in connected " + "state. Not updating values.\n", + (path->second_bias ? "Secondary" : "Primary"), + path->parent->config->name); + return -EINVAL; + } else { + update_path_parameters(params, path); + viport_kick(path->viport); + vnic_disconnected(path->parent, path); + return count; + } + } + } else { + printk(KERN_WARNING PFX "Either dgid, iocguid, pkey is zero. " + "No update.\n"); + return -EINVAL; + } +} + +static ssize_t vnic_create_primary(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic_ib_port *target = + container_of(info, struct vnic_ib_port, pdev_info); + + int ret = -EINVAL; + struct vnic *vnic; + struct list_head *ptr; + int vnic_found = 0; + + list_for_each(ptr, &vnic_list) { + vnic = list_entry(ptr, struct vnic, list_ptrs); + if (!strncmp(vnic->config->name, buf, IFNAMSIZ)) { + vnic_found = 1; + vnic->params.primary_path.opt_mask |= VNIC_OPT_NAME; + break; + } + } + + if (!vnic_found) { + printk(KERN_ERR PFX + "You need to first echo the interface name into " + "/sys/class/infiniband_qlgc_vnic/interfaces/create_vnic\n"); + goto out; + } + + if (vnic_all_connection_params(&vnic->params.primary_path)) + goto out; + + if (vnic->primary_path.viport) { + if (vnic->primary_path.viport->state == VIPORT_DISCONNECTED) + ret = update_params_and_connect(&vnic->params.primary_path, + &vnic->primary_path, count); + else + printk(KERN_ERR PFX "No connection establishment. " + "Connection already present.\n"); + goto out; + } + + vnic->params.primary_path.ibdev = target->dev->dev; + vnic->params.primary_path.ibport = target; + vnic->params.primary_path.port = target->port_num; + + if (vnic->params.rx_csum != -1) + vnic->config->use_rx_csum = vnic->params.rx_csum; + if (vnic->params.tx_csum != -1) + vnic->config->use_tx_csum = vnic->params.tx_csum; + + if (vnic_create_netpath(&vnic->primary_path, &vnic->params.primary_path)) { + printk(KERN_ERR PFX "creating primary netpath failed\n"); + goto free_vnic; + } + + if (vnic && !vnic->primary_path.viport) { + printk(KERN_ERR PFX "no valid netpaths\n"); + goto free_vnic; + } + + return count; + +free_vnic: + vnic_free(vnic); + ret = -EINVAL; +out: + return ret; +} + +DEVICE_ATTR(create_primary, S_IWUSR, NULL, vnic_create_primary); + +static ssize_t vnic_create_secondary(struct device *dev, + struct device_attribute *dev_attr, + const char *buf, size_t count) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic_ib_port *target = + container_of(info, struct vnic_ib_port, pdev_info); + + struct vnic *vnic = NULL; + int ret = -EINVAL; + struct list_head *ptr; + int vnic_found = 0; + + list_for_each(ptr, &vnic_list) { + vnic = list_entry(ptr, struct vnic, list_ptrs); + if (!strncmp(vnic->config->name, buf, IFNAMSIZ)) { + vnic_found = 1; + vnic->params.secondary_path.opt_mask |= VNIC_OPT_NAME; + break; + } + } + + if (!vnic_found) { + printk(KERN_ERR PFX + "You need to first echo the interface name into " + "/sys/class/infiniband_qlgc_vnic/interfaces/create_vnic\n"); + goto out; + } + + if (vnic_all_connection_params(&vnic->params.secondary_path)) + goto out; + + if (vnic->secondary_path.viport) { + if (vnic->secondary_path.viport->state == VIPORT_DISCONNECTED) + ret = update_params_and_connect(&vnic->params.secondary_path, + &vnic->secondary_path, count); + else + printk(KERN_ERR PFX "No connection establishment. " + "Connection already present.\n"); + goto out; + } + + vnic->params.secondary_path.ibdev = target->dev->dev; + vnic->params.secondary_path.ibport = target; + vnic->params.secondary_path.port = target->port_num; + + if (vnic_create_netpath(&vnic->secondary_path, &vnic->params.secondary_path)) { + printk(KERN_ERR PFX "creating secondary netpath failed\n"); + goto out; + } + + return count; + +out: + return ret; +} + +DEVICE_ATTR(create_secondary, S_IWUSR, NULL, vnic_create_secondary); diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h new file mode 100644 index 0000000..1a52344 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_sys.h @@ -0,0 +1,52 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_SYS_H_INCLUDED +#define VNIC_SYS_H_INCLUDED + +struct dev_info { + struct device dev; + struct completion released; +}; + +extern struct class vnic_class; +extern struct dev_info interface_dev; +extern struct attribute_group vnic_dev_attr_group; +extern struct attribute_group vnic_path_attr_group; +extern struct device_attribute dev_attr_create_primary; +extern struct device_attribute dev_attr_create_secondary; +extern struct device_attribute dev_attr_delete_vnic; +extern struct device_attribute dev_attr_create_vnic; + +extern void vnic_release_dev(struct device *dev); + +#endif /*VNIC_SYS_H_INCLUDED*/ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:06:47 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:06:47 -0400 Subject: [ofa-general] [PATCH v4 09/14] QLogic VNIC: IB Multicast for Ethernet broadcast/multicast In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210647.11186.25121.stgit@dale> From: Usha Srinivasan Implementation of ethernet broadcasting and multicasting for QLogic VNIC interface by making use of underlying IB multicasting. Signed-off-by: Usha Srinivasan Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c | 319 +++++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h | 77 +++++ 2 files changed, 396 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c new file mode 100644 index 0000000..f40ea20 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.c @@ -0,0 +1,319 @@ +/* + * Copyright (c) 2008 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include +#include +#include "vnic_viport.h" +#include "vnic_main.h" +#include "vnic_util.h" + +static inline void vnic_set_multicast_state_invalid(struct viport *viport) +{ + viport->mc_info.state = MCAST_STATE_INVALID; + viport->mc_info.mc = NULL; + memset(&viport->mc_info.mgid, 0, sizeof(union ib_gid)); +} + +int vnic_mc_init(struct viport *viport) +{ + MCAST_FUNCTION("vnic_mc_init %p\n", viport); + vnic_set_multicast_state_invalid(viport); + viport->mc_info.retries = 0; + spin_lock_init(&viport->mc_info.lock); + + return 0; +} + +void vnic_mc_uninit(struct viport *viport) +{ + unsigned long flags; + MCAST_FUNCTION("vnic_mc_uninit %p\n", viport); + + spin_lock_irqsave(&viport->mc_info.lock, flags); + if ((viport->mc_info.state != MCAST_STATE_INVALID) && + (viport->mc_info.state != MCAST_STATE_RETRIED)) { + MCAST_ERROR("%s mcast state is not INVALID or RETRIED %d\n", + control_ifcfg_name(&viport->control), + viport->mc_info.state); + } + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + MCAST_FUNCTION("vnic_mc_uninit done\n"); +} + + +/* This function is called when NEED_MCAST_COMPLETION is set. + * It finishes off the join multicast work. + */ +int vnic_mc_join_handle_completion(struct viport *viport) +{ + unsigned int ret = 0; + + MCAST_FUNCTION("vnic_mc_join_handle_completion()\n"); + if (viport->mc_info.state != MCAST_STATE_JOINING) { + MCAST_ERROR("%s unexpected mcast state in handle_completion: " + " %d\n", control_ifcfg_name(&viport->control), + viport->mc_info.state); + ret = -1; + goto out; + } + viport->mc_info.state = MCAST_STATE_ATTACHING; + MCAST_INFO("%s Attaching QP %lx mgid:" + VNIC_GID_FMT " mlid:%x\n", + control_ifcfg_name(&viport->control), jiffies, + VNIC_GID_RAW_ARG(viport->mc_info.mgid.raw), + viport->mc_info.mlid); + ret = ib_attach_mcast(viport->mc_data.ib_conn.qp, &viport->mc_info.mgid, + viport->mc_info.mlid); + if (ret) { + MCAST_ERROR("%s Attach mcast qp failed %d\n", + control_ifcfg_name(&viport->control), ret); + ret = -1; + goto out; + } + viport->mc_info.state = MCAST_STATE_JOINED_ATTACHED; + MCAST_INFO("%s UD QP successfully attached to mcast group\n", + control_ifcfg_name(&viport->control)); + +out: + return ret; +} + +/* NOTE: ib_sa.h says "returning a non-zero value from this callback will + * result in destroying the multicast tracking structure. + */ +static int vnic_mc_join_complete(int status, + struct ib_sa_multicast *multicast) +{ + struct viport *viport = (struct viport *)multicast->context; + unsigned long flags; + + MCAST_FUNCTION("vnic_mc_join_complete() status:%x\n", status); + if (status) { + spin_lock_irqsave(&viport->mc_info.lock, flags); + if (status == -ENETRESET) { + vnic_set_multicast_state_invalid(viport); + viport->mc_info.retries = 0; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + MCAST_ERROR("%s got ENETRESET\n", + control_ifcfg_name(&viport->control)); + goto out; + } + /* perhaps the mcgroup hasn't yet been created - retry */ + viport->mc_info.retries++; + viport->mc_info.mc = NULL; + if (viport->mc_info.retries > MAX_MCAST_JOIN_RETRIES) { + viport->mc_info.state = MCAST_STATE_RETRIED; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + MCAST_ERROR("%s join failed 0x%x - max retries:%d " + "exceeded\n", + control_ifcfg_name(&viport->control), + status, viport->mc_info.retries); + } else { + viport->mc_info.state = MCAST_STATE_INVALID; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + spin_lock_irqsave(&viport->lock, flags); + viport->updates |= NEED_MCAST_JOIN; + spin_unlock_irqrestore(&viport->lock, flags); + viport_kick(viport); + MCAST_ERROR("%s join failed 0x%x - retrying; " + "retries:%d\n", + control_ifcfg_name(&viport->control), + status, viport->mc_info.retries); + } + goto out; + } + + /* finish join work from main state loop for viport - in case + * the work itself cannot be done in a callback environment */ + spin_lock_irqsave(&viport->lock, flags); + viport->mc_info.mlid = be16_to_cpu(multicast->rec.mlid); + viport->updates |= NEED_MCAST_COMPLETION; + spin_unlock_irqrestore(&viport->lock, flags); + viport_kick(viport); + MCAST_INFO("%s setting NEED_MCAST_COMPLETION %x %x\n", + control_ifcfg_name(&viport->control), + multicast->rec.mlid, viport->mc_info.mlid); +out: + return status; +} + +void vnic_mc_join_setup(struct viport *viport, union ib_gid *mgid) +{ + unsigned long flags; + + MCAST_FUNCTION("in vnic_mc_join_setup\n"); + spin_lock_irqsave(&viport->mc_info.lock, flags); + if (viport->mc_info.state != MCAST_STATE_INVALID) { + if (viport->mc_info.state == MCAST_STATE_DETACHING) + MCAST_ERROR("%s detach in progress\n", + control_ifcfg_name(&viport->control)); + else if (viport->mc_info.state == MCAST_STATE_RETRIED) + MCAST_ERROR("%s max join retries exceeded\n", + control_ifcfg_name(&viport->control)); + else { + /* join/attach in progress or done */ + /* verify that the current mgid is same as prev mgid */ + if (memcmp(mgid, &viport->mc_info.mgid, sizeof(union ib_gid)) != 0) { + /* Separate MGID for each IOC */ + MCAST_ERROR("%s Multicast Group MGIDs not " + "unique; mgids: " VNIC_GID_FMT + " " VNIC_GID_FMT "\n", + control_ifcfg_name(&viport->control), + VNIC_GID_RAW_ARG(mgid->raw), + VNIC_GID_RAW_ARG(viport->mc_info.mgid.raw)); + } else + MCAST_INFO("%s join already issued: %d\n", + control_ifcfg_name(&viport->control), + viport->mc_info.state); + + } + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + return; + } + viport->mc_info.mgid = *mgid; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + spin_lock_irqsave(&viport->lock, flags); + viport->updates |= NEED_MCAST_JOIN; + spin_unlock_irqrestore(&viport->lock, flags); + viport_kick(viport); + MCAST_INFO("%s setting NEED_MCAST_JOIN \n", + control_ifcfg_name(&viport->control)); +} + +int vnic_mc_join(struct viport *viport) +{ + struct ib_sa_mcmember_rec rec; + ib_sa_comp_mask comp_mask; + unsigned long flags; + int ret = 0; + + MCAST_FUNCTION("vnic_mc_join()\n"); + if (!viport->mc_data.ib_conn.qp) { + MCAST_ERROR("%s qp is NULL\n", + control_ifcfg_name(&viport->control)); + ret = -1; + goto out; + } + spin_lock_irqsave(&viport->mc_info.lock, flags); + if (viport->mc_info.state != MCAST_STATE_INVALID) { + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + MCAST_INFO("%s Multicast join already issued\n", + control_ifcfg_name(&viport->control)); + goto out; + } + viport->mc_info.state = MCAST_STATE_JOINING; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + + memset(&rec, 0, sizeof(rec)); + rec.join_state = 2; /* bit 1 is Nonmember */ + rec.mgid = viport->mc_info.mgid; + rec.port_gid = viport->config->path_info.path.sgid; + + comp_mask = IB_SA_MCMEMBER_REC_MGID | + IB_SA_MCMEMBER_REC_PORT_GID | + IB_SA_MCMEMBER_REC_JOIN_STATE; + + MCAST_INFO("%s Joining Multicast group%lx mgid:" + VNIC_GID_FMT " port_gid: " VNIC_GID_FMT "\n", + control_ifcfg_name(&viport->control), jiffies, + VNIC_GID_RAW_ARG(rec.mgid.raw), + VNIC_GID_RAW_ARG(rec.port_gid.raw)); + + viport->mc_info.mc = ib_sa_join_multicast(&vnic_sa_client, + viport->config->ibdev, viport->config->port, + &rec, comp_mask, GFP_KERNEL, + vnic_mc_join_complete, viport); + + if (IS_ERR(viport->mc_info.mc)) { + MCAST_ERROR("%s Multicast joining failed " VNIC_GID_FMT + ".\n", + control_ifcfg_name(&viport->control), + VNIC_GID_RAW_ARG(rec.mgid.raw)); + viport->mc_info.state = MCAST_STATE_INVALID; + ret = -1; + goto out; + } + MCAST_INFO("%s Multicast group join issued mgid:" + VNIC_GID_FMT " port_gid: " VNIC_GID_FMT "\n", + control_ifcfg_name(&viport->control), + VNIC_GID_RAW_ARG(rec.mgid.raw), + VNIC_GID_RAW_ARG(rec.port_gid.raw)); +out: + return ret; +} + +void vnic_mc_leave(struct viport *viport) +{ + unsigned long flags; + unsigned int ret; + struct ib_sa_multicast *mc; + + MCAST_FUNCTION("vnic_mc_leave()\n"); + + spin_lock_irqsave(&viport->mc_info.lock, flags); + if ((viport->mc_info.state == MCAST_STATE_INVALID) || + (viport->mc_info.state == MCAST_STATE_RETRIED)) { + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + return; + } + + if (viport->mc_info.state == MCAST_STATE_JOINED_ATTACHED) { + + viport->mc_info.state = MCAST_STATE_DETACHING; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + ret = ib_detach_mcast(viport->mc_data.ib_conn.qp, + &viport->mc_info.mgid, + viport->mc_info.mlid); + if (ret) { + MCAST_ERROR("%s UD QP Detach failed %d\n", + control_ifcfg_name(&viport->control), ret); + return; + } + MCAST_INFO("%s UD QP detached succesfully\n", + control_ifcfg_name(&viport->control)); + spin_lock_irqsave(&viport->mc_info.lock, flags); + } + mc = viport->mc_info.mc; + vnic_set_multicast_state_invalid(viport); + viport->mc_info.retries = 0; + spin_unlock_irqrestore(&viport->mc_info.lock, flags); + + if (mc) { + MCAST_INFO("%s Freeing up multicast structure.\n", + control_ifcfg_name(&viport->control)); + ib_sa_free_multicast(mc); + } + MCAST_FUNCTION("vnic_mc_leave done\n"); + return; +} diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h new file mode 100644 index 0000000..e049180 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_multicast.h @@ -0,0 +1,77 @@ +/* + * Copyright (c) 2008 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef __VNIC_MULTICAST_H__ +#define __VNIC_MULTTCAST_H__ + +enum { + MCAST_STATE_INVALID = 0x00, /* join not attempted or failed */ + MCAST_STATE_JOINING = 0x01, /* join mcgroup in progress */ + MCAST_STATE_ATTACHING = 0x02, /* join completed with success, + * attach qp to mcgroup in progress + */ + MCAST_STATE_JOINED_ATTACHED = 0x03, /* join completed with success */ + MCAST_STATE_DETACHING = 0x04, /* detach qp in progress */ + MCAST_STATE_RETRIED = 0x05, /* retried join and failed */ +}; + +#define MAX_MCAST_JOIN_RETRIES 5 /* used to retry join */ + +struct mc_info { + u8 state; + spinlock_t lock; + union ib_gid mgid; + u16 mlid; + struct ib_sa_multicast *mc; + u8 retries; +}; + + +int vnic_mc_init(struct viport *viport); +void vnic_mc_uninit(struct viport *viport); +extern char *control_ifcfg_name(struct control *control); + +/* This function is called when a viport gets a multicast mgid from EVIC + and must join the multicast group. It sets up NEED_MCAST_JOIN flag, which + results in vnic_mc_join being called later. */ +void vnic_mc_join_setup(struct viport *viport, union ib_gid *mgid); + +/* This function is called when NEED_MCAST_JOIN flag is set. */ +int vnic_mc_join(struct viport *viport); + +/* This function is called when NEED_MCAST_COMPLETION is set. + It finishes off the join multicast work. */ +int vnic_mc_join_handle_completion(struct viport *viport); + +void vnic_mc_leave(struct viport *viport); + +#endif /* __VNIC_MULTICAST_H__ */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:07:17 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:07:17 -0400 Subject: [ofa-general] [PATCH v4 10/14] QLogic VNIC: Driver Statistics collection In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210717.11186.54173.stgit@dale> From: Amar Mudrankit Collection of statistics about QLogic VNIC interfaces is implemented in this patch. Signed-off-by: Amar Mudrankit Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath --- drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c | 234 ++++++++++++ drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h | 497 +++++++++++++++++++++++++ 2 files changed, 731 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c b/drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c new file mode 100644 index 0000000..d11a8df --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_stats.c @@ -0,0 +1,234 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include +#include + +#include "vnic_main.h" + +cycles_t vnic_recv_ref; + +/* + * TODO: Statistics reporting for control path, data path, + * RDMA times, IOs etc + * + */ +static ssize_t show_lifetime(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + cycles_t time = get_cycles() - vnic->statistics.start_time; + + return sprintf(buf, "%llu\n", (unsigned long long)time); +} + +static DEVICE_ATTR(lifetime, S_IRUGO, show_lifetime, NULL); + +static ssize_t show_conntime(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + if (vnic->statistics.conn_time) + return sprintf(buf, "%llu\n", + (unsigned long long)vnic->statistics.conn_time); + return 0; +} + +static DEVICE_ATTR(connection_time, S_IRUGO, show_conntime, NULL); + +static ssize_t show_disconnects(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + u32 num; + + if (vnic->statistics.disconn_ref) + num = vnic->statistics.disconn_num + 1; + else + num = vnic->statistics.disconn_num; + + return sprintf(buf, "%d\n", num); +} + +static DEVICE_ATTR(disconnects, S_IRUGO, show_disconnects, NULL); + +static ssize_t show_total_disconn_time(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + cycles_t time; + + if (vnic->statistics.disconn_ref) + time = vnic->statistics.disconn_time + + get_cycles() - vnic->statistics.disconn_ref; + else + time = vnic->statistics.disconn_time; + + return sprintf(buf, "%llu\n", (unsigned long long)time); +} + +static DEVICE_ATTR(total_disconn_time, S_IRUGO, show_total_disconn_time, NULL); + +static ssize_t show_carrier_losses(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + u32 num; + + if (vnic->statistics.carrier_ref) + num = vnic->statistics.carrier_off_num + 1; + else + num = vnic->statistics.carrier_off_num; + + return sprintf(buf, "%d\n", num); +} + +static DEVICE_ATTR(carrier_losses, S_IRUGO, show_carrier_losses, NULL); + +static ssize_t show_total_carr_loss_time(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + cycles_t time; + + if (vnic->statistics.carrier_ref) + time = vnic->statistics.carrier_off_time + + get_cycles() - vnic->statistics.carrier_ref; + else + time = vnic->statistics.carrier_off_time; + + return sprintf(buf, "%llu\n", (unsigned long long)time); +} + +static DEVICE_ATTR(total_carrier_loss_time, S_IRUGO, + show_total_carr_loss_time, NULL); + +static ssize_t show_total_recv_time(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + return sprintf(buf, "%llu\n", + (unsigned long long)vnic->statistics.recv_time); +} + +static DEVICE_ATTR(total_recv_time, S_IRUGO, show_total_recv_time, NULL); + +static ssize_t show_recvs(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + return sprintf(buf, "%d\n", vnic->statistics.recv_num); +} + +static DEVICE_ATTR(recvs, S_IRUGO, show_recvs, NULL); + +static ssize_t show_multicast_recvs(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + return sprintf(buf, "%d\n", vnic->statistics.multicast_recv_num); +} + +static DEVICE_ATTR(multicast_recvs, S_IRUGO, show_multicast_recvs, NULL); + +static ssize_t show_total_xmit_time(struct device *dev, + struct device_attribute *dev_attr, + char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + return sprintf(buf, "%llu\n", + (unsigned long long)vnic->statistics.xmit_time); +} + +static DEVICE_ATTR(total_xmit_time, S_IRUGO, show_total_xmit_time, NULL); + +static ssize_t show_xmits(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + return sprintf(buf, "%d\n", vnic->statistics.xmit_num); +} + +static DEVICE_ATTR(xmits, S_IRUGO, show_xmits, NULL); + +static ssize_t show_failed_xmits(struct device *dev, + struct device_attribute *dev_attr, char *buf) +{ + struct dev_info *info = container_of(dev, struct dev_info, dev); + struct vnic *vnic = container_of(info, struct vnic, stat_info); + + return sprintf(buf, "%d\n", vnic->statistics.xmit_fail); +} + +static DEVICE_ATTR(failed_xmits, S_IRUGO, show_failed_xmits, NULL); + +static struct attribute *vnic_stats_attrs[] = { + &dev_attr_lifetime.attr, + &dev_attr_xmits.attr, + &dev_attr_total_xmit_time.attr, + &dev_attr_failed_xmits.attr, + &dev_attr_recvs.attr, + &dev_attr_multicast_recvs.attr, + &dev_attr_total_recv_time.attr, + &dev_attr_connection_time.attr, + &dev_attr_disconnects.attr, + &dev_attr_total_disconn_time.attr, + &dev_attr_carrier_losses.attr, + &dev_attr_total_carrier_loss_time.attr, + NULL +}; + +struct attribute_group vnic_stats_attr_group = { + .attrs = vnic_stats_attrs, +}; diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h new file mode 100644 index 0000000..a241b71 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_stats.h @@ -0,0 +1,497 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_STATS_H_INCLUDED +#define VNIC_STATS_H_INCLUDED + +#include "vnic_main.h" +#include "vnic_ib.h" +#include "vnic_sys.h" + +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS + +static inline void vnic_connected_stats(struct vnic *vnic) +{ + if (vnic->statistics.conn_time == 0) { + vnic->statistics.conn_time = + get_cycles() - vnic->statistics.start_time; + } + + if (vnic->statistics.disconn_ref != 0) { + vnic->statistics.disconn_time += + get_cycles() - vnic->statistics.disconn_ref; + vnic->statistics.disconn_num++; + vnic->statistics.disconn_ref = 0; + } + +} + +static inline void vnic_stop_xmit_stats(struct vnic *vnic) +{ + if (vnic->statistics.xmit_ref == 0) + vnic->statistics.xmit_ref = get_cycles(); +} + +static inline void vnic_restart_xmit_stats(struct vnic *vnic) +{ + if (vnic->statistics.xmit_ref != 0) { + vnic->statistics.xmit_off_time += + get_cycles() - vnic->statistics.xmit_ref; + vnic->statistics.xmit_off_num++; + vnic->statistics.xmit_ref = 0; + } +} + +static inline void vnic_recv_pkt_stats(struct vnic *vnic) +{ + vnic->statistics.recv_time += get_cycles() - vnic_recv_ref; + vnic->statistics.recv_num++; +} + +static inline void vnic_multicast_recv_pkt_stats(struct vnic *vnic) +{ + vnic->statistics.multicast_recv_num++; +} + +static inline void vnic_pre_pkt_xmit_stats(cycles_t *time) +{ + *time = get_cycles(); +} + +static inline void vnic_post_pkt_xmit_stats(struct vnic *vnic, + cycles_t time) +{ + vnic->statistics.xmit_time += get_cycles() - time; + vnic->statistics.xmit_num++; + +} + +static inline void vnic_xmit_fail_stats(struct vnic *vnic) +{ + vnic->statistics.xmit_fail++; +} + +static inline void vnic_carrier_loss_stats(struct vnic *vnic) +{ + if (vnic->statistics.carrier_ref != 0) { + vnic->statistics.carrier_off_time += + get_cycles() - vnic->statistics.carrier_ref; + vnic->statistics.carrier_off_num++; + vnic->statistics.carrier_ref = 0; + } +} + +static inline int vnic_setup_stats_files(struct vnic *vnic) +{ + init_completion(&vnic->stat_info.released); + vnic->stat_info.dev.class = NULL; + vnic->stat_info.dev.parent = &vnic->dev_info.dev; + vnic->stat_info.dev.release = vnic_release_dev; + snprintf(vnic->stat_info.dev.bus_id, BUS_ID_SIZE, + "stats"); + + if (device_register(&vnic->stat_info.dev)) { + SYS_ERROR("create_vnic: error in registering" + " stat class dev\n"); + goto stats_out; + } + + if (sysfs_create_group(&vnic->stat_info.dev.kobj, + &vnic_stats_attr_group)) + goto err_stats_file; + + return 0; +err_stats_file: + device_unregister(&vnic->stat_info.dev); + wait_for_completion(&vnic->stat_info.released); +stats_out: + return -1; +} + +static inline void vnic_cleanup_stats_files(struct vnic *vnic) +{ + sysfs_remove_group(&vnic->dev_info.dev.kobj, + &vnic_stats_attr_group); + device_unregister(&vnic->stat_info.dev); + wait_for_completion(&vnic->stat_info.released); +} + +static inline void vnic_disconn_stats(struct vnic *vnic) +{ + if (!vnic->statistics.disconn_ref) + vnic->statistics.disconn_ref = get_cycles(); + + if (vnic->statistics.carrier_ref == 0) + vnic->statistics.carrier_ref = get_cycles(); +} + +static inline void vnic_alloc_stats(struct vnic *vnic) +{ + vnic->statistics.start_time = get_cycles(); +} + +static inline void control_note_rsptime_stats(cycles_t *time) +{ + *time = get_cycles(); +} + +static inline void control_update_rsptime_stats(struct control *control, + cycles_t response_time) +{ + response_time -= control->statistics.request_time; + control->statistics.response_time += response_time; + control->statistics.response_num++; + if (control->statistics.response_max < response_time) + control->statistics.response_max = response_time; + if ((control->statistics.response_min == 0) || + (control->statistics.response_min > response_time)) + control->statistics.response_min = response_time; + +} + +static inline void control_note_reqtime_stats(struct control *control) +{ + control->statistics.request_time = get_cycles(); +} + +static inline void control_timeout_stats(struct control *control) +{ + control->statistics.timeout_num++; +} + +static inline void data_kickreq_stats(struct data *data) +{ + data->statistics.kick_reqs++; +} + +static inline void data_no_xmitbuf_stats(struct data *data) +{ + data->statistics.no_xmit_bufs++; +} + +static inline void data_xmits_stats(struct data *data) +{ + data->statistics.xmit_num++; +} + +static inline void data_recvs_stats(struct data *data) +{ + data->statistics.recv_num++; +} + +static inline void data_note_kickrcv_time(void) +{ + vnic_recv_ref = get_cycles(); +} + +static inline void data_rcvkicks_stats(struct data *data) +{ + data->statistics.kick_recvs++; +} + + +static inline void vnic_ib_conntime_stats(struct vnic_ib_conn *ib_conn) +{ + ib_conn->statistics.connection_time = get_cycles(); +} + +static inline void vnic_ib_note_comptime_stats(cycles_t *time) +{ + *time = get_cycles(); +} + +static inline void vnic_ib_callback_stats(struct vnic_ib_conn *ib_conn) +{ + ib_conn->statistics.num_callbacks++; +} + +static inline void vnic_ib_comp_stats(struct vnic_ib_conn *ib_conn, + u32 *comp_num) +{ + ib_conn->statistics.num_ios++; + *comp_num = *comp_num + 1; + +} + +static inline void vnic_ib_io_stats(struct io *io, + struct vnic_ib_conn *ib_conn, + cycles_t comp_time) +{ + if ((io->type == RECV) || (io->type == RECV_UD)) + io->time = comp_time; + else if (io->type == RDMA) { + ib_conn->statistics.rdma_comp_time += comp_time - io->time; + ib_conn->statistics.rdma_comp_ios++; + } else if (io->type == SEND) { + ib_conn->statistics.send_comp_time += comp_time - io->time; + ib_conn->statistics.send_comp_ios++; + } +} + +static inline void vnic_ib_maxio_stats(struct vnic_ib_conn *ib_conn, + u32 comp_num) +{ + if (comp_num > ib_conn->statistics.max_ios) + ib_conn->statistics.max_ios = comp_num; +} + +static inline void vnic_ib_connected_time_stats(struct vnic_ib_conn *ib_conn) +{ + ib_conn->statistics.connection_time = + get_cycles() - ib_conn->statistics.connection_time; + +} + +static inline void vnic_ib_pre_rcvpost_stats(struct vnic_ib_conn *ib_conn, + struct io *io, + cycles_t *time) +{ + *time = get_cycles(); + if (io->time != 0) { + ib_conn->statistics.recv_comp_time += *time - io->time; + ib_conn->statistics.recv_comp_ios++; + } + +} + +static inline void vnic_ib_post_rcvpost_stats(struct vnic_ib_conn *ib_conn, + cycles_t time) +{ + ib_conn->statistics.recv_post_time += get_cycles() - time; + ib_conn->statistics.recv_post_ios++; +} + +static inline void vnic_ib_pre_sendpost_stats(struct io *io, + cycles_t *time) +{ + io->time = *time = get_cycles(); +} + +static inline void vnic_ib_post_sendpost_stats(struct vnic_ib_conn *ib_conn, + struct io *io, + cycles_t time) +{ + time = get_cycles() - time; + if (io->swr.opcode == IB_WR_RDMA_WRITE) { + ib_conn->statistics.rdma_post_time += time; + ib_conn->statistics.rdma_post_ios++; + } else { + ib_conn->statistics.send_post_time += time; + ib_conn->statistics.send_post_ios++; + } +} +#else /*CONFIG_INIFINIBAND_VNIC_STATS*/ + +static inline void vnic_connected_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_stop_xmit_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_restart_xmit_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_recv_pkt_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_multicast_recv_pkt_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_pre_pkt_xmit_stats(cycles_t *time) +{ + ; +} + +static inline void vnic_post_pkt_xmit_stats(struct vnic *vnic, + cycles_t time) +{ + ; +} + +static inline void vnic_xmit_fail_stats(struct vnic *vnic) +{ + ; +} + +static inline int vnic_setup_stats_files(struct vnic *vnic) +{ + return 0; +} + +static inline void vnic_cleanup_stats_files(struct vnic *vnic) +{ + ; +} + +static inline void vnic_carrier_loss_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_disconn_stats(struct vnic *vnic) +{ + ; +} + +static inline void vnic_alloc_stats(struct vnic *vnic) +{ + ; +} + +static inline void control_note_rsptime_stats(cycles_t *time) +{ + ; +} + +static inline void control_update_rsptime_stats(struct control *control, + cycles_t response_time) +{ + ; +} + +static inline void control_note_reqtime_stats(struct control *control) +{ + ; +} + +static inline void control_timeout_stats(struct control *control) +{ + ; +} + +static inline void data_kickreq_stats(struct data *data) +{ + ; +} + +static inline void data_no_xmitbuf_stats(struct data *data) +{ + ; +} + +static inline void data_xmits_stats(struct data *data) +{ + ; +} + +static inline void data_recvs_stats(struct data *data) +{ + ; +} + +static inline void data_note_kickrcv_time(void) +{ + ; +} + +static inline void data_rcvkicks_stats(struct data *data) +{ + ; +} + +static inline void vnic_ib_conntime_stats(struct vnic_ib_conn *ib_conn) +{ + ; +} + +static inline void vnic_ib_note_comptime_stats(cycles_t *time) +{ + ; +} + +static inline void vnic_ib_callback_stats(struct vnic_ib_conn *ib_conn) + +{ + ; +} +static inline void vnic_ib_comp_stats(struct vnic_ib_conn *ib_conn, + u32 *comp_num) +{ + ; +} + +static inline void vnic_ib_io_stats(struct io *io, + struct vnic_ib_conn *ib_conn, + cycles_t comp_time) +{ + ; +} + +static inline void vnic_ib_maxio_stats(struct vnic_ib_conn *ib_conn, + u32 comp_num) +{ + ; +} + +static inline void vnic_ib_connected_time_stats(struct vnic_ib_conn *ib_conn) +{ + ; +} + +static inline void vnic_ib_pre_rcvpost_stats(struct vnic_ib_conn *ib_conn, + struct io *io, + cycles_t *time) +{ + ; +} + +static inline void vnic_ib_post_rcvpost_stats(struct vnic_ib_conn *ib_conn, + cycles_t time) +{ + ; +} + +static inline void vnic_ib_pre_sendpost_stats(struct io *io, + cycles_t *time) +{ + ; +} + +static inline void vnic_ib_post_sendpost_stats(struct vnic_ib_conn *ib_conn, + struct io *io, + cycles_t time) +{ + ; +} +#endif /*CONFIG_INIFINIBAND_VNIC_STATS*/ + +#endif /*VNIC_STATS_H_INCLUDED*/ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:07:47 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:07:47 -0400 Subject: [ofa-general] [PATCH v4 11/14] QLogic VNIC: Driver utility file - implements various utility macros In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210747.11186.9892.stgit@dale> From: Poornima Kamath This patch adds the driver utility file which mainly contains utility macros for debugging of QLogic VNIC driver. Signed-off-by: Poornima Kamath Signed-off-by: Ramachandra K Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/vnic_util.h | 236 ++++++++++++++++++++++++++ 1 files changed, 236 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/vnic_util.h diff --git a/drivers/infiniband/ulp/qlgc_vnic/vnic_util.h b/drivers/infiniband/ulp/qlgc_vnic/vnic_util.h new file mode 100644 index 0000000..095fa3a --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/vnic_util.h @@ -0,0 +1,236 @@ +/* + * Copyright (c) 2006 QLogic, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef VNIC_UTIL_H_INCLUDED +#define VNIC_UTIL_H_INCLUDED + +#define MODULE_NAME "QLGC_VNIC" + +#define VNIC_MAJORVERSION 1 +#define VNIC_MINORVERSION 1 + +#define ALIGN_DOWN(x, a) ((x)&(~((a)-1))) + +extern u32 vnic_debug; + +enum { + DEBUG_IB_INFO = 0x00000001, + DEBUG_IB_FUNCTION = 0x00000002, + DEBUG_IB_FSTATUS = 0x00000004, + DEBUG_IB_ASSERTS = 0x00000008, + DEBUG_CONTROL_INFO = 0x00000010, + DEBUG_CONTROL_FUNCTION = 0x00000020, + DEBUG_CONTROL_PACKET = 0x00000040, + DEBUG_CONFIG_INFO = 0x00000100, + DEBUG_DATA_INFO = 0x00001000, + DEBUG_DATA_FUNCTION = 0x00002000, + DEBUG_NETPATH_INFO = 0x00010000, + DEBUG_VIPORT_INFO = 0x00100000, + DEBUG_VIPORT_FUNCTION = 0x00200000, + DEBUG_LINK_STATE = 0x00400000, + DEBUG_VNIC_INFO = 0x01000000, + DEBUG_VNIC_FUNCTION = 0x02000000, + DEBUG_MCAST_INFO = 0x04000000, + DEBUG_MCAST_FUNCTION = 0x08000000, + DEBUG_SYS_INFO = 0x10000000, + DEBUG_SYS_VERBOSE = 0x40000000 +}; + +#define PRINT(level, x, fmt, arg...) \ + printk(level "%s: " fmt, MODULE_NAME, ##arg) + +#define PRINT_CONDITIONAL(level, x, condition, fmt, arg...) \ + do { \ + if (condition) \ + printk(level "%s: %s: " fmt, \ + MODULE_NAME, x, ##arg); \ + } while (0) + +#define IB_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "IB", fmt, ##arg) +#define IB_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "IB", fmt, ##arg) + +#define IB_FUNCTION(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "IB", \ + (vnic_debug & DEBUG_IB_FUNCTION), \ + fmt, ##arg) + +#define IB_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "IB", \ + (vnic_debug & DEBUG_IB_INFO), \ + fmt, ##arg) + +#define IB_ASSERT(x) \ + do { \ + if ((vnic_debug & DEBUG_IB_ASSERTS) && !(x)) \ + panic("%s assertion failed, file: %s," \ + " line %d: ", \ + MODULE_NAME, __FILE__, __LINE__) \ + } while (0) + +#define CONTROL_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "CONTROL", fmt, ##arg) +#define CONTROL_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "CONTROL", fmt, ##arg) + +#define CONTROL_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "CONTROL", \ + (vnic_debug & DEBUG_CONTROL_INFO), \ + fmt, ##arg) + +#define CONTROL_FUNCTION(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "CONTROL", \ + (vnic_debug & DEBUG_CONTROL_FUNCTION), \ + fmt, ##arg) + +#define CONTROL_PACKET(pkt) \ + do { \ + if (vnic_debug & DEBUG_CONTROL_PACKET) \ + control_log_control_packet(pkt); \ + } while (0) + +#define CONFIG_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "CONFIG", fmt, ##arg) +#define CONFIG_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "CONFIG", fmt, ##arg) + +#define CONFIG_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "CONFIG", \ + (vnic_debug & DEBUG_CONFIG_INFO), \ + fmt, ##arg) + +#define DATA_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "DATA", fmt, ##arg) +#define DATA_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "DATA", fmt, ##arg) + +#define DATA_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "DATA", \ + (vnic_debug & DEBUG_DATA_INFO), \ + fmt, ##arg) + +#define DATA_FUNCTION(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "DATA", \ + (vnic_debug & DEBUG_DATA_FUNCTION), \ + fmt, ##arg) + + +#define MCAST_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "MCAST", fmt, ##arg) +#define MCAST_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "MCAST", fmt, ##arg) + +#define MCAST_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "MCAST", \ + (vnic_debug & DEBUG_MCAST_INFO), \ + fmt, ##arg) + +#define MCAST_FUNCTION(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "MCAST", \ + (vnic_debug & DEBUG_MCAST_FUNCTION), \ + fmt, ##arg) + +#define NETPATH_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "NETPATH", fmt, ##arg) +#define NETPATH_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "NETPATH", fmt, ##arg) + +#define NETPATH_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "NETPATH", \ + (vnic_debug & DEBUG_NETPATH_INFO), \ + fmt, ##arg) + +#define VIPORT_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "VIPORT", fmt, ##arg) +#define VIPORT_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "VIPORT", fmt, ##arg) + +#define VIPORT_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "VIPORT", \ + (vnic_debug & DEBUG_VIPORT_INFO), \ + fmt, ##arg) + +#define VIPORT_FUNCTION(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "VIPORT", \ + (vnic_debug & DEBUG_VIPORT_FUNCTION), \ + fmt, ##arg) + +#define LINK_STATE(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "LINK", \ + (vnic_debug & DEBUG_LINK_STATE), \ + fmt, ##arg) + +#define VNIC_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "NIC", fmt, ##arg) +#define VNIC_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "NIC", fmt, ##arg) +#define VNIC_INIT(fmt, arg...) \ + PRINT(KERN_INFO, "NIC", fmt, ##arg) + +#define VNIC_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "NIC", \ + (vnic_debug & DEBUG_VNIC_INFO), \ + fmt, ##arg) + +#define VNIC_FUNCTION(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "NIC", \ + (vnic_debug & DEBUG_VNIC_FUNCTION), \ + fmt, ##arg) + +#define SYS_PRINT(fmt, arg...) \ + PRINT(KERN_INFO, "SYS", fmt, ##arg) +#define SYS_ERROR(fmt, arg...) \ + PRINT(KERN_ERR, "SYS", fmt, ##arg) + +#define SYS_INFO(fmt, arg...) \ + PRINT_CONDITIONAL(KERN_INFO, \ + "SYS", \ + (vnic_debug & DEBUG_SYS_INFO), \ + fmt, ##arg) + +#endif /* VNIC_UTIL_H_INCLUDED */ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:08:18 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:08:18 -0400 Subject: [ofa-general] [PATCH v4 12/14] QLogic VNIC: Driver Kconfig and Makefile. In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210818.11186.51465.stgit@dale> From: Ramachandra K Kconfig and Makefile for the QLogic VNIC driver. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/ulp/qlgc_vnic/Kconfig | 19 +++++++++++++++++++ drivers/infiniband/ulp/qlgc_vnic/Makefile | 13 +++++++++++++ 2 files changed, 32 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/ulp/qlgc_vnic/Kconfig create mode 100644 drivers/infiniband/ulp/qlgc_vnic/Makefile diff --git a/drivers/infiniband/ulp/qlgc_vnic/Kconfig b/drivers/infiniband/ulp/qlgc_vnic/Kconfig new file mode 100644 index 0000000..7b4030e --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/Kconfig @@ -0,0 +1,19 @@ +config INFINIBAND_QLGC_VNIC + tristate "QLogic VNIC - Support for QLogic Ethernet Virtual I/O Controller" + depends on INFINIBAND && NETDEVICES && INET + ---help--- + Support for the QLogic Ethernet Virtual I/O Controller + (EVIC). In conjunction with the EVIC, this provides virtual + ethernet interfaces and transports ethernet packets over + InfiniBand so that you can communicate with Ethernet networks + using your IB device. + +config INFINIBAND_QLGC_VNIC_STATS + bool "QLogic VNIC Statistics" + depends on INFINIBAND_QLGC_VNIC + default n + ---help--- + This option compiles statistics collecting code into the + data path of the QLogic VNIC driver to help in profiling and fine + tuning. This adds some overhead in the interest of gathering + data. diff --git a/drivers/infiniband/ulp/qlgc_vnic/Makefile b/drivers/infiniband/ulp/qlgc_vnic/Makefile new file mode 100644 index 0000000..509dd67 --- /dev/null +++ b/drivers/infiniband/ulp/qlgc_vnic/Makefile @@ -0,0 +1,13 @@ +obj-$(CONFIG_INFINIBAND_QLGC_VNIC) += qlgc_vnic.o + +qlgc_vnic-y := vnic_main.o \ + vnic_ib.o \ + vnic_viport.o \ + vnic_control.o \ + vnic_data.o \ + vnic_netpath.o \ + vnic_config.o \ + vnic_sys.o \ + vnic_multicast.o + +qlgc_vnic-$(CONFIG_INFINIBAND_QLGC_VNIC_STATS) += vnic_stats.o From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:08:48 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:08:48 -0400 Subject: [ofa-general] [PATCH v4 13/14] QLogic VNIC: Modifications to IB Kconfig and Makefile In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210848.11186.47989.stgit@dale> From: Ramachandra K This patch modifies the toplevel Infiniband Kconfig and Makefile to include QLogic VNIC as new ULP. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- drivers/infiniband/Kconfig | 2 ++ drivers/infiniband/Makefile | 1 + 2 files changed, 3 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig index a5dc78a..0775df5 100644 --- a/drivers/infiniband/Kconfig +++ b/drivers/infiniband/Kconfig @@ -53,4 +53,6 @@ source "drivers/infiniband/ulp/srp/Kconfig" source "drivers/infiniband/ulp/iser/Kconfig" +source "drivers/infiniband/ulp/qlgc_vnic/Kconfig" + endif # INFINIBAND diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile index ed35e44..845271e 100644 --- a/drivers/infiniband/Makefile +++ b/drivers/infiniband/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_INFINIBAND_NES) += hw/nes/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ obj-$(CONFIG_INFINIBAND_ISER) += ulp/iser/ +obj-$(CONFIG_INFINIBAND_QLGC_VNIC) += ulp/qlgc_vnic/ From ramachandra.kuchimanchi at qlogic.com Tue Jun 10 14:09:18 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Tue, 10 Jun 2008 17:09:18 -0400 Subject: [ofa-general] [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <20080610210918.11186.64253.stgit@dale> From: Amar Mudrankit This patch adds the file sysfs-class-infiniband-qlgc-vnic which describes the sysfs files managed by QLogic VNIC. Signed-off-by: Ramachandra K Signed-off-by: Poornima Kamath Signed-off-by: Amar Mudrankit --- .../ABI/testing/sysfs-class-infiniband-qlgc-vnic | 179 ++++++++++++++++++++ 1 files changed, 179 insertions(+), 0 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic diff --git a/Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic b/Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic new file mode 100644 index 0000000..6cf22ac --- /dev/null +++ b/Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic @@ -0,0 +1,179 @@ +What: /sys/class/infiniband_qlgc_vnic/ +Date: June 2008 +KernelVersion: 2.6.27 +Contact: Ramachandra K +Description: + +sysfs interface: +---------------- + +The QLogic Virtual NIC module when loaded on host creates following set of +files into the sysfs: + +/sys/class/infiniband_qlgc_vnic/interfaces + + create_vnic (0200) To create a new VNIC interface, write the name + of the interface to this file. + + delete_vnic (0200) To delete an existing VNIC interface, write + name of that VNIC interface to this file. + +/sys/class/infiniband_qlgc_vnic/interfaces// + + vnic_state (0444) State of the VNIC interface. + + current_path (0444) In case of failover configuration, which + connection among the primary and secondary is + currently being used. + + multicast (0644) Get/Set IB multicast value for the VNIC + interface. Only root can Enable/Disable IB + multicast for that VNIC interface. + + rx_csum (0644) Get/Set receive checksum boolean value for VNIC + interface. Only root can Enable/Disable the + receive checksum operation for that VNIC + interface. + + tx_csum (0644) Get/Set transmit checksum boolean value for VNIC + interface. Only root can Enable/Disable the + transmit checksum operation for that VNIC + interface. + +/sys/class/infiniband_qlgc_vnic/interfaces//[primary-secondary]_path + + dgid (0644) Get/set destination GID value for this + connection of VNIC interface. + + hca_info (0444) Get the information about the HCA and IB port + through which this connection has been + configured. + + heartbeat (0644) Get/Set the heartbeat value for this connection + of VNIC interface. + + instance (0644) Get/Set instance number associated with this + connection of VNIC interface. + + ioc_guid (0644) Set/Get GUID value of I/O Controller to which + this connection of VNIC interface should be/is + connecting to. + + ioc_string (0644) Set/Get string value of I/O Controller to which + this connection of VNIC interface should be/is + connecting to. + + link_state (0444) Link state for this connection. + + multicast_state (0444) IB multicast state for this connection. + + pkey (0644) Get/Set pkey value for this connection of VNIC + interface. + + viport_state (0444) Viport state of this connection. + +/sys/class/infiniband_qlgc_vnic/vnic-- + + create_primary (0200) Write the name of the VNIC interface to this + file to create primary connection for VNIC + interface through given HCA and port, once all + parameters required for connection establishment + has been already set into primary_path of the + VNIC interface. + + create_secondary(0200) Write the name of the VNIC interface to this + file to create secondary connection for VNIC + interface through given HCA and port, once all + parameters required for connection establishment + has been already set into secondary_path of the + VNIC interface. + +/sys/class/infiniband_qlgc_vnic/interfaces//stats + + carrier_losses (0444) No of times the carrier loss was + detected for this VNIC interface. + + connection_time (0444) Amount of time this VNIC was in + connected state. + + disconnects (0444) No of times this VNIC interface got + disconnected from EVIC. + + failed_xmits (0444) No of transmits over this VNIC interface + that have failed. + + lifetime (0444) Amount of time elapsed since this VNIC + interface was created. + + multicast_recvs (0444) No of IB multicast packets received over + this VNIC interface. + + recvs (0444) Total no of packets received on this + VNIC interface. + + xmits (0444) Total no of packets that have been + successfully transmitted over this VNIC + interface. + + total_recv_time (0444) Amount of time taken in receive + processing, from the time driver got the + receive completion till the skb was sent + to kernel. + + total_xmit_time (0444) Amount of time taken in transmit + processing, from the time skb was + received from kernel till a RDMA write + was done for it. + + total_disconn_time (0444) Amount of time this VNIC interface was/is + in disconnected state. + + total_carrier_loss_time (0444) Amount of time there was no carrier + present for this VNIC interface. + +Example: +-------- + +QLogic VNIC sysfs interface can be used directly to create a VNIC interface. + +# echo the name of new VNIC interface to create_vnic +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/create_vnic + +# echo parameters to configure the VNIC interface. Some of these are +# interface level parameters which needs to be passed only once per VNIC +# interface. + +echo -n true > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/rx_csum +echo -n false > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/tx_csum +echo -n false > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/multicast + +echo -n 00066a01de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_guid +echo -n fe8000000000000100066a11de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/dgid +echo -n ffff > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/pkey +echo -n 0 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/instance +echo -n 100 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/heartbeat +string="EVIC in Chassis 0x00066a00db000010, Slot 4, Ioc 1" +echo -n $string > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_string + +# Trigger the connection establishment process for the path by choosing required +# HCA and IB port combination. + +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/vnic-mlx4_0-1/create_primary + +# In case of failover configuration, pass the parameters of the failover +# connection to the driver and trigger its connection establishment +# process. + +echo -n 00066a01de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/ioc_guid +echo -n fe8000000000000100066a11de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/dgid +echo -n ffff > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/pkey +echo -n 1 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/instance +echo -n 100 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/heartbeat +string="EVIC in Chassis 0x00066a00db000010, Slot 4, Ioc 1" +echo -n $string > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/ioc_string + +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/vnic-mlx4_0-2/create_secondary + +# An interface can be deleted by echoing the name of the interface to be deleted +# to delete_vnic file +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic From hrosenstock at xsigo.com Tue Jun 10 14:39:34 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 10 Jun 2008 14:39:34 -0700 Subject: [ofa-general] [PATCH] ibsim: Remove some unused routines Message-ID: <1213133974.14047.298.camel@hrosenstock-ws.xsigo.com> ibsim: Remove some unused routines Signed-off-by: Hal Rosenstock diff --git a/umad2sim/sim_client.c b/umad2sim/sim_client.c index 32c45b0..644766e --- a/umad2sim/sim_client.c +++ b/umad2sim/sim_client.c @@ -61,41 +61,6 @@ static unsigned int remote_mode = 0; -int sim_client_recv_packet(struct sim_client *sc, void *buf, int size) -{ - int len, ret, cnt; - - if (size < sizeof(struct sim_request)) { - LOG("buffer too small %d < %zu", size, - sizeof(struct sim_request)); - return -1; - } - - ret = 0; - len = sizeof(struct sim_request); - while (len > 0) { - cnt = read(sc->fd_pktin, buf + ret, len); - if (cnt < 0) { - LOG("sim_client_recv_packet: read(%d) failed (%m)", - size); - return -1; - } else if (cnt == 0) - return ret; - len -= cnt; - ret += cnt; - } - - return size; -} - -int sim_client_send_packet(struct sim_client *sc, char *p, int size) -{ - if (write(sc->fd_pktout, p, size) == size) - return 0; - IBWARN("write failed: %m"); - return -1; -} - static int sim_ctl(struct sim_client *sc, int type, void *data, int len) { struct sim_ctl ctl; diff --git a/umad2sim/sim_client.h b/umad2sim/sim_client.h index ffabcf2..605b305 --- a/umad2sim/sim_client.h +++ b/umad2sim/sim_client.h @@ -46,8 +46,6 @@ struct sim_client { uint16_t pkeys[SIM_CTL_MAX_DATA/sizeof(uint16_t)]; }; -extern int sim_client_send_packet(struct sim_client *sc, char *p, int size); -extern int sim_client_recv_packet(struct sim_client *sc, void *buf, int size); extern int sim_client_set_sm(struct sim_client *sc, unsigned issm); extern int sim_client_init(struct sim_client *sc, char *nodeid); extern void sim_client_exit(struct sim_client *sc); From Jeffrey.C.Becker at nasa.gov Tue Jun 10 15:08:42 2008 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Tue, 10 Jun 2008 15:08:42 -0700 Subject: [ofa-general] opensm routing Message-ID: <484EFB6A.2030907@nasa.gov> Hi all. I was looking into doing some subnet partitioning to separate compute nodes from Lustre nodes, and I saw the following in ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt OpenSM Partition Management --------------------------- Roadmap: Phase 1 - provide partition management at the EndPort (HCA, Router and Switch Port 0) level with no routing affects. Phase 2 - routing engine should take partitions into account. ... Phase 2 functionality: The partition policy should be considered during the routing such that links are associated with particular partition or a set of partitions. Policy should be enhanced to provide hints for how to do that (correlating to QoS too). The exact algorithm is TBD. What is the status of Pkey-aware routing? Thanks. -jeff From shemminger at vyatta.com Tue Jun 10 15:21:47 2008 From: shemminger at vyatta.com (Stephen Hemminger) Date: Tue, 10 Jun 2008 15:21:47 -0700 Subject: [ofa-general] Re: [PATCH v4 04/14] QLogic VNIC: Implementation of Control path of communication protocol In-Reply-To: <20080610210415.11186.95157.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> <20080610210415.11186.95157.stgit@dale> Message-ID: <20080610152147.042b230f@extreme> On Tue, 10 Jun 2008 17:04:15 -0400 Ramachandra K wrote: > +#ifdef CONFIG_INFINIBAND_QLGC_VNIC_STATS > + struct { > + cycles_t request_time; /* intermediate value */ > + cycles_t response_time; > + u32 response_num; > + cycles_t response_max; > + cycles_t response_min; > + u32 timeout_num; > + } statistics; Shouldn't you be using ktime_t rather than cycles_t? And then convert it to some standard time base (ns, ms, or USER_HZ) when reporting to user space. From ducwj at js-star.cn Tue Jun 10 14:55:46 2008 From: ducwj at js-star.cn (humberto boris) Date: Tue, 10 Jun 2008 21:55:46 +0000 Subject: [ofa-general] Brands hand watch at low pricCce Message-ID: <000801c8cb53$01c1a123$825326bf@uqtao> Look great and successful without having to spend a lot. Nothing impresses co-workers and buddy more than a studded Bell & Ross, or a classic Breitling strapped around your wrist. King Replicas houses more than 20+ world famous brands: Dolce & Gabbana Longines Patek Philippe Technomarine and many more! Sale ends this weekend - so visit today for the best off2r. - Visit our site: www.busniie[DOT]com (copy this link and replace "[DOT]" to ".") From yevgenyp at mellanox.co.il Tue Jun 10 22:11:51 2008 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Wed, 11 Jun 2008 08:11:51 +0300 Subject: [ofa-general][PATCH 3/12 v1] mlx4: Qp range reservation In-Reply-To: References: <480F4E10.9080203@mellanox.co.il> Message-ID: <484F5E97.9020401@mellanox.co.il> Roland Dreier wrote: > > +static unsigned long find_aligned_range(unsigned long *bitmap, > > + u32 start, u32 nbits, > > + int len, int align) > > Seems that a lot of this code can be replaced by a call to > bitmap_find_free_region()? > > - R. > bitmap_find_free_region() always searches from the beginning of the bitmap, while we search from the last bit that was set for efficiency reasons. Moving the bitmap pointer to the "last" location while passing it to the function, is not so good because the "last" index is not always aligned to size of long. Thanks, Yevgeny -- Yevgeny Petrilin Mellanox Technalogies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext.7677) Mobile: +972 (54) 783 9222 E-mail: yevgenyp at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From ogerlitz at voltaire.com Tue Jun 10 23:36:23 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 11 Jun 2008 09:36:23 +0300 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: References: Message-ID: <484F7267.6050405@voltaire.com> Roland Dreier wrote: > @@ -486,7 +500,11 @@ enum ibv_wr_opcode { > + IBV_WR_LSO, LSO is not exposed to user space by this patch, so maybe better to just reserve/skip this enum value? > --- a/src/cmd.c > +++ b/src/cmd.c > @@ -159,6 +159,18 @@ int ibv_cmd_query_device(struct ibv_context *context, > + /* > + * If the kernel driver says that it supports memory > + * management extensions, then move the flag to > + * IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS so that the low-level > + * driver needs to move the flag back to show it supports the > + * operations as well. > + */ > + if (device_attr->device_cap_flags & IBV_DEVICE_MEM_MGT_EXTENSIONS) { > + device_attr->device_cap_flags &= ~IBV_DEVICE_MEM_MGT_EXTENSIONS; > + device_attr->device_cap_flags |= IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS; > + } this sets IBV_DEVICE_MEM_MGT_EXTENSIONS off and the IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS on, correct? > --- a/src/verbs.c > +++ b/src/verbs.c > @@ -79,7 +79,13 @@ enum ibv_rate mult_to_ibv_rate(int mult) > int __ibv_query_device(struct ibv_context *context, > struct ibv_device_attr *device_attr) > { > - return context->ops.query_device(context, device_attr); > + int ret; > + > + ret = context->ops.query_device(context, device_attr); > + > + device_attr->device_cap_flags &= ~IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS; and this sets IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS off, so at the bottom line the user will never see IBV_DEVICE_MEM_MGT_EXTENSIONS being set to ON? or the user space hw library should somwhow add this bit? Or. From kaber at trash.net Tue Jun 10 23:47:59 2008 From: kaber at trash.net (Patrick McHardy) Date: Wed, 11 Jun 2008 08:47:59 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <20080610210918.11186.64253.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> Message-ID: <484F751F.7030407@trash.net> Since I didn't receive anything but a "its too complex" in response to my previous question, let me ask again: why are you inventing a new sysfs interface (combined with ~15 module parameters) instead of using the existing standard interfaces for this? Ramachandra K wrote: > +sysfs interface: > +---------------- > + > +The QLogic Virtual NIC module when loaded on host creates following set of > +files into the sysfs: > + > +/sys/class/infiniband_qlgc_vnic/interfaces > + > + create_vnic (0200) To create a new VNIC interface, write the name > + of the interface to this file. > + > + delete_vnic (0200) To delete an existing VNIC interface, write > + name of that VNIC interface to this file. This is exactly what rtnl_link is for. > +/sys/class/infiniband_qlgc_vnic/interfaces// > + > + vnic_state (0444) State of the VNIC interface. This I don't understand - you seem to be registering and unregistering the net_device based some state machine, in some cases even triggered by timers. Whats the idea behind this? > + > + current_path (0444) In case of failover configuration, which > + connection among the primary and secondary is > + currently being used. > + > + multicast (0644) Get/Set IB multicast value for the VNIC > + interface. Only root can Enable/Disable IB > + multicast for that VNIC interface. > + > + rx_csum (0644) Get/Set receive checksum boolean value for VNIC > + interface. Only root can Enable/Disable the > + receive checksum operation for that VNIC > + interface. > + > + tx_csum (0644) Get/Set transmit checksum boolean value for VNIC > + interface. Only root can Enable/Disable the > + transmit checksum operation for that VNIC > + interface. Why is this not done using ethtool? > +Example: > +-------- > + > +QLogic VNIC sysfs interface can be used directly to create a VNIC interface. > + > +# echo the name of new VNIC interface to create_vnic > +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/create_vnic > + > +# echo parameters to configure the VNIC interface. Some of these are > +# interface level parameters which needs to be passed only once per VNIC > +# interface. > + > +echo -n true > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/rx_csum > +echo -n false > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/tx_csum > +echo -n false > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/multicast > + > +echo -n 00066a01de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_guid > +echo -n fe8000000000000100066a11de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/dgid > +echo -n ffff > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/pkey > +echo -n 0 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/instance > +echo -n 100 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/heartbeat > +string="EVIC in Chassis 0x00066a00db000010, Slot 4, Ioc 1" > +echo -n $string > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_string > + > +# Trigger the connection establishment process for the path by choosing required > +# HCA and IB port combination. > + > +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/vnic-mlx4_0-1/create_primary > + > +# In case of failover configuration, pass the parameters of the failover > +# connection to the driver and trigger its connection establishment > +# process. > + > +echo -n 00066a01de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/ioc_guid > +echo -n fe8000000000000100066a11de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/dgid > +echo -n ffff > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/pkey > +echo -n 1 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/instance > +echo -n 100 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/heartbeat > +string="EVIC in Chassis 0x00066a00db000010, Slot 4, Ioc 1" > +echo -n $string > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/secondary_path/ioc_string > + > +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/vnic-mlx4_0-2/create_secondary > + > +# An interface can be deleted by echoing the name of the interface to be deleted > +# to delete_vnic file > +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic This clearly shows the advantage of being able to use echo for configuration. From ogerlitz at voltaire.com Wed Jun 11 00:05:57 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 11 Jun 2008 10:05:57 +0300 Subject: [ofa-general] missed cq event In-Reply-To: <20080610195825.GA22901@osc.edu> References: <20080610182655.GE22762@osc.edu> <20080610195825.GA22901@osc.edu> Message-ID: <484F7955.2060602@voltaire.com> Pete Wyckoff wrote: > You're certainly right that the need to reap even unsignaled sends > is needed for some apps. I'm just looking for a shortcut if my app > does not ever need to find out about send completions. Just to > simplify the code, not expecting anything will go faster or require > less memory. One way we could go is enhance the verbs API to let an app tell the driver to post a NOP work request into the send queue once every n posts, it has some devil in the details but I don't think it impossible or too ugly... Anyway, looking in the IB spec (eg section 11.4.1.1 POST SEND REQUEST table Table 93 Operation Type Matrix) I don't see an evidence that such opcode exists... the mlx4 driver defines MLX4_NOP_OPCODE as a possible value to a send WR, so I assume this specific HW supports it. Does anyone know if NOP is supported by IB and/or iWARP? Or. From thomas.eggestad at gmail.com Wed Jun 11 00:54:19 2008 From: thomas.eggestad at gmail.com (Thomas Eggestad) Date: Wed, 11 Jun 2008 09:54:19 +0200 Subject: [ofa-general] unsing ibverbs 1.0 with rdmacm In-Reply-To: References: Message-ID: On 6/10/08, Roland Dreier wrote: > Maybe you could come up with some hack that manually converts back and > forth between the two versions of various structures, but as I said > before, I think it would be much simpler to just link against and use > the 1.1 ABI if you want to use librdmacm. This raises a question to this scenario: 1) Commercial (binary) applications needs to support many versions of OFED (ibverbs and rdmacm). 2) ABI ibverbs 1.2 is released. Question: What ibverbs ABI context will rdmacm ABI RDMACM_1.0 give access to? As an application developer I would very much like that it would always give ibverbs 1.1. And for RDMACM_1.1 I would like to have the ability to specify what version of ibverbs I want my context in. I know it is not interesting to work with compatibility with old stuff. But it is very useful for us. Regards, Thomas From monis at Voltaire.COM Wed Jun 11 01:17:36 2008 From: monis at Voltaire.COM (Moni Shoua) Date: Wed, 11 Jun 2008 11:17:36 +0300 Subject: [ofa-general] [PATCH] IB/ipoib: refresh paths instead of fushing them on SM change event Message-ID: <484F8A20.9070101@Voltaire.COM> The patch tries to solve the problem of device going down and paths being flushed on SM change event. The method is to mark the paths as candidates for refresh (with a valid flag) and wait for ARP probe to start a flow of path lookup that leads to patch query which ends up in path (and address handle) refresh. The solution requires a different and less intrusive handling of SM change event. For that, the second argument of the flush function changed it's meaning from boolean flag to a level. In most cases, SM failover doesn't cause LID change so traffic won't stop. In the rare cases of LID change, the remote host (the one that hadn't changed its LID) will lose connectivity until paths are refreshed. This is no worse than the current state. In fact, preventing the device from going down saves packets that otherwise would be lost. Signed-off-by: Moni Levy Signed-off-by: Moni Shoua --- drivers/infiniband/ulp/ipoib/ipoib.h | 11 ++++-- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 37 +++++++++++++------- drivers/infiniband/ulp/ipoib/ipoib_main.c | 52 ++++++++++++++++++++++++++--- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 19 +++++----- 4 files changed, 89 insertions(+), 30 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index ca126fc..05ad81e 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -276,10 +276,11 @@ struct ipoib_dev_priv { struct delayed_work pkey_poll_task; struct delayed_work mcast_task; - struct work_struct flush_task; + struct work_struct flush_light; + struct work_struct flush_normal; + struct work_struct flush_heavy; struct work_struct restart_task; struct delayed_work ah_reap_task; - struct work_struct pkey_event_task; struct ib_device *ca; u8 port; @@ -358,6 +359,7 @@ struct ipoib_path { struct completion done; struct rb_node rb_node; + int valid; struct list_head list; }; @@ -423,11 +425,14 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_ah *address, u32 qpn); void ipoib_reap_ah(struct work_struct *work); +void ipoib_refresh_paths(struct net_device *dev); void ipoib_flush_paths(struct net_device *dev); struct ipoib_dev_priv *ipoib_intf_alloc(const char *format); int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port); -void ipoib_ib_dev_flush(struct work_struct *work); +void ipoib_ib_dev_flush_light(struct work_struct *work); +void ipoib_ib_dev_flush_normal(struct work_struct *work); +void ipoib_ib_dev_flush_heavy(struct work_struct *work); void ipoib_pkey_event(struct work_struct *work); void ipoib_ib_dev_cleanup(struct net_device *dev); diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index f429bce..1d9f158 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -898,7 +898,7 @@ int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port) return 0; } -static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) +static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int level) { struct ipoib_dev_priv *cpriv; struct net_device *dev = priv->dev; @@ -911,7 +911,7 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) * the parent is down. */ list_for_each_entry(cpriv, &priv->child_intfs, list) - __ipoib_ib_dev_flush(cpriv, pkey_event); + __ipoib_ib_dev_flush(cpriv, level); mutex_unlock(&priv->vlan_mutex); @@ -925,7 +925,7 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) return; } - if (pkey_event) { + if (level == 2) { if (ib_find_pkey(priv->ca, priv->port, priv->pkey, &new_index)) { clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags); ipoib_ib_dev_down(dev, 0); @@ -943,11 +943,15 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) priv->pkey_index = new_index; } - ipoib_dbg(priv, "flushing\n"); + if (level == 0) { + ipoib_refresh_paths(dev); + ipoib_mcast_dev_flush(dev); + } - ipoib_ib_dev_down(dev, 0); + if (level >= 1) + ipoib_ib_dev_down(dev, 0); - if (pkey_event) { + if (level >= 2) { ipoib_ib_dev_stop(dev, 0); ipoib_ib_dev_open(dev); } @@ -957,29 +961,36 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) * we get here, don't bring it back up if it's not configured up */ if (test_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags)) { - ipoib_ib_dev_up(dev); + if (level >= 1) + ipoib_ib_dev_up(dev); ipoib_mcast_restart_task(&priv->restart_task); } } -void ipoib_ib_dev_flush(struct work_struct *work) +void ipoib_ib_dev_flush_light(struct work_struct *work) { struct ipoib_dev_priv *priv = - container_of(work, struct ipoib_dev_priv, flush_task); + container_of(work, struct ipoib_dev_priv, flush_light); - ipoib_dbg(priv, "Flushing %s\n", priv->dev->name); __ipoib_ib_dev_flush(priv, 0); } -void ipoib_pkey_event(struct work_struct *work) +void ipoib_ib_dev_flush_normal(struct work_struct *work) { struct ipoib_dev_priv *priv = - container_of(work, struct ipoib_dev_priv, pkey_event_task); + container_of(work, struct ipoib_dev_priv, flush_normal); - ipoib_dbg(priv, "Flushing %s and restarting its QP\n", priv->dev->name); __ipoib_ib_dev_flush(priv, 1); } +void ipoib_ib_dev_flush_heavy(struct work_struct *work) +{ + struct ipoib_dev_priv *priv = + container_of(work, struct ipoib_dev_priv, flush_heavy); + + __ipoib_ib_dev_flush(priv, 2); +} + void ipoib_ib_dev_cleanup(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index 2442090..6682e9c 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -259,6 +259,13 @@ static int __path_add(struct net_device *dev, struct ipoib_path *path) return 0; } +static void path_refresh(struct net_device *dev, struct ipoib_path *path) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + ipoib_dbg(priv, "mark path LID 0x%04x GID " IPOIB_GID_FMT " invalid\n", + be16_to_cpu(path->pathrec.dlid), IPOIB_GID_ARG(path->pathrec.dgid)); + path->valid = 0; +} static void path_free(struct net_device *dev, struct ipoib_path *path) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -350,6 +357,27 @@ void ipoib_path_iter_read(struct ipoib_path_iter *iter, #endif /* CONFIG_INFINIBAND_IPOIB_DEBUG */ +void ipoib_refresh_paths(struct net_device *dev) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + struct ipoib_path *path, *tp; + spin_lock_irq(&priv->tx_lock); + spin_lock(&priv->lock); + + list_for_each_entry_safe(path, tp, &priv->path_list, list) { + if (path->query) + ib_sa_cancel_query(path->query_id, path->query); + spin_unlock(&priv->lock); + spin_unlock_irq(&priv->tx_lock); + wait_for_completion(&path->done); + path_refresh(dev, path); + spin_lock_irq(&priv->tx_lock); + spin_lock(&priv->lock); + } + spin_unlock(&priv->lock); + spin_unlock_irq(&priv->tx_lock); +} + void ipoib_flush_paths(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -386,6 +414,7 @@ static void path_rec_completion(int status, struct net_device *dev = path->dev; struct ipoib_dev_priv *priv = netdev_priv(dev); struct ipoib_ah *ah = NULL; + struct ipoib_ah *old_ah = NULL; struct ipoib_neigh *neigh, *tn; struct sk_buff_head skqueue; struct sk_buff *skb; @@ -409,6 +438,9 @@ static void path_rec_completion(int status, spin_lock_irqsave(&priv->lock, flags); + if (path->ah) + ipoib_put_ah(path->ah); + path->ah = ah; if (ah) { @@ -421,6 +453,8 @@ static void path_rec_completion(int status, __skb_queue_tail(&skqueue, skb); list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) { + if (neigh->ah) + old_ah = neigh->ah; kref_get(&path->ah->ref); neigh->ah = path->ah; memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, @@ -443,13 +477,15 @@ static void path_rec_completion(int status, while ((skb = __skb_dequeue(&neigh->queue))) __skb_queue_tail(&skqueue, skb); } + path->valid = 1; } path->query = NULL; complete(&path->done); spin_unlock_irqrestore(&priv->lock, flags); - + if (old_ah) + ipoib_put_ah(old_ah); while ((skb = __skb_dequeue(&skqueue))) { skb->dev = dev; if (dev_queue_xmit(skb)) @@ -471,6 +507,7 @@ static struct ipoib_path *path_rec_create(struct net_device *dev, void *gid) return NULL; path->dev = dev; + path->valid = 0; skb_queue_head_init(&path->queue); @@ -623,8 +660,12 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev, spin_lock(&priv->lock); path = __path_find(dev, phdr->hwaddr + 4); - if (!path) { - path = path_rec_create(dev, phdr->hwaddr + 4); + if (!path || !path->valid) { + if (!path) { + ipoib_dbg(priv,"path does not exist\n"); + path = path_rec_create(dev, phdr->hwaddr + 4); + } else + ipoib_dbg(priv,"path exists but invalid\n"); if (path) { /* put pseudoheader back on for next time */ skb_push(skb, sizeof *phdr); @@ -989,9 +1030,10 @@ static void ipoib_setup(struct net_device *dev) INIT_LIST_HEAD(&priv->multicast_list); INIT_DELAYED_WORK(&priv->pkey_poll_task, ipoib_pkey_poll); - INIT_WORK(&priv->pkey_event_task, ipoib_pkey_event); INIT_DELAYED_WORK(&priv->mcast_task, ipoib_mcast_join_task); - INIT_WORK(&priv->flush_task, ipoib_ib_dev_flush); + INIT_WORK(&priv->flush_light, ipoib_ib_dev_flush_light); + INIT_WORK(&priv->flush_normal, ipoib_ib_dev_flush_normal); + INIT_WORK(&priv->flush_heavy, ipoib_ib_dev_flush_heavy); INIT_WORK(&priv->restart_task, ipoib_mcast_restart_task); INIT_DELAYED_WORK(&priv->ah_reap_task, ipoib_reap_ah); } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c index 8766d29..4dd2a41 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c @@ -289,15 +289,16 @@ void ipoib_event(struct ib_event_handler *handler, if (record->element.port_num != priv->port) return; - if (record->event == IB_EVENT_PORT_ERR || - record->event == IB_EVENT_PORT_ACTIVE || - record->event == IB_EVENT_LID_CHANGE || - record->event == IB_EVENT_SM_CHANGE || - record->event == IB_EVENT_CLIENT_REREGISTER) { - ipoib_dbg(priv, "Port state change event\n"); - queue_work(ipoib_workqueue, &priv->flush_task); + ipoib_dbg(priv, "Event %d on device %s port %d\n",record->event, + record->device->name, record->element.port_num); + if ( record->event == IB_EVENT_SM_CHANGE || + record->event == IB_EVENT_CLIENT_REREGISTER) { + queue_work(ipoib_workqueue, &priv->flush_light); + } else if (record->event == IB_EVENT_PORT_ERR || + record->event == IB_EVENT_PORT_ACTIVE || + record->event == IB_EVENT_LID_CHANGE) { + queue_work(ipoib_workqueue, &priv->flush_normal); } else if (record->event == IB_EVENT_PKEY_CHANGE) { - ipoib_dbg(priv, "P_Key change event on port:%d\n", priv->port); - queue_work(ipoib_workqueue, &priv->pkey_event_task); + queue_work(ipoib_workqueue, &priv->flush_heavy); } } From jackm at dev.mellanox.co.il Wed Jun 11 01:37:22 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Wed, 11 Jun 2008 11:37:22 +0300 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: References: Message-ID: <200806111137.22966.jackm@dev.mellanox.co.il> On Tuesday 10 June 2008 00:52, Roland Dreier wrote: > Here's the complete (untested) libibverbs patch I am working on now... I > think I got everything needed... It looks OK to me. I'm not crazy about using an extra capability flag (IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS), but I also don't see any way around this if we are to preserve backwards binary compatibility. For the same reason, I don't see any alternative regarding the unnamed union -- I actually think that the unnamed union works well here. - Jack > diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h > index 0db083a..b503a28 100644 > --- a/include/infiniband/kern-abi.h > +++ b/include/infiniband/kern-abi.h > @@ -306,7 +306,10 @@ struct ibv_kern_wc { > __u32 opcode; > __u32 vendor_err; > __u32 byte_len; > - __u32 imm_data; > + union { > + __u32 imm_data; > + __u32 invalidate_rkey; > + }; > __u32 qp_num; > __u32 src_qp; > __u32 wc_flags; > @@ -572,7 +575,10 @@ struct ibv_kern_send_wr { > __u32 num_sge; > __u32 opcode; > __u32 send_flags; > - __u32 imm_data; > + union { > + __u32 imm_data; > + __u32 invalidate_rkey; > + }; > union { > struct { > __u64 remote_addr; > diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h > index a04cc62..e6e2b10 100644 > --- a/include/infiniband/verbs.h > +++ b/include/infiniband/verbs.h > @@ -92,7 +92,17 @@ enum ibv_device_cap_flags { > IBV_DEVICE_SYS_IMAGE_GUID = 1 << 11, > IBV_DEVICE_RC_RNR_NAK_GEN = 1 << 12, > IBV_DEVICE_SRQ_RESIZE = 1 << 13, > - IBV_DEVICE_N_NOTIFY_CQ = 1 << 14 > + IBV_DEVICE_N_NOTIFY_CQ = 1 << 14, > + /* > + * IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS is used by libibverbs to > + * signal to low-level driver libraries that the kernel set > + * the "send with invalidate" capaibility bit. Applications > + * should only test IBV_DEVICE_MEM_MGT_EXTENSIONS and never > + * look at IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS. > + */ > + IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS = 1 << 16, > + IBV_DEVICE_MEM_WINDOW = 1 << 17, > + IBV_DEVICE_MEM_MGT_EXTENSIONS = 1 << 21, > }; > > enum ibv_atomic_cap { > @@ -257,7 +267,8 @@ enum ibv_wc_opcode { > > enum ibv_wc_flags { > IBV_WC_GRH = 1 << 0, > - IBV_WC_WITH_IMM = 1 << 1 > + IBV_WC_WITH_IMM = 1 << 1, > + IBV_WC_WITH_INVALIDATE = 1 << 2, > }; > > struct ibv_wc { > @@ -266,7 +277,10 @@ struct ibv_wc { > enum ibv_wc_opcode opcode; > uint32_t vendor_err; > uint32_t byte_len; > - uint32_t imm_data; /* in network byte order */ > + union { > + uint32_t imm_data; /* in network byte order */ > + uint32_t invalidate_rkey; > + }; > uint32_t qp_num; > uint32_t src_qp; > enum ibv_wc_flags wc_flags; > @@ -486,7 +500,11 @@ enum ibv_wr_opcode { > IBV_WR_SEND_WITH_IMM, > IBV_WR_RDMA_READ, > IBV_WR_ATOMIC_CMP_AND_SWP, > - IBV_WR_ATOMIC_FETCH_AND_ADD > + IBV_WR_ATOMIC_FETCH_AND_ADD, > + IBV_WR_LSO, > + IBV_WR_SEND_WITH_INV, > + IBV_WR_RDMA_READ_WITH_INV, > + IBV_WR_LOCAL_INV, > }; > > enum ibv_send_flags { > @@ -509,7 +527,10 @@ struct ibv_send_wr { > int num_sge; > enum ibv_wr_opcode opcode; > enum ibv_send_flags send_flags; > - uint32_t imm_data; /* in network byte order */ > + union { > + uint32_t imm_data; /* in network byte order */ > + uint32_t invalidate_rkey; > + }; > union { > struct { > uint64_t remote_addr; > diff --git a/src/cmd.c b/src/cmd.c > index 66d7134..1945143 100644 > --- a/src/cmd.c > +++ b/src/cmd.c > @@ -159,6 +159,18 @@ int ibv_cmd_query_device(struct ibv_context *context, > device_attr->local_ca_ack_delay = resp.local_ca_ack_delay; > device_attr->phys_port_cnt = resp.phys_port_cnt; > > + /* > + * If the kernel driver says that it supports memory > + * management extensions, then move the flag to > + * IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS so that the low-level > + * driver needs to move the flag back to show it supports the > + * operations as well. > + */ > + if (device_attr->device_cap_flags & IBV_DEVICE_MEM_MGT_EXTENSIONS) { > + device_attr->device_cap_flags &= ~IBV_DEVICE_MEM_MGT_EXTENSIONS; > + device_attr->device_cap_flags |= IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS; > + } > + > return 0; > } > > diff --git a/src/compat-1_0.c b/src/compat-1_0.c > index 459ade9..0df8b68 100644 > --- a/src/compat-1_0.c > +++ b/src/compat-1_0.c > @@ -535,7 +535,18 @@ symver(__ibv_ack_async_event_1_0, ibv_ack_async_event, IBVERBS_1.0); > int __ibv_query_device_1_0(struct ibv_context_1_0 *context, > struct ibv_device_attr *device_attr) > { > - return ibv_query_device(context->real_context, device_attr); > + int ret; > + > + ret = ibv_query_device(context->real_context, device_attr); > + > + /* > + * ABI 1.0 consumers are never expecting memory management > + * extension support. > + */ > + device_attr->device_cap_flags &= ~(IBV_DEVICE_MEM_MGT_EXTENSIONS | > + IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS); > + > + return ret; > } > symver(__ibv_query_device_1_0, ibv_query_device, IBVERBS_1.0); > > diff --git a/src/verbs.c b/src/verbs.c > index 9e370ce..4ea342f 100644 > --- a/src/verbs.c > +++ b/src/verbs.c > @@ -79,7 +79,13 @@ enum ibv_rate mult_to_ibv_rate(int mult) > int __ibv_query_device(struct ibv_context *context, > struct ibv_device_attr *device_attr) > { > - return context->ops.query_device(context, device_attr); > + int ret; > + > + ret = context->ops.query_device(context, device_attr); > + > + device_attr->device_cap_flags &= ~IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS; > + > + return ret; > } > default_symver(__ibv_query_device, ibv_query_device); > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sdenham at us.ibm.com Wed Jun 11 03:57:03 2008 From: sdenham at us.ibm.com (Scott Denham) Date: Wed, 11 Jun 2008 04:57:03 -0600 Subject: [ofa-general] AUTO: AUTO: Out of office, business travel (returning 06/14/2008) Message-ID: I am out of the office until 06/14/2008. >From June 8-13 I will be away from the office at the EAGE annual meeting. I have very irregular email connectivity and no phone access. I will respond to your message as soon practical. Note: This is an automated response to your message "general Digest, Vol 17, Issue 34" sent on 6/9/08 8:48:13. This is the only notification you will receive while this person is away. -------------- next part -------------- An HTML attachment was scrubbed... URL: From upshall-george at nf.sympatico.ca Wed Jun 11 04:06:04 2008 From: upshall-george at nf.sympatico.ca (upshall-george at nf.sympatico.ca) Date: Wed, 11 Jun 2008 7:06:04 -0400 Subject: [ofa-general] File You Claim Message-ID: <6u5k2v$1gf7ok@toip38-bus.srvr.bell.ca> You won £1,000,000 GBP.Get back to us via return email with your Name,and Address\Country and your Phone Number for more information on how you won and the delivery of your won prize Email:mrcollinspowell at gmail.com From vlad at lists.openfabrics.org Wed Jun 11 05:13:01 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 11 Jun 2008 05:13:01 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080611-0456 daily build status Message-ID: <20080611121301.CD9EDE60F5A@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-user_mad-mod --with-user_access-mod --with-mthca-mod --with-mlx4-mod --with-core-mod --with-addr_trans-mod --with-rds-mod --with-cxgb3-mod --with-nes-mod Passed: Failed: Build failed on i686 with 2.6.15-23-server Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check/drivers/infiniband/core/addr.c:378: error: implicit declaration of function 'register_netevent_notifier' /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check/drivers/infiniband/core/addr.c: In function 'addr_cleanup': /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check/drivers/infiniband/core/addr.c:384: error: implicit declaration of function 'unregister_netevent_notifier' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_check] Error 2 make[1]: Leaving directory `/usr/src/linux-headers-2.6.15-23-server' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 1 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 2 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: error: too many arguments to function 'ip_route_output_key' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080611-0456_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From tnuvniqipwex at bloomers.ebay.sun.com Wed Jun 11 05:31:46 2008 From: tnuvniqipwex at bloomers.ebay.sun.com (Simon Contreras) Date: Wed, 11 Jun 2008 13:31:46 +0100 Subject: [ofa-general] Luxury Message-ID: <01c8cbc7$7e6c0d00$c97ba85b@tnuvniqipwex> We are now bringing forward the replicas of the watches, Accessories, boxsets, Cufflinks, Keychains, Lighters, Pens, Tiffany & Co Jewelry. A replica presupposes the exact copy of the original, that is the quality and all the otherfeatures the original possesses. For instance,you might get a $18000 Rolex replica for a $229.00 price only! Our replica is of such an amazing quality that it can be destinguished from the originalonly in the jewellery workshop.You're anxiously wanting to be glamorous enough for demanding high soiciety standarts, but your earnings are not that sky high? You are willing to reward your lover with an outrageous gift but you don't want that to empty your wallet? Come, visit our shop!P.S When applying for a well-paid job, when flirting with a girl, whenasking for a credit in the bank you get your chances doubled with A Rolex President replica on your hand!PSS: The only missing thing here is a luxury suit!Go check the site -------------- next part -------------- An HTML attachment was scrubbed... URL: From moshek at voltaire.com Wed Jun 11 05:52:38 2008 From: moshek at voltaire.com (Moshe Kazir) Date: Wed, 11 Jun 2008 15:52:38 +0300 Subject: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp Message-ID: <39C75744D164D948A170E9792AF8E7CAC5AFD1@exil.voltaire.com> Moshe ____________________________________________________________ Moshe Katzir | +972-9971-8639 (o) | +972-52-860-6042 (m) Voltaire - The Grid Backbone www.voltaire.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ofa_kernel-1.3.1-2.6.16-54-0.2.5_lustre.1.6.4.3-smp.backport.patch Type: application/octet-stream Size: 739 bytes Desc: ofa_kernel-1.3.1-2.6.16-54-0.2.5_lustre.1.6.4.3-smp.backport.patch URL: From hrosenstock at xsigo.com Wed Jun 11 06:59:22 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 06:59:22 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags/src/saquery.c: fix potential core dump (Was: Re: infiniband-diags/saquery.c:print_multicast_member_record question) In-Reply-To: <20080610131803.34c7a889.weiny2@llnl.gov> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> <20080610112617.3563f3ed.weiny2@llnl.gov> <1213122919.14047.280.camel@hrosenstock-ws.xsigo.com> <20080610131803.34c7a889.weiny2@llnl.gov> Message-ID: <1213192762.14047.325.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-10 at 13:18 -0700, Ira Weiny wrote: > On Tue, 10 Jun 2008 11:35:19 -0700 > Hal Rosenstock wrote: > > > On Tue, 2008-06-10 at 11:26 -0700, Ira Weiny wrote: > > > On Tue, 10 Jun 2008 11:16:24 -0700 > > > Hal Rosenstock wrote: > > > > > > > > > > > Looks to me like it relies on some node GUID being same as port GUID. > > > > While that's allowable for one port, it won't always be the case. > > > > > > In my test system the Node GUID and port GUID are different and this works. I > > > am specifically using the port guid out of the NodeInfo struct of the > > > NodeRecord. So I should be using the port guid vs port GID ID _only_. > > > > Not sure what you mean; port GID 0 = subnet prefix + port GUID. Not sure > > what SMs support other port GIDs than this but OpenSM doesn't. > > I mean the port gid == subnet prefix + port GUID. Therefore if you mask off > the subnet prefix (ie only use the lower 64bits of the gid) you should end up > with the GUID, right? Yes. > > > > > > > Have you found some configuration in which this does not > > > > > work? > > > > > > > > Yes. > > > > > > :-( Sorry, I'm pretty sure this should be ok... > > > > It's possible it's an environment thing but there's a couple of > > suboptimialities in print_multicast_member_record: > > That is probably true. I see you have already sent a patch to not do the node > record queries unless necessary. > > > > > 1. Does osmv_get_query_node_rec always return a non NULL pointer ? > > :-/ Yea that is a bug. Yes using node_record would be bad if for some reason > the node_record did not exists. However, the port should not be a member of > the mcast group if it does not exist in the SA. So... you should not have a > mcast member record. > > I think there is a race condition here that I did not check, that is bad. > > 2. If the loop fails to find a match, the last node description is used. > > Yep, that is a bug as well. I guess this masks the above situation. Would you > agree this is a corner case? Unless your fabric is changing quickly there > should be a node record for each member record right? > > Patch below. Looks better to me. > Should there be a warning printed in the case the node_record is not found? NodeDescription seems sufficient to me. -- Hal > Ira > > > From ac10f52fd65b0b34b409ca2aa266a7363ae32e2c Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Tue, 10 Jun 2008 13:07:43 -0700 > Subject: [PATCH] infiniband-diags/src/saquery.c: fix potential core dump and/or incorrect node > descriptions from being printed if a corresponding node_record is not found for > a multicast member record. > > Signed-off-by: Ira K. Weiny > --- > infiniband-diags/src/saquery.c | 17 +++++++++++------ > 1 files changed, 11 insertions(+), 6 deletions(-) > > diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c > index e727940..89754f6 100644 > --- a/infiniband-diags/src/saquery.c > +++ b/infiniband-diags/src/saquery.c > @@ -354,14 +354,19 @@ print_multicast_member_record(ib_member_rec_t *p_mcmr) > uint64_t gid_prefix = cl_ntoh64( p_mcmr->port_gid.unicast.prefix ); > uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); > uint16_t mlid = cl_ntoh16( p_mcmr->mlid ); > - ib_node_record_t *node_record = NULL; > int i = 0; > + char *node_name = ""; > > - /* go through and find the node description for this node GID */ > + /* go through the node records searching for a port guid which matches > + * this port gid interface id. > + * This gives us a node name to print, if available. > + */ > for (i = 0; i < result.result_cnt; i++) { > - node_record = osmv_get_query_node_rec(result.p_result_madw, i); > - if (cl_ntoh64(node_record->node_info.port_guid) == gid_interface_id) > + ib_node_record_t *nr = osmv_get_query_node_rec(result.p_result_madw, i); > + if (cl_ntoh64(nr->node_info.port_guid) == gid_interface_id) { > + node_name = clean_nodedesc((char *)nr->node_desc.description); > break; > + } > } > > if (requested_name) { > @@ -370,7 +375,7 @@ print_multicast_member_record(ib_member_rec_t *p_mcmr) > "0x%016" PRIx64 " (%s)\n", > gid_prefix, > gid_interface_id, > - clean_nodedesc((char *)node_record->node_desc.description) > + node_name > ); > } > } else { > @@ -391,7 +396,7 @@ print_multicast_member_record(ib_member_rec_t *p_mcmr) > gid_interface_id, > p_mcmr->scope_state, > p_mcmr->proxy_join, > - clean_nodedesc((char *)node_record->node_desc.description) > + node_name > ); > } > } From hrosenstock at xsigo.com Wed Jun 11 06:53:30 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 06:53:30 -0700 Subject: [ofa-general] [PATCH] ibsim: Support short RMPP packets Message-ID: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> ibsim: Support for short RMPP packets (up to 256 bytes total) If this is acceptable, I'll follow this up with long packet support. Signed-off-by: Hal Rosenstock diff --git a/ibsim/ibsim.c b/ibsim/ibsim.c index e4ab6bc..bdfe252 100644 --- a/ibsim/ibsim.c +++ b/ibsim/ibsim.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -494,7 +495,7 @@ static int sim_read_pkt(int fd, int client) VERB("%s %d bytes (%zu) to client %d fd %d", dcl == cl ? "replying" : "forwarding", - size, sizeof(struct sim_request), dcl->id, dcl->fd); + size, sizeof(struct sim_req256), dcl->id, dcl->fd); // reply ret = write(dcl->fd, buf, size); @@ -508,7 +509,7 @@ static int sim_read_pkt(int fd, int client) } IBWARN("write failed: %m - pkt dropped"); if (dcl != cl) { /* reply timeout */ - struct sim_request *r = (struct sim_request *)buf; + struct sim_req256 *r = (struct sim_req256 *)buf; r->status = htonl(110); write(cl->fd, buf, size); } diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 675d95b..eef0bab --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -1128,14 +1129,14 @@ Smpfn *get_handle_fn(ib_rpc_t rpc, int response) return fn; } - return 0; // No MGTCLASS matched . + return 0; // No MGTCLASS matched. } int process_packet(Client * cl, void *p, int size, Client ** dcl) { struct sim_request *r = p; Port *port; - uint8_t data[256]; + uint8_t data[MAD_BLOCK_SIZE]; int status, tlid, tqp; int response; Smpfn *fn; @@ -1144,9 +1145,10 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) *dcl = cl; - DEBUG("client %d, size %d", cl->id, size); - if (size != sizeof(*r)) { - IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); + DEBUG("client %d size %d", cl->id, size); + if (size > sizeof(*r) + MAD_BLOCK_SIZE) { + IBWARN("unsupported packet size %d (> %zu)", size, + sizeof(*r) + MAD_BLOCK_SIZE); return -1; } @@ -1183,7 +1185,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) VERB("forward pkt to client %d pid %d attr %d", (*dcl)->id, (*dcl)->pid, rpc.attr.id); forward_MAD(r->mad, &rpc, &path); - return sizeof(*r); // forward only + return size; // forward only } if (port->errrate && (random() % 100) < port->errrate) { @@ -1214,12 +1216,12 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) VERB("PKT roll back did not succeed"); goto _dropped; } - return sizeof(*r); + return sizeof(*r) + MAD_BLOCK_SIZE; _dropped: r->status = htonl(110); *dcl = cl; - return sizeof(*r); + return sizeof(*r) + MAD_BLOCK_SIZE; } static int encode_trap128(Port * port, char *data) @@ -1279,7 +1281,7 @@ static int encode_trap_header(char *buf) int send_trap(Port * port, int trapnum) { - struct sim_request req; + struct sim_req256 req; Client *cl; int ret, lid = port->lid; char *data = req.mad + 64; /* data offset */ diff --git a/include/ibsim.h b/include/ibsim.h index 84568e6..14a3f90 100644 --- a/include/ibsim.h +++ b/include/ibsim.h @@ -1,5 +1,6 @@ /* * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -61,6 +62,8 @@ struct sim_port { #define SIM_MAGIC 0xdeadbeef #define SIM_CTL_MAX_DATA 64 +#define MAD_BLOCK_SIZE 256 + struct sim_request { uint32_t dlid; uint32_t slid; @@ -68,7 +71,17 @@ struct sim_request { uint32_t sqp; uint32_t status; uint64_t context; - char mad[256]; + char mad[0]; +}; + +struct sim_req256 { + uint32_t dlid; + uint32_t slid; + uint32_t dqp; + uint32_t sqp; + uint32_t status; + uint64_t context; + char mad[MAD_BLOCK_SIZE]; }; enum SIM_CTL_TYPES { diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c index 4cbf8da..9c69fb5 100644 --- a/umad2sim/umad2sim.c +++ b/umad2sim/umad2sim.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -376,7 +377,7 @@ static int dev_sysfs_create(struct umad2sim_dev *dev) static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) { - struct sim_request req; + struct sim_req256 req; ib_user_mad_t *umad = (ib_user_mad_t *) buf; unsigned mgmt_class; int cnt; @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); DEBUG("umad2sim_read: got %d...\n", cnt); - if (cnt < sizeof(req)) { - ERROR("umad2sim_read: partial request - skip.\n"); - umad->status = EAGAIN; - return umad_size(); - } mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); @@ -411,7 +407,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) umad->status = ntohl(req.status); umad->timeout_ms = 0; umad->retries = 0; - umad->length = umad_size() + sizeof(req.mad); + umad->length = umad_size() + cnt; umad->addr.qpn = req.sqp; umad->addr.qkey = 0; // agent->qkey; @@ -431,9 +427,9 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) static ssize_t umad2sim_write(struct umad2sim_dev *dev, const void *buf, size_t count) { - struct sim_request req; + struct sim_request *req; ib_user_mad_t *umad = (ib_user_mad_t *) buf; - int cnt; + int cnt, ocnt; #ifdef SIMULATE_SEND_ERRORS { static int err_count; @@ -464,25 +460,32 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, mad_get_field(umad_get_mad(umad), 0, IB_MAD_ATTRMOD_F) ); - req.dlid = umad->addr.lid; - req.slid = req.dlid == 0xffff ? 0xffff : 0; /* 0 - means auto + cnt = count - umad_size(); + if (cnt > MAD_BLOCK_SIZE) + cnt = MAD_BLOCK_SIZE; + req = malloc(sizeof(*req) + cnt); + if (!req) { + ERROR("umad2sim_write: no mem for sim req: %m"); + return -1; + } + + req->dlid = umad->addr.lid; + req->slid = req->dlid == 0xffff ? 0xffff : 0; /* 0 - means auto (supported by ibsim) */ ; - req.dqp = umad->addr.qpn; - req.sqp = htonl(dev->agents[umad->agent_id].qpn); - req.status = 0; - req.context = 0; + req->dqp = umad->addr.qpn; + req->sqp = htonl(dev->agents[umad->agent_id].qpn); + req->status = 0; + req->context = 0; - cnt = count - umad_size(); - if (cnt > sizeof(req.mad)) - cnt = sizeof(req.mad); - memcpy(req.mad, umad_get_mad(umad), cnt); + memcpy(req->mad, umad_get_mad(umad), cnt); - cnt = write(dev->sim_client.fd_pktout, (void *)&req, sizeof(req)); - if (cnt < 0) { + ocnt = write(dev->sim_client.fd_pktout, req, sizeof(*req) + cnt); + free(req); + if (ocnt < 0) { ERROR("umad2sim_write: cannot write\n"); return -1; } - if (cnt < sizeof(req)) { + if (ocnt < sizeof(*req) + cnt) { ERROR("umad2sim_write: partial write\n"); } From hrosenstock at xsigo.com Wed Jun 11 07:42:59 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 07:42:59 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <484EFB6A.2030907@nasa.gov> References: <484EFB6A.2030907@nasa.gov> Message-ID: <1213195379.14047.333.camel@hrosenstock-ws.xsigo.com> Jeff, On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: > Hi all. I was looking into doing some subnet partitioning to separate > compute nodes from Lustre nodes, and I saw the following in > ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt > > OpenSM Partition Management > --------------------------- > > Roadmap: > Phase 1 - provide partition management at the EndPort (HCA, Router and Switch > Port 0) level with no routing affects. This is where we currently are at. Partitioning works fine with this "level" of support and has been used with IPoIB AFAIK although other ULPs may support it too (e.g. RDMA CM based ones may get it for "free"). > Phase 2 - routing engine should take partitions into account. > ... > Phase 2 functionality: > > The partition policy should be considered during the routing such that > links are associated with particular partition or a set of > partitions. Policy should be enhanced to provide hints for how to do > that (correlating to QoS too). The exact algorithm is TBD. > > > What is the status of Pkey-aware routing? I'm unaware of any progress on phase 2 support. BTW, this is a dual edged sword as to whether this would cause additional routing time on the SM node and whether there are any worthwhile benefits. -- Hal > Thanks. > > -jeff > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jean-vincent.ficet at bull.net Wed Jun 11 07:51:31 2008 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Wed, 11 Jun 2008 16:51:31 +0200 Subject: [ofa-general] Issues with _osm_lin_fwd_tbl Message-ID: <484FE673.4060406@bull.net> Hi, The function opensm/osm_lin_fwd_table.c:osm_lin_tbl_new() invokes __osm_lin_tbl_compute_obj_size with a 'size' parameter that is obviously a LID (uint16_t). However, the prototype for __osm_lin_tbl_compute_obj_size is: static inline size_t __osm_lin_tbl_compute_obj_size(IN const uint32_t num_ports) Furthermore, the comments in the osm_lin_fwd_tbl_t declaration say: * port_tbl * The array that specifies the port number which routes the * corresponding LID. Index is by LID. * Am I right in saying that the prototype for __osm_lin_tbl_compute_obj_size() should be: static inline size_t __osm_lin_tbl_compute_obj_size(IN const uint16_t num_lids) This would avoid casting its return value as follows in osm_lin_tbl_new(): p_tbl->size = (uint16_t) size; Also, could anyone explain why the osm_lin_fwd_tbl_t is declared as follows: typedef struct _osm_lin_fwd_tbl { uint16_t size; uint8_t port_tbl[1]; } osm_lin_fwd_tbl_t; This prototype is obviously quite different than the corresponding malloc'ed space in __osm_lin_tbl_compute_obj_size(): return (sizeof(osm_lin_fwd_tbl_t) + (num_ports - 1)); Thanks, Vincent From mhagen at iol.unh.edu Wed Jun 11 08:36:42 2008 From: mhagen at iol.unh.edu (Mikkel Hagen) Date: Wed, 11 Jun 2008 11:36:42 -0400 Subject: [ofa-general] Open Fabrics Interop Working Group Logo List Message-ID: <484FF10A.2080202@iol.unh.edu> The OFA Interop Logo Group (OFILG) would like to announce that the logo list from the last interop event in March is now complete and can be seen at: http://www.iol.unh.edu/services/testing/ofa/interoplist/mar08/mar08.php If you have any questions/comments you can email the UNH-IOL group at ofalab at iol.unh.edu or the interop group of the OFA at interop-wg at list.openfabrics.org Thank you! -- Mikkel Hagen Project Assistant - Fibre Channel/SAS/SATA Consortiums Research and Development Engineer - iWARP Consortium FC/SAS/SATA:1-603-862-0701 iWARP:1-603-862-5083 Fax:1-603-862-4181 UNH-IOL 121 Technology Drive, Suite 2 Durham, NH 03824 From hrosenstock at xsigo.com Wed Jun 11 08:40:37 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 08:40:37 -0700 Subject: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <484FE673.4060406@bull.net> References: <484FE673.4060406@bull.net> Message-ID: <1213198837.14047.339.camel@hrosenstock-ws.xsigo.com> Hi Vincent, On Wed, 2008-06-11 at 16:51 +0200, Vincent Ficet wrote: > Hi, > > The function opensm/osm_lin_fwd_table.c:osm_lin_tbl_new() invokes > __osm_lin_tbl_compute_obj_size with a 'size' parameter that is obviously > a LID (uint16_t). However, the prototype for > __osm_lin_tbl_compute_obj_size is: > > static inline size_t __osm_lin_tbl_compute_obj_size(IN const uint32_t > num_ports) > Furthermore, the comments in the osm_lin_fwd_tbl_t declaration say: > > * port_tbl > * The array that specifies the port number which routes the > * corresponding LID. Index is by LID. > * I think it comes from the IB spec where LFT block indicates port (indexed by LID). > Am I right in saying that the prototype for > __osm_lin_tbl_compute_obj_size() should be: > > static inline size_t __osm_lin_tbl_compute_obj_size(IN const uint16_t > num_lids) > > This would avoid casting its return value as follows in osm_lin_tbl_new(): > > p_tbl->size = (uint16_t) size; Seems reasonable to me. > Also, could anyone explain why the osm_lin_fwd_tbl_t is declared as follows: > > typedef struct _osm_lin_fwd_tbl { > uint16_t size; > uint8_t port_tbl[1]; > } osm_lin_fwd_tbl_t; > > This prototype is obviously quite different than the corresponding > malloc'ed space in __osm_lin_tbl_compute_obj_size(): > > return (sizeof(osm_lin_fwd_tbl_t) + (num_ports - 1)); I think this is because the osm_lin_fwd_tbl_t already includes the first entry so this looks "right" to me. Do you care to supply a patch for this ? I also see one other issue in the area of the code you are asking about. -- Hal > Thanks, > > Vincent > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Wed Jun 11 08:46:01 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 08:46:01 -0700 Subject: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <1213198837.14047.339.camel@hrosenstock-ws.xsigo.com> References: <484FE673.4060406@bull.net> <1213198837.14047.339.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213199161.14047.342.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-11 at 08:40 -0700, Hal Rosenstock wrote: > I also see one other issue in the area of the code you are asking > about. My bad; what I thought might be a problem isn't. -- Hal From hrosenstock at xsigo.com Wed Jun 11 08:55:39 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 08:55:39 -0700 Subject: [ofa-general] [PATCH] opensm/osm_lin_fwd_tbl.c: Minor change to __osm_lin_tbl_compute_obj_size In-Reply-To: <1213198837.14047.339.camel@hrosenstock-ws.xsigo.com> References: <484FE673.4060406@bull.net> <1213198837.14047.339.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213199739.14047.347.camel@hrosenstock-ws.xsigo.com> opensm/osm_lin_fwd_tbl.c: Minor change to __osm_lin_tbl_compute_obj_size Change input parameter to be uint16_t r.t. uint32_6 Also, input parameter is really num_lids and not num_ports Based on comments from Vincent Ficet Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_lin_fwd_tbl.c b/opensm/opensm/osm_lin_fwd_tbl.c index 5c969c7..7d1eeff 100644 --- a/opensm/opensm/osm_lin_fwd_tbl.c +++ b/opensm/opensm/osm_lin_fwd_tbl.c @@ -50,9 +50,9 @@ #include #include -static inline size_t __osm_lin_tbl_compute_obj_size(IN const uint32_t num_ports) +static inline size_t __osm_lin_tbl_compute_obj_size(IN const uint16_t num_lids) { - return (sizeof(osm_lin_fwd_tbl_t) + (num_ports - 1)); + return (sizeof(osm_lin_fwd_tbl_t) + (num_lids - 1)); } /********************************************************************** @@ -74,7 +74,7 @@ osm_lin_fwd_tbl_t *osm_lin_tbl_new(IN uint16_t const size) */ if (p_tbl != NULL) { memset(p_tbl, OSM_NO_PATH, __osm_lin_tbl_compute_obj_size(size)); - p_tbl->size = (uint16_t) size; + p_tbl->size = size; } return (p_tbl); } From johnw.macinnis at ns.sympatico.ca Wed Jun 11 09:19:48 2008 From: johnw.macinnis at ns.sympatico.ca (UK LOTTERY INTL.) Date: Wed, 11 Jun 2008 12:19:48 -0400 Subject: [ofa-general] Contact Mr.Pinkett Griffin Message-ID: <6tl3j9$1h294j@toip41-bus.srvr.bell.ca> UK NATIONAL LOTTERY BOARD ONLINE NOTIFICATION Contact Mr.Pinkett Griffin for the claim of £1.000.000pounds which you have won in UK-LOTTERY COMPANY. Provide your Names,Address,Age,Occupation, Tel,Country.Email:mails.notification at btinternet.com Sincerely, Mrs.Rose Woods. UK LOTTERY BOARD From tuan.phamdo at intel.com Wed Jun 11 09:31:09 2008 From: tuan.phamdo at intel.com (Phamdo, Tuan) Date: Wed, 11 Jun 2008 09:31:09 -0700 Subject: [ofa-general] RE: [Interop-wg] Open Fabrics Interop Working Group Logo List In-Reply-To: <484FF10A.2080202@iol.unh.edu> References: <484FF10A.2080202@iol.unh.edu> Message-ID: <07E871F0BCEEB04C9ADA882FA534AEA10315EE21@fmsmsx882.amr.corp.intel.com> I would like to thank the IWG for an outstanding job, especially Lamprey Networks and UNH teams. Regards, Co-Chair IWG Tuan Phamdo Server Platform Group Office: 503-712-4168 Mobile: 503-799-9552 -----Original Message----- From: interop-wg-bounces at lists.openfabrics.org [mailto:interop-wg-bounces at lists.openfabrics.org] On Behalf Of Mikkel Hagen Sent: Wednesday, June 11, 2008 8:37 AM To: interop-wg at lists.openfabrics.org; general at lists.openfabrics.org; promoters at lists.openfabrics.org Subject: [Interop-wg] Open Fabrics Interop Working Group Logo List The OFA Interop Logo Group (OFILG) would like to announce that the logo list from the last interop event in March is now complete and can be seen at: http://www.iol.unh.edu/services/testing/ofa/interoplist/mar08/mar08.php If you have any questions/comments you can email the UNH-IOL group at ofalab at iol.unh.edu or the interop group of the OFA at interop-wg at list.openfabrics.org Thank you! -- Mikkel Hagen Project Assistant - Fibre Channel/SAS/SATA Consortiums Research and Development Engineer - iWARP Consortium FC/SAS/SATA:1-603-862-0701 iWARP:1-603-862-5083 Fax:1-603-862-4181 UNH-IOL 121 Technology Drive, Suite 2 Durham, NH 03824 _______________________________________________ Interop-wg mailing list Interop-wg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/interop-wg From ogerlitz at voltaire.com Wed Jun 11 09:32:01 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 11 Jun 2008 19:32:01 +0300 (IDT) Subject: [ofa-general] S/G support for connected mode?! Message-ID: Hi Roland, I see the under connected mode the ipoib driver does not advertise S/G (NETIF_F_SG) support at its features bitmap. I understand that this would force the stack to do a copy in case the packet to be sent is made of page/fragments held at the SKB, etc. Doesn't it make sense to support S/G also for connected mode? I know that for CSUM support, if S/G is not supported, then the stack ignores the NETIF_F_IP_CSUM bit even if its being set by the nic, but does it apply also the other way - that is if CSUM is not supported then the stack ignores NETIF_F_SG? if yes, why? Or. From roland.list at gmail.com Wed Jun 11 09:32:39 2008 From: roland.list at gmail.com (Roland Dreier) Date: Wed, 11 Jun 2008 09:32:39 -0700 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: <484F7267.6050405@voltaire.com> References: <484F7267.6050405@voltaire.com> Message-ID: > LSO is not exposed to user space by this patch, so maybe better to just reserve/skip this enum value? Sure, not a big deal either way, I'll change it. > and this sets IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS off, so at the bottom line > the user will never see IBV_DEVICE_MEM_MGT_EXTENSIONS being set to ON? or > the user space hw library should somwhow add this bit? Yes, the low-level driver library needs to look at it. I tried to explain it in the comment: + * If the kernel driver says that it supports memory + * management extensions, then move the flag to + * IBV_DEVICE_KERN_MEM_MGT_EXTENSIONS so that the low-level + * driver needs to move the flag back to show it supports the + * operations as well. From roland.list at gmail.com Wed Jun 11 09:34:12 2008 From: roland.list at gmail.com (Roland Dreier) Date: Wed, 11 Jun 2008 09:34:12 -0700 Subject: [ofa-general] S/G support for connected mode?! In-Reply-To: References: Message-ID: > I see the under connected mode the ipoib driver does not advertise S/G > (NETIF_F_SG) support at its features bitmap. I understand that this would > force the stack to do a copy in case the packet to be sent is made of > page/fragments held at the SKB, etc. > > Doesn't it make sense to support S/G also for connected mode? I know that > for CSUM support, if S/G is not supported, then the stack ignores the NETIF_F_IP_CSUM > bit even if its being set by the nic, but does it apply also the other way - > that is if CSUM is not supported then the stack ignores NETIF_F_SG? if yes, why? As I understand things, F_SG is not used if F_IP_CSUM isn't set. Because the network stack would have to compute the the checksum anyway, so it basically has to linearize the skb anyway. - R. From Jeffrey.C.Becker at nasa.gov Wed Jun 11 09:43:56 2008 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Wed, 11 Jun 2008 09:43:56 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213198145.5369.254.camel@cardanus.llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> Message-ID: <485000CC.8070502@nasa.gov> Basically, we have an Altix ICE cluster connected by a pair of hypercube Infiniband fabrics. External to that, we have some Lustre nodes connected into the cluster with Infiniband. Our goal is to keep Lustre traffic separate from compute (MPI) traffic. Ideally, we'd have 2 subnets and an IB router between the Lustre fabric and the compute fabric to accomplish this. Barring that, I thought we could use partitions as follows: compute HCA's and switch ports are on both partitions with full membership in compute partition, and limited membership in I/O partition. The Lustre nodes and switches would only be in the I/O partition (full membership). That way, inter compute node (MPI) traffic would be disallowed from using routes through the I/O fabric (by partition membership), and I/O traffic could not interfere with compute (via separate partitions). Is this scheme feasible? If that's not possible, the next idea is to modify OpenSM to assign large weights to the links between the compute and I/O fabrics, so that the MinHop algorithm would never consider using these links for inter-compute node traffic. Thoughts? Thanks. -jeff Al Chu wrote: > Hey Jeff, > > Out of my curiosity, are you just trying to change the routing to > improve job performance? i.e. lustre nodes get special routing vs. > compute nodes? > > Al > > On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: > >> Hi all. I was looking into doing some subnet partitioning to separate >> compute nodes from Lustre nodes, and I saw the following in >> ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt >> >> OpenSM Partition Management >> --------------------------- >> >> Roadmap: >> Phase 1 - provide partition management at the EndPort (HCA, Router and Switch >> Port 0) level with no routing affects. >> Phase 2 - routing engine should take partitions into account. >> ... >> Phase 2 functionality: >> >> The partition policy should be considered during the routing such that >> links are associated with particular partition or a set of >> partitions. Policy should be enhanced to provide hints for how to do >> that (correlating to QoS too). The exact algorithm is TBD. >> >> >> What is the status of Pkey-aware routing? Thanks. >> >> -jeff >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> From 4Nrmattson at amcore.com Wed Jun 11 08:05:39 2008 From: 4Nrmattson at amcore.com (byram sidarta) Date: Wed, 11 Jun 2008 15:05:39 +0000 Subject: [ofa-general] Brands watch hand at low pricCce Message-ID: <000601c8cbe3$02f257dc$30e5f1a2@crvlhpwx> Look good and fortunate without having to spend a lot of money. Nothing affect co-workers and friends more than a studded Alain Silberstein, or a classic Bvlgari strapped around your wrist. King Replicas houses more than 30+ world famous brands: Emporio Armani Jacob & Co Omega Technomarine and many more! Sale ends this week - so visit now for the best off2r! - Visit our site: www.busniie[DOT]com (copy this link and replace "[DOT]" to ".") From hrosenstock at xsigo.com Wed Jun 11 10:19:01 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 10:19:01 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <485000CC.8070502@nasa.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> Message-ID: <1213204741.14047.353.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-11 at 09:43 -0700, Jeff Becker wrote: > Basically, we have an Altix ICE cluster connected by a pair of hypercube > Infiniband fabrics. External to that, we have some Lustre nodes > connected into the cluster with Infiniband. Our goal is to keep Lustre > traffic separate from compute (MPI) traffic. Ideally, we'd have 2 > subnets and an IB router between the Lustre fabric and the compute > fabric to accomplish this. > > Barring that, I thought we could use partitions as follows: compute > HCA's and switch ports are on both partitions with full membership in > compute partition, and limited membership in I/O partition. The Lustre > nodes and switches would only be in the I/O partition (full > membership). That way, inter compute node (MPI) traffic would be > disallowed from using routes through the I/O fabric (by partition > membership), and I/O traffic could not interfere with compute (via > separate partitions). Is this scheme feasible? It sounds feasible to me but depending on what is meant "by not interfere". Maybe LMC 1 might help here with MPI on base LID and I/O on non base LID (with recent patches on list). -- Hal > If that's not possible, the next idea is to modify OpenSM to assign > large weights to the links between the compute and I/O fabrics, so that > the MinHop algorithm would never consider using these links for > inter-compute node traffic. > Thoughts? Thanks. > > -jeff > > Al Chu wrote: > > Hey Jeff, > > > > Out of my curiosity, are you just trying to change the routing to > > improve job performance? i.e. lustre nodes get special routing vs. > > compute nodes? > > > > Al > > > > On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: > > > >> Hi all. I was looking into doing some subnet partitioning to separate > >> compute nodes from Lustre nodes, and I saw the following in > >> ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt > >> > >> OpenSM Partition Management > >> --------------------------- > >> > >> Roadmap: > >> Phase 1 - provide partition management at the EndPort (HCA, Router and Switch > >> Port 0) level with no routing affects. > >> Phase 2 - routing engine should take partitions into account. > >> ... > >> Phase 2 functionality: > >> > >> The partition policy should be considered during the routing such that > >> links are associated with particular partition or a set of > >> partitions. Policy should be enhanced to provide hints for how to do > >> that (correlating to QoS too). The exact algorithm is TBD. > >> > >> > >> What is the status of Pkey-aware routing? Thanks. > >> > >> -jeff > >> > >> _______________________________________________ > >> general mailing list > >> general at lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >> > >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > >> > From hrosenstock at xsigo.com Wed Jun 11 10:54:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 10:54:29 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/libvendor/osm_vendor_ibumad_sa.c: Eliminate unneeded define Message-ID: <1213206869.14047.362.camel@hrosenstock-ws.xsigo.com> OpenSM/libvendor/osm_vendor_ibumad_sa.c: Eliminate unneeded define Signed-off-by: Hal Rosenstock diff --git a/opensm/libvendor/osm_vendor_ibumad_sa.c b/opensm/libvendor/osm_vendor_ibumad_sa.c index 547d1e5..cef20aa 100644 --- a/opensm/libvendor/osm_vendor_ibumad_sa.c +++ b/opensm/libvendor/osm_vendor_ibumad_sa.c @@ -44,8 +44,6 @@ #include #include -#define MAX_PORTS 64 - /***************************************************************************** *****************************************************************************/ From hrosenstock at xsigo.com Wed Jun 11 11:24:55 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 11:24:55 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/include/opensm: Fix some commentary typos Message-ID: <1213208695.14047.375.camel@hrosenstock-ws.xsigo.com> OpenSM/include/opensm: Fix some commentary typos Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_mcm_info.h b/opensm/include/opensm/osm_mcm_info.h index 2f5eb11..b40771c 100644 --- a/opensm/include/opensm/osm_mcm_info.h +++ b/opensm/include/opensm/osm_mcm_info.h @@ -63,7 +63,7 @@ BEGIN_C_DECLS * * DESCRIPTION * Multicast Membership Info object. -* This object contains information about a nodes membership +* This object contains information about a node's membership * in a particular multicast group. * * This object should be treated as opaque and should diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index ce827d1..9d89e1e 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -181,7 +181,7 @@ typedef struct _osm_mgrp { * representing the member ports of this multicast group. * * mcmember_rec -* Hold the parameters of the Multicast Group. +* Holds the parameters of the Multicast Group. * * well_known * Indicates that this is the wellknow multicast group which From hrosenstock at xsigo.com Wed Jun 11 11:27:17 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 11:27:17 -0700 Subject: [ofa-general] [PATCHv2][TRIVIAL] OpenSM/include/opensm: Fix some commentary typos Message-ID: <1213208837.14047.377.camel@hrosenstock-ws.xsigo.com> OpenSM/include/opensm: Fix some commentary typos Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_mcm_info.h b/opensm/include/opensm/osm_mcm_info.h index 2f5eb11..b40771c 100644 --- a/opensm/include/opensm/osm_mcm_info.h +++ b/opensm/include/opensm/osm_mcm_info.h @@ -63,7 +63,7 @@ BEGIN_C_DECLS * * DESCRIPTION * Multicast Membership Info object. -* This object contains information about a nodes membership +* This object contains information about a node's membership * in a particular multicast group. * * This object should be treated as opaque and should diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index ce827d1..57e5ef5 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -181,10 +181,10 @@ typedef struct _osm_mgrp { * representing the member ports of this multicast group. * * mcmember_rec -* Hold the parameters of the Multicast Group. +* Holds the parameters of the Multicast Group. * * well_known -* Indicates that this is the wellknow multicast group which +* Indicates that this is the wellknown multicast group which * is created during the initialization of SM/SA and will be * present even if there are no ports for this group * From hrosenstock at xsigo.com Wed Jun 11 12:42:19 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 12:42:19 -0700 Subject: [ofa-general] [PATCH] ibsim/sim.h: Allow max ports based on 36 port switches Message-ID: <1213213339.14047.390.camel@hrosenstock-ws.xsigo.com> ibsim/sim.h: Allow max ports based on 36 port switches Signed-off-by: Hal Rosenstock diff --git a/ibsim/sim.h b/ibsim/sim.h index a795ded..936bb85 --- a/ibsim/sim.h +++ b/ibsim/sim.h @@ -40,7 +40,7 @@ #define MAXNETNODES 2048 #define MAXNETSWITCHS 256 -#define MAXNETPORTS (MAXNETSWITCHS*24+MAXNETNODES*2) +#define MAXNETPORTS (MAXNETSWITCHS*36+MAXNETNODES*2) #define MAXNETALIASES MAXNETPORTS #define MAXLINEARCAP (30*1024) From hrosenstock at xsigo.com Wed Jun 11 12:47:34 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 12:47:34 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] OpenSM/include/osm_port_profile.h: Fix some typos Message-ID: <1213213654.14047.394.camel@hrosenstock-ws.xsigo.com> OpenSM/include/osm_port_profile.h: Fix some typos Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index d735388..9997d0b 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -147,7 +147,7 @@ osm_port_prof_path_count_inc(IN osm_port_profile_t * const p_prof) /* * PARAMETERS -* p_pro +* p_prof * [in] Pointer to the Port Profile object. * * RETURN VALUE @@ -175,7 +175,7 @@ osm_port_prof_path_count_get(IN const osm_port_profile_t * const p_prof) /* * PARAMETERS -* p_pro +* p_prof * [in] Pointer to the Port Profile object. * * RETURN VALUE From jgunthorpe at obsidianresearch.com Wed Jun 11 13:19:01 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 11 Jun 2008 14:19:01 -0600 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080608054426.GK22807@obsidianresearch.com> <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080611201901.GB3193@obsidianresearch.com> On Sun, Jun 08, 2008 at 08:02:51AM -0700, Hal Rosenstock wrote: > The only issues with the prefix length are the scope and PKey fields > potentially making this non contiguous but I think the syntax can be > extended for this if needed. Well, I was trying to explain, that if you need this kind of function then split the matching and tagging into two steps. Then it is no problem and you can retain simple to use prefix matching: > > Table 1: > > MGID/PREFIX GROUP NAME > > ff11:601b::/32 IPV6-SNM > > ff1e:601b::/32 IPV6-SNM > > ff1e:601b:89:/48 IPV6-SNM-PKEY_89 > > ff00::/8 OTHERS > > > > Table 2: > > NAME MLIDS > > IPV6-SNM 4 > > IPV6_SNM-PKEY_89 128 > > OTHERS 128 > In terms of IB routers, if an MLID was overloaded, wouldn't they > filter on scope as well (link local scope would be filtered), right? I expect an IB router to ignore the M-DLID entirely, it contains no information that it needs. > > > As PKey is part of the MGID, does this need to be addressed (and if > > > so) how ? > > > > I don't think pkey needs special treatment - prefix matching takes care > > of it. If you feel seperate allocations for each pkey may someday be > > necessary then having a table scheme rather than a single value takes > > care of it nicely. > > So you think it's "safe" to ignore the related IBA compliance ? It does > look to me like that should be changed. I think that was also Roland's > take a while ago. Yes, I don't see the point of the compliance. Switch pkey filtering is done at ingress or egress, and does not depend on the DLID. The only way I could see this making sense is if the original author(s) had in mind some kind of specific optimization for the switch design that would preclude doing multicast replicating, then egress pkey checking. At worst, I'd provide an option to treat pkey in the same manner as MTU/SL and not worry about it unless a switch vendor comes forward to say they need rule #10 to be followed. Jason From hrosenstock at xsigo.com Wed Jun 11 14:05:09 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 11 Jun 2008 14:05:09 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080611201901.GB3193@obsidianresearch.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080608054426.GK22807@obsidianresearch.com> <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> <20080611201901.GB3193@obsidianresearch.com> Message-ID: <1213218309.14047.397.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-11 at 14:19 -0600, Jason Gunthorpe wrote: > I expect an IB router to ignore the M-DLID entirely, it contains no > information that it needs. What about M-DGID ? It needs to at least filter on link local scope if the MLID were overloaded. Other MGID scopes are under admin control. -- Hal From jgunthorpe at obsidianresearch.com Wed Jun 11 14:09:10 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 11 Jun 2008 15:09:10 -0600 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1213218309.14047.397.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080608054426.GK22807@obsidianresearch.com> <1212937371.14047.38.camel@hrosenstock-ws.xsigo.com> <20080611201901.GB3193@obsidianresearch.com> <1213218309.14047.397.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080611210910.GM22807@obsidianresearch.com> On Wed, Jun 11, 2008 at 02:05:09PM -0700, Hal Rosenstock wrote: > On Wed, 2008-06-11 at 14:19 -0600, Jason Gunthorpe wrote: > > I expect an IB router to ignore the M-DLID entirely, it contains no > > information that it needs. > > What about M-DGID ? It needs to at least filter on link local scope if > the MLID were overloaded. Other MGID scopes are under admin control. Yes, it must act based on M-DGID depending on how it is configured, minimally by discaring link local scope. That is mostly likely all folded into the larger multicast routing table a router will need have. Jason From 973krishnan.venkat at tata-aig.com Wed Jun 11 16:13:00 2008 From: 973krishnan.venkat at tata-aig.com (killian bennie) Date: Wed, 11 Jun 2008 23:13:00 +0000 Subject: [ofa-general] Best of Adidas, Boss, UGG Message-ID: <000401c8cc27$02049d38$efe85794@udngmhgs> The international largest luxury store for shoes and bags is just one click away. Recommended by tens thousands of grateful customers all around the world, we have many of populars brands including: Prada Armani Boss Paul Smith Prada Shoes Here you willc see millions of owsome designs for shoes, and leather products, at at temp't1ng priceE. Sale ends this week, so visit us immediately and start pampering yourself and your loved ones! - Visit our site: www.rapaette[DOT]com (copy this link and replace "[DOT]" to ".") From ogerlitz at voltaire.com Wed Jun 11 23:11:20 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Jun 2008 09:11:20 +0300 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: <484E9181.4000107@voltaire.com> References: <15ddcffd0806031202l248b4fdbgcd6cd98f5c22ce65@mail.gmail.com> <484E9181.4000107@voltaire.com> Message-ID: <4850BE08.3050900@voltaire.com> Or Gerlitz wrote: > This way, each call to the MCG attach verb would be duplicated in the > code, since if it fails when the "block loopback" param was set it > would now be called again unset, etc. I think this turns to be more > complex then doing a query for a cap bit and then setting the block > mcast through a QP create flag once. To my taste, the fact that the QP > create flag is used by the driver at the time of MCG attach does not > make it a more complex implementation. Hi Roland, Can you comment on this thread, we want to get the patches into 2.6.27 and then to libibverbs. thanks, Or. From ogerlitz at voltaire.com Wed Jun 11 23:17:20 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Jun 2008 09:17:20 +0300 Subject: [ofa-general] enabling invalidate operations in libibverbs API In-Reply-To: References: <484F7267.6050405@voltaire.com> Message-ID: <4850BF70.9010908@voltaire.com> Roland Dreier wrote: > Yes, the low-level driver library needs to look at it. I tried to explain it in the comment: > + so that the low-level driver needs to move the flag back to show it supports the > + operations as well. OK, got it. Or. From tziporet at dev.mellanox.co.il Wed Jun 11 23:24:28 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 12 Jun 2008 09:24:28 +0300 Subject: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp In-Reply-To: <39C75744D164D948A170E9792AF8E7CAC5AFD1@exil.voltaire.com> References: <39C75744D164D948A170E9792AF8E7CAC5AFD1@exil.voltaire.com> Message-ID: <4850C11C.80106@mellanox.co.il> Moshe, What do you expect us to do with these backport patches? Tziporet From moshek at voltaire.com Wed Jun 11 23:32:27 2008 From: moshek at voltaire.com (Moshe Kazir) Date: Thu, 12 Jun 2008 09:32:27 +0300 Subject: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp In-Reply-To: <4850C11C.80106@mellanox.co.il> References: <39C75744D164D948A170E9792AF8E7CAC5AFD1@exil.voltaire.com> <4850C11C.80106@mellanox.co.il> Message-ID: <39C75744D164D948A170E9792AF8E7CAC5AFD6@exil.voltaire.com> OFED-1.3.1 & OFED-1.3 (OFA_kernel*src.rpm) have comppile errors on SLES10 SP1 Lustre without this patch .... Add it to OFED-1.4 ? Moshe ____________________________________________________________ Moshe Katzir | +972-9971-8639 (o) | +972-52-860-6042 (m) Voltaire - The Grid Backbone www.voltaire.com -----Original Message----- From: Tziporet Koren [mailto:tziporet at dev.mellanox.co.il] Sent: Thursday, June 12, 2008 9:24 AM To: Moshe Kazir Cc: Vladimir Sokolovsky (Mellanox); general at lists.openfabrics.org; Yair Goldel; Olga Shern; Avi Dahan Subject: Re: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp Moshe, What do you expect us to do with these backport patches? Tziporet From olgas at voltaire.com Wed Jun 11 23:46:18 2008 From: olgas at voltaire.com (Olga Shern) Date: Thu, 12 Jun 2008 09:46:18 +0300 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup Message-ID: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> Hi All, We have found something that seems like Infiniband Spec hole, This issue is system issue that prevents from partial P_Key setup to go into production. Short Setup & test description: ------------------------------------------ * Node A: P_Key XXX (full member) * Node B, C, D, E, F: P_Key XXx (partial member) 1. Send ping from B -> A : ping is OK 2. Send ping from C -> A : ping is OK 3. Send ping from B -> C : no ping also OK * Get traps Bad P_Key in SM - from all HCA in the fabric both for test 1 & 2 (one time) and also for test 3 (all the time). Probably the ARP request that is MC traffic generate the trap in HCA, for test 1 & 2 we have only one ARP but for test 3 we send ARP all the time because we do not get any ARP reply. * The trap number SM get is 257 (HCA trap) if we will do P_Key switch enforcement we will probably get 259 . * We get trap also from the originator of the MC traffic even though that receive switch relay error counter is increased (when out port==in port), the switch does not drop the packet ? Additional questions/issues: * Do we have a way to suppress port traps from SMA ?? i.e. that the port will not generate traps that can "kill the SM" - as its look this is bug in the spec where we can't send any mc traffic (even ARP) when we have partial members and we do not have a way to suppress the traps. * What will happen in the HCA when we get many traps (mc packets from many nodes) and they need to keep all events until SM will acknowledge? - Is there limitation in the number of on-going traps (any HCA specific issues)? Best Regards Olga -------------- next part -------------- An HTML attachment was scrubbed... URL: From benh at kernel.crashing.org Thu Jun 12 00:23:15 2008 From: benh at kernel.crashing.org (Benjamin Herrenschmidt) Date: Thu, 12 Jun 2008 17:23:15 +1000 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806101644.34693.ossrosch@linux.vnet.ibm.com> References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> Message-ID: <1213255395.14478.28.camel@pasglop> On Tue, 2008-06-10 at 16:44 +0200, Stefan Roscher wrote: > During corner case testing, we noticed that some versions of ehca > do not properly transition to interrupt done in special load situations. > This can be resolved by periodically triggering EOI through H_EOI, > if eqes are pending. > > Signed-off-by: Stefan Roscher This patch is fine with me as long as the FW/HW people can confirm that calling spurrious EOI's like that will not affect other interrupts. The side effect of writing 0xff to the xirr should be irrelevant as long as this is not done from within a HW interrupt handler (timer interrupts or softirqs are fine). Due to the already incestuous relationship between HCA and the hypervisor, I don't mind having the H call directly in the driver. So as long as the FW/HW people are ok with that workaround, then it has my ack as well. Acked-by: Benjamin Herrenschmidt --- From paulus at samba.org Thu Jun 12 01:37:32 2008 From: paulus at samba.org (Paul Mackerras) Date: Thu, 12 Jun 2008 18:37:32 +1000 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> <200806101701.47090.ossrosch@linux.vnet.ibm.com> Message-ID: <18512.57420.267003.723681@cargo.ozlabs.ibm.com> Roland Dreier writes: > > > So just to be clear: this is a workaround for a hardware/firmware bug? > > > Yes it is. > > OK, so paulus et al... does it seem like a good approach to call H_EOI > from driver code (given that this driver makes tons of other hcalls)? If this workaround is approved by the relevant hardware & firmware folks, and demonstrably fixes the bug, then I am happy with it. Paul. From vlad at lists.openfabrics.org Thu Jun 12 02:16:00 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 12 Jun 2008 02:16:00 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080612-0200 daily build status Message-ID: <20080612091600.DDD44E610AD@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 1 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 2 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: error: too many arguments to function 'ip_route_output_key' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080612-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From paulus at samba.org Thu Jun 12 02:39:14 2008 From: paulus at samba.org (Paul Mackerras) Date: Thu, 12 Jun 2008 19:39:14 +1000 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806101644.34693.ossrosch@linux.vnet.ibm.com> References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> Message-ID: <18512.61122.976550.619180@cargo.ozlabs.ibm.com> Stefan Roscher writes: > During corner case testing, we noticed that some versions of ehca > do not properly transition to interrupt done in special load situations. > This can be resolved by periodically triggering EOI through H_EOI, > if eqes are pending. > > Signed-off-by: Stefan Roscher If this is OKed by the hypervisor team, then you can add: Acked-by: Paul Mackerras Paul. From sashak at voltaire.com Thu Jun 12 03:00:51 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 13:00:51 +0300 Subject: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <484FE673.4060406@bull.net> References: <484FE673.4060406@bull.net> Message-ID: <20080612100051.GI14494@sashak.voltaire.com> On 16:51 Wed 11 Jun , Vincent Ficet wrote: > > Also, could anyone explain why the osm_lin_fwd_tbl_t is declared as > follows: > > typedef struct _osm_lin_fwd_tbl { > uint16_t size; > uint8_t port_tbl[1]; > } osm_lin_fwd_tbl_t; AFAIR the reason is a poor windows compiler which doesn't support array[0] like definitions in structures. Sasha From hrosenstock at xsigo.com Thu Jun 12 03:17:09 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 03:17:09 -0700 Subject: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <20080612100051.GI14494@sashak.voltaire.com> References: <484FE673.4060406@bull.net> <20080612100051.GI14494@sashak.voltaire.com> Message-ID: <1213265829.14047.409.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 13:00 +0300, Sasha Khapyorsky wrote: > On 16:51 Wed 11 Jun , Vincent Ficet wrote: > > > > Also, could anyone explain why the osm_lin_fwd_tbl_t is declared as > > follows: > > > > typedef struct _osm_lin_fwd_tbl { > > uint16_t size; > > uint8_t port_tbl[1]; > > } osm_lin_fwd_tbl_t; > > AFAIR the reason is a poor windows compiler which doesn't support > array[0] like definitions in structures. Yes, I too recall something being said along those lines quite a while ago. Is this accurate ? If so, is this still a restriction of the Windows compilers in use (as I think the build environment and perhaps the compilers have changed quite a bit since then) ? Can someone knowledgeable in Windows comment on this ? -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Thu Jun 12 03:30:26 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 03:30:26 -0700 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> Message-ID: <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> Hi Olga, On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > Hi All, > > > > We have found something that seems like Infiniband Spec hole, What's the spec hole ? > This issue is system issue that prevents from partial P_Key setup to > go into production. Indeed :-( > Short Setup & test description: > ------------------------------------------ > * Node A: P_Key XXX (full member) > * Node B, C, D, E, F: P_Key XXx (partial member) > > 1. Send ping from B -> A : ping is OK > 2. Send ping from C -> A : ping is OK > 3. Send ping from B -> C : no ping also OK > * Get traps Bad P_Key in SM - from all HCA in the fabric both for > test 1 & 2 (one time) and also for test 3 (all the time). > > Probably the ARP request that is MC traffic generate the trap in HCA, > for test 1 > & 2 we have only one ARP but for test 3 we send ARP all the time > because > we do not get any ARP reply. > > * The trap number SM get is 257 (HCA trap) if we will do P_Key > switch enforcement we will probably get 259 Is this with OpenSM or VSM ? -- Hal > * We get trap also from the originator of the MC traffic even > though that receive switch relay error counter is increased (when out > port==in port), the switch does not drop the packet ? > > Additional questions/issues: > * Do we have a way to suppress port traps from SMA ?? i.e. that > the port will not generate traps that can "kill the SM" - as its look > this is bug in the spec where we can't send any mc traffic (even ARP) > when we have partial members and we do not have a way to suppress the > traps. > > > * What will happen in the HCA when we get many traps (mc packets > from many nodes) and they need to keep all events until SM will > acknowledge? - Is there limitation in the number of on-going > traps (any HCA specific issues)? > > > > > > Best Regards > > Olga > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ronli.voltaire at gmail.com Thu Jun 12 03:50:00 2008 From: ronli.voltaire at gmail.com (Ron Livne) Date: Thu, 12 Jun 2008 13:50:00 +0300 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: References: Message-ID: <3b5e77ad0806120350r7dff2239h60a44a8bc50ec943@mail.gmail.com> On Mon, Jun 2, 2008 at 7:35 PM, Roland Dreier wrote: > > + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), > > This clashes with MEM_MGT_EXTENSIONS. looking in the for-2.6.27 branch of your tree, IB_DEVICE_SEND_W_INV (21) was removed and IB_DEVICE_MEM_MGT_EXTENSIONS (21) was added instead. So it seems that the value of 22 is free for adding a cap bit, if we decide we go this way. > > Do we really need to burn a capability bit for this? Would it cause any > problems to make IB_QP_BLOCK_LOOPBACK a hint that the underlying HCA may > or may not support? > > > enum ib_qp_create_flags { > > IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, > > + IB_QP_BLOCK_LOOPBACK = 1 << 1, > > }; > > The naming convention is that this should be IB_QP_CREATE_BLOCK_LOOPBACK. You are right, will be fixed. > > - R. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonid at mellanox.co.il Thu Jun 12 03:50:45 2008 From: leonid at mellanox.co.il (Leonid Keller) Date: Thu, 12 Jun 2008 13:50:45 +0300 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <1213265829.14047.409.camel@hrosenstock-ws.xsigo.com> References: <484FE673.4060406@bull.net><20080612100051.GI14494@sashak.voltaire.com> <1213265829.14047.409.camel@hrosenstock-ws.xsigo.com> Message-ID: <6C2C79E72C305246B504CBA17B5500C9043B16F5@mtlexch01.mtl.com> One can use zero-sized arrays as a structure member, having disabled the warning 4200. Here is a real example from hw\mlx4\kernel\bus\core\cache.c: #pragma warning( disable : 4200) struct ib_pkey_cache { int table_len; __be16 table[0]; }; #pragma warning( default : 4200) > -----Original Message----- > From: ofw-bounces at lists.openfabrics.org > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Hal Rosenstock > Sent: Thursday, June 12, 2008 1:17 PM > To: Sasha Khapyorsky > Cc: ofw at lists.openfabrics.org; Vincent Ficet; > general at lists.openfabrics.org > Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl > > On Thu, 2008-06-12 at 13:00 +0300, Sasha Khapyorsky wrote: > > On 16:51 Wed 11 Jun , Vincent Ficet wrote: > > > > > > Also, could anyone explain why the osm_lin_fwd_tbl_t is > declared as > > > follows: > > > > > > typedef struct _osm_lin_fwd_tbl { > > > uint16_t size; > > > uint8_t port_tbl[1]; > > > } osm_lin_fwd_tbl_t; > > > > AFAIR the reason is a poor windows compiler which doesn't support > > array[0] like definitions in structures. > > Yes, I too recall something being said along those lines > quite a while ago. > > Is this accurate ? If so, is this still a restriction of the > Windows compilers in use (as I think the build environment > and perhaps the compilers have changed quite a bit since > then) ? Can someone knowledgeable in Windows comment on this ? > > -- Hal > > > Sasha > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ > ofw mailing list > ofw at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > From hrosenstock at xsigo.com Thu Jun 12 03:59:11 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 03:59:11 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <20080610025939.GK10301@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> Message-ID: <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: > Basically this addresses the problem described by Al Chu in: > > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html > > When base lid paths become completely disbalanced on a fabrics with > lmc > 0. > > One feedback was from Yiftah Shahar: > > "I think that our requirements should be that even when you are working > with LMC>0 then the base LID routing should not be affected. > One way to achieve this goal is to first run the base-LID routing (so > all base LID improvement will be also in LMC>0) and then start with the > other LIDs as round-robbing starting from the base-lid-port + 1 > according current routing algorithm rules (keeping min-hop, up/down...)." > > We had some discussion with Al and Yiftah about this and considered that > in addition to "pure" base lid paths preservation (which is good thing by > itself) proposed method solves original lid disbalancing problem as well. Would you elaborate on the motivation behind the requirement to maintain/preserve the base LID routing ? -- Hal From olga.shern at gmail.com Thu Jun 12 04:08:47 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Thu, 12 Jun 2008 14:08:47 +0300 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> Message-ID: On 6/12/08, Hal Rosenstock wrote: > > Hi Olga, > > On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > > Hi All, > > > > > > > > We have found something that seems like Infiniband Spec hole, > > What's the spec hole ? According to the Infiniband spec - partial member cannot "talk" with partial member only with full member. Therefore if partial member sending MC packet - all other partial members of this partition will generate BAD PKEY trap. It means that the behavior that we see is according to Infiniband Spec - but very problematic > This issue is system issue that prevents from partial P_Key setup to > > go into production. > > Indeed :-( > > > Short Setup & test description: > > ------------------------------------------ > > * Node A: P_Key XXX (full member) > > * Node B, C, D, E, F: P_Key XXx (partial member) > > > > 1. Send ping from B -> A : ping is OK > > 2. Send ping from C -> A : ping is OK > > 3. Send ping from B -> C : no ping also OK > > * Get traps Bad P_Key in SM - from all HCA in the fabric both for > > test 1 & 2 (one time) and also for test 3 (all the time). > > > > Probably the ARP request that is MC traffic generate the trap in HCA, > > for test 1 > > & 2 we have only one ARP but for test 3 we send ARP all the time > > because > > we do not get any ARP reply. > > > > * The trap number SM get is 257 (HCA trap) if we will do P_Key > > switch enforcement we will probably get 259 > > Is this with OpenSM or VSM ? We tested it with Voltaire SM but it should behave the same with OpenSM. -- Hal > > > * We get trap also from the originator of the MC traffic even > > though that receive switch relay error counter is increased (when out > > port==in port), the switch does not drop the packet ? > > > > Additional questions/issues: > > * Do we have a way to suppress port traps from SMA ?? i.e. that > > the port will not generate traps that can "kill the SM" - as its look > > this is bug in the spec where we can't send any mc traffic (even ARP) > > when we have partial members and we do not have a way to suppress the > > traps. > > > > > > * What will happen in the HCA when we get many traps (mc packets > > from many nodes) and they need to keep all events until SM will > > acknowledge? - Is there limitation in the number of on-going > > traps (any HCA specific issues)? > > > > > > > > > > > > Best Regards > > > > Olga > > > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Thu Jun 12 04:33:55 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 14:33:55 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612113355.GB6256@sashak.voltaire.com> On 03:59 Thu 12 Jun , Hal Rosenstock wrote: > > Would you elaborate on the motivation behind the requirement to > maintain/preserve the base LID routing ? I see couple advantages: 1. Application which works on base LIDs only will not be affected by LMC 2. Changing LMC on a fabric will not change routing paths (when LIDs are not reassigned) 3. Finally it does better balancing for secondary LIDs ("port offsetting") Sasha From hrosenstock at xsigo.com Thu Jun 12 04:39:48 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 04:39:48 -0700 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 14:08 +0300, Olga Shern (Voltaire) wrote: > > > On 6/12/08, Hal Rosenstock wrote: > Hi Olga, > > On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > > Hi All, > > > > > > > > We have found something that seems like Infiniband Spec > hole, > > What's the spec hole ? > > According to the Infiniband spec - partial member cannot "talk" with > partial member only with full member. > Therefore if partial member sending MC packet - all other partial > members of this partition will generate BAD PKEY trap. > It means that the behavior that we see is according to Infiniband > Spec - but very problematic Originally, multicast groups were all full member only and more recently was this extended to allow partial members and this was missed. A comment should be filed against the spec on this. > > This issue is system issue that prevents from partial P_Key > setup to > > go into production. > > Indeed :-( > > > Short Setup & test description: > > ------------------------------------------ > > * Node A: P_Key XXX (full member) > > * Node B, C, D, E, F: P_Key XXx (partial member) > > > > 1. Send ping from B -> A : ping is OK > > 2. Send ping from C -> A : ping is OK > > 3. Send ping from B -> C : no ping also OK > > * Get traps Bad P_Key in SM - from all HCA in the fabric > both for > > test 1 & 2 (one time) and also for test 3 (all the time). What does all the time mean ? Does this mean with one test 3 ping, the traps are repeated ? If so, at what rate ? > > Probably the ARP request that is MC traffic generate the > trap in HCA, > > for test 1 > > & 2 we have only one ARP but for test 3 we send ARP all the > time > > because > > we do not get any ARP reply. > > > > * The trap number SM get is 257 (HCA trap) if we will do > P_Key > > switch enforcement we will probably get 259 > > Is this with OpenSM or VSM ? > > We tested it with Voltaire SM but it should behave the same with > OpenSM. That's likely but I'm not sure yet. > -- Hal > > > * We get trap also from the originator of the MC traffic > even > > though that receive switch relay error counter is increased > (when out > > port==in port), the switch does not drop the packet ? The implementation of that counter is broken and occurs "normally". The increment of this counter is relatively meaningless :-( > > Additional questions/issues: > > * Do we have a way to suppress port traps from SMA ?? i.e. > that > > the port will not generate traps that can "kill the SM" - as > its look > > this is bug in the spec where we can't send any mc traffic > (even ARP) > > when we have partial members and we do not have a way to > suppress the > > traps. All the SM can do is TrapRepress. > > * What will happen in the HCA when we get many traps (mc > packets > > from many nodes) and they need to keep all events until SM > will > > acknowledge? - Is there limitation in the number of on- > going > > traps (any HCA specific issues)? Assuming you mean events from which traps are generated, I think this is left as an implementation dependent detail in terms of the spec. An implementation needs to take care not to lose certain events; others like this aren't critical but that's left to the specific SMA implementation. -- Hal > > > > > > > > > > Best Regards > > > > Olga > > > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi- > bin/mailman/listinfo/general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From hrosenstock at xsigo.com Thu Jun 12 04:43:08 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 04:43:08 -0700 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <6C2C79E72C305246B504CBA17B5500C9043B16F5@mtlexch01.mtl.com> References: <484FE673.4060406@bull.net> <20080612100051.GI14494@sashak.voltaire.com> <1213265829.14047.409.camel@hrosenstock-ws.xsigo.com> <6C2C79E72C305246B504CBA17B5500C9043B16F5@mtlexch01.mtl.com> Message-ID: <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 13:50 +0300, Leonid Keller wrote: > One can use zero-sized arrays as a structure member, having disabled the > warning 4200. > Here is a real example from hw\mlx4\kernel\bus\core\cache.c: > > #pragma warning( disable : 4200) > struct ib_pkey_cache { > int table_len; > __be16 table[0]; > }; > #pragma warning( default : 4200) So to use this with Windows, any zero size array needs to be surrounded by the pragma pair ? -- Hal > > -----Original Message----- > > From: ofw-bounces at lists.openfabrics.org > > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Hal Rosenstock > > Sent: Thursday, June 12, 2008 1:17 PM > > To: Sasha Khapyorsky > > Cc: ofw at lists.openfabrics.org; Vincent Ficet; > > general at lists.openfabrics.org > > Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl > > > > On Thu, 2008-06-12 at 13:00 +0300, Sasha Khapyorsky wrote: > > > On 16:51 Wed 11 Jun , Vincent Ficet wrote: > > > > > > > > Also, could anyone explain why the osm_lin_fwd_tbl_t is > > declared as > > > > follows: > > > > > > > > typedef struct _osm_lin_fwd_tbl { > > > > uint16_t size; > > > > uint8_t port_tbl[1]; > > > > } osm_lin_fwd_tbl_t; > > > > > > AFAIR the reason is a poor windows compiler which doesn't support > > > array[0] like definitions in structures. > > > > Yes, I too recall something being said along those lines > > quite a while ago. > > > > Is this accurate ? If so, is this still a restriction of the > > Windows compilers in use (as I think the build environment > > and perhaps the compilers have changed quite a bit since > > then) ? Can someone knowledgeable in Windows comment on this ? > > > > -- Hal > > > > > Sasha > > > _______________________________________________ > > > general mailing list > > > general at lists.openfabrics.org > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > _______________________________________________ > > ofw mailing list > > ofw at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > > From hrosenstock at xsigo.com Thu Jun 12 04:45:07 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 04:45:07 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 14:22 +0300, Olga Shern (Voltaire) wrote: > > > On 6/12/08, Olga Shern (Voltaire) wrote: > > > On 6/12/08, Hal Rosenstock wrote: > On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky > wrote: > > Basically this addresses the problem described by Al > Chu in: > > > > http://lists.openfabrics.org/pipermail/general/2008- > April/049132.html > > > > When base lid paths become completely disbalanced on > a fabrics with > > lmc > 0. > > > > One feedback was from Yiftah Shahar: > > > > "I think that our requirements should be that even > when you are working > > with LMC>0 then the base LID routing should not be > affected. > > > One way to achieve this goal is to first run the > base-LID routing (so > > all base LID improvement will be also in LMC>0) and > then start with the > > other LIDs as round-robbing starting from the base- > lid-port + 1 > > according current routing algorithm rules (keeping > min-hop, up/down...)." > > > > We had some discussion with Al and Yiftah about this > and considered that > > in addition to "pure" base lid paths preservation > (which is good thing by > > itself) proposed method solves original lid > disbalancing problem as well. > > Would you elaborate on the motivation behind the > requirement to > maintain/preserve the base LID routing ? > > -- Hal > > LMC > 0 is mainly used by MPI, all other protocols / > applications will use only one LID (PATH), That's today; what happens when more ULPs are able to take advantage of alternate LIDs ? -- Hal > therefore we should make sure that performance of these > protocols / applications will not be effected if fabric is > configured with LMC > 0 > > > I meant "affected" and not "effected" of course > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi- > bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > From sashak at voltaire.com Thu Jun 12 04:46:18 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 14:46:18 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612114618.GE6256@sashak.voltaire.com> On 04:45 Thu 12 Jun , Hal Rosenstock wrote: > > > > LMC > 0 is mainly used by MPI, all other protocols / > > applications will use only one LID (PATH), > > That's today; what happens when more ULPs are able to take advantage of > alternate LIDs ? Will be even better since those LIDs will be better balanced now. Sasha From hrosenstock at xsigo.com Thu Jun 12 04:49:55 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 04:49:55 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <20080612113355.GB6256@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <20080612113355.GB6256@sashak.voltaire.com> Message-ID: <1213271395.14047.463.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 14:33 +0300, Sasha Khapyorsky wrote: > On 03:59 Thu 12 Jun , Hal Rosenstock wrote: > > > > Would you elaborate on the motivation behind the requirement to > > maintain/preserve the base LID routing ? > > I see couple advantages: > > 1. Application which works on base LIDs only will not be affected by LMC Yes; that's the obvious one. > 2. Changing LMC on a fabric will not change routing paths (when LIDs are > not reassigned) That's when LMC is reduced rather than increased. > 3. Finally it does better balancing for secondary LIDs ("port > offsetting") Isn't that accomodated in the patch but separate from the base LID preservation ? Preserving/maintaining base LIDs is a policy decision and perhaps this should be an option with this as the default. For some balancing all paths might be more important that not disrupting base LID traffic. -- Hal > Sasha From hrosenstock at xsigo.com Thu Jun 12 04:50:51 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 04:50:51 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <20080612114618.GE6256@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> <20080612114618.GE6256@sashak.voltaire.com> Message-ID: <1213271452.14047.464.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 14:46 +0300, Sasha Khapyorsky wrote: > On 04:45 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > LMC > 0 is mainly used by MPI, all other protocols / > > > applications will use only one LID (PATH), > > > > That's today; what happens when more ULPs are able to take advantage of > > alternate LIDs ? > > Will be even better since those LIDs will be better balanced now. Right but why is OK to disrupt non base LID traffic but not base LID traffic ? Is it only because it's only being used by one ULP currrently ? -- Hal > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ogerlitz at voltaire.com Thu Jun 12 05:03:21 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Jun 2008 15:03:21 +0300 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> Message-ID: <48511089.8050609@voltaire.com> Roland Dreier wrote: > Also any thoughts on how an app/ULP can sanely choose which vector to use? For apps that use verbs directly this might be somehow simple, for example a file/block I/O target that has a receiver process/thread per core/cpu and this process is shielded (eg through CPU affinity mechanisms) to a specific cpu. This process gives the index of the CPU it runs on as an input to the CQ creation verb, done. Or. From sashak at voltaire.com Thu Jun 12 05:11:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:11:02 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213271395.14047.463.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <20080612113355.GB6256@sashak.voltaire.com> <1213271395.14047.463.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612121102.GF6256@sashak.voltaire.com> On 04:49 Thu 12 Jun , Hal Rosenstock wrote: > On Thu, 2008-06-12 at 14:33 +0300, Sasha Khapyorsky wrote: > > On 03:59 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > Would you elaborate on the motivation behind the requirement to > > > maintain/preserve the base LID routing ? > > > > I see couple advantages: > > > > 1. Application which works on base LIDs only will not be affected by LMC > > Yes; that's the obvious one. > > > 2. Changing LMC on a fabric will not change routing paths (when LIDs are > > not reassigned) > > That's when LMC is reduced rather than increased. Also when increased and LIDs are not reassigned. > > 3. Finally it does better balancing for secondary LIDs ("port > > offsetting") > > Isn't that accomodated in the patch but separate from the base LID > preservation ? It is integrated in the patch - balancing for each LID starts from its lower LID's port + 1. Not doing this would be really bad. > Preserving/maintaining base LIDs is a policy decision and perhaps this > should be an option with this as the default. For some balancing all > paths might be more important that not disrupting base LID traffic. This is the trick - by preserving base LID traffic and offsetting over other LID ports we get better than before balancing. So right now it is hard to me to see when proposed option would be useful. But probably it would, I think we can add it then. Sasha From sashak at voltaire.com Thu Jun 12 05:16:03 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:16:03 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213271452.14047.464.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> <20080612114618.GE6256@sashak.voltaire.com> <1213271452.14047.464.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612121603.GG6256@sashak.voltaire.com> On 04:50 Thu 12 Jun , Hal Rosenstock wrote: > On Thu, 2008-06-12 at 14:46 +0300, Sasha Khapyorsky wrote: > > On 04:45 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > > > LMC > 0 is mainly used by MPI, all other protocols / > > > > applications will use only one LID (PATH), > > > > > > That's today; what happens when more ULPs are able to take advantage of > > > alternate LIDs ? > > > > Will be even better since those LIDs will be better balanced now. > > Right but why is OK to disrupt non base LID traffic but not base LID > traffic ? Is it only because it's only being used by one ULP > currrently ? How this disrupt non base LID traffic more than it was now? Potentially it could be affected by only base LIDs balancing it is against all LIDs balancing today. Sasha From sashak at voltaire.com Thu Jun 12 05:19:27 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:19:27 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] OpenSM/include/osm_port_profile.h: Fix some typos In-Reply-To: <1213213654.14047.394.camel@hrosenstock-ws.xsigo.com> References: <1213213654.14047.394.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612121927.GI6256@sashak.voltaire.com> On 12:47 Wed 11 Jun , Hal Rosenstock wrote: > OpenSM/include/osm_port_profile.h: Fix some typos > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 12 05:18:58 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:18:58 +0300 Subject: [ofa-general] Re: [PATCHv2][TRIVIAL] OpenSM/include/opensm: Fix some commentary typos In-Reply-To: <1213208837.14047.377.camel@hrosenstock-ws.xsigo.com> References: <1213208837.14047.377.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612121858.GH6256@sashak.voltaire.com> On 11:27 Wed 11 Jun , Hal Rosenstock wrote: > OpenSM/include/opensm: Fix some commentary typos > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 12 05:20:37 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:20:37 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] OpenSM/libvendor/osm_vendor_ibumad_sa.c: Eliminate unneeded define In-Reply-To: <1213206869.14047.362.camel@hrosenstock-ws.xsigo.com> References: <1213206869.14047.362.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612122037.GJ6256@sashak.voltaire.com> On 10:54 Wed 11 Jun , Hal Rosenstock wrote: > OpenSM/libvendor/osm_vendor_ibumad_sa.c: Eliminate unneeded define > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 12 05:22:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:22:02 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_lin_fwd_tbl.c: Minor change to __osm_lin_tbl_compute_obj_size In-Reply-To: <1213199739.14047.347.camel@hrosenstock-ws.xsigo.com> References: <484FE673.4060406@bull.net> <1213198837.14047.339.camel@hrosenstock-ws.xsigo.com> <1213199739.14047.347.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612122202.GK6256@sashak.voltaire.com> On 08:55 Wed 11 Jun , Hal Rosenstock wrote: > opensm/osm_lin_fwd_tbl.c: Minor change to __osm_lin_tbl_compute_obj_size > > Change input parameter to be uint16_t r.t. uint32_6 > Also, input parameter is really num_lids and not num_ports > > Based on comments from Vincent Ficet > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 12 05:28:01 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:28:01 +0300 Subject: [ofa-general] Re: [PATCH] ibsim: Remove some unused routines In-Reply-To: <1213133974.14047.298.camel@hrosenstock-ws.xsigo.com> References: <1213133974.14047.298.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612122801.GL6256@sashak.voltaire.com> On 14:39 Tue 10 Jun , Hal Rosenstock wrote: > ibsim: Remove some unused routines > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Jun 12 05:33:56 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:33:56 +0300 Subject: [ofa-general] Re: [PATCH] ibsim/sim.h: Allow max ports based on 36 port switches In-Reply-To: <1213213339.14047.390.camel@hrosenstock-ws.xsigo.com> References: <1213213339.14047.390.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612123356.GM6256@sashak.voltaire.com> On 12:42 Wed 11 Jun , Hal Rosenstock wrote: > ibsim/sim.h: Allow max ports based on 36 port switches > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From olga.shern at gmail.com Thu Jun 12 04:21:04 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Thu, 12 Jun 2008 14:21:04 +0300 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> Message-ID: On 6/12/08, Hal Rosenstock wrote: > > On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: > > Basically this addresses the problem described by Al Chu in: > > > > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html > > > > When base lid paths become completely disbalanced on a fabrics with > > lmc > 0. > > > > One feedback was from Yiftah Shahar: > > > > "I think that our requirements should be that even when you are working > > with LMC>0 then the base LID routing should not be affected. > > > One way to achieve this goal is to first run the base-LID routing (so > > all base LID improvement will be also in LMC>0) and then start with the > > other LIDs as round-robbing starting from the base-lid-port + 1 > > according current routing algorithm rules (keeping min-hop, up/down...)." > > > > We had some discussion with Al and Yiftah about this and considered that > > in addition to "pure" base lid paths preservation (which is good thing by > > itself) proposed method solves original lid disbalancing problem as well. > > Would you elaborate on the motivation behind the requirement to > maintain/preserve the base LID routing ? > > -- Hal LMC > 0 is mainly used by MPI, all other protocols / applications will use only one LID (PATH), therefore we should make sure that performance of these protocols / applications will not be effected if fabric is configured with LMC > 0 _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olga.shern at gmail.com Thu Jun 12 04:22:33 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Thu, 12 Jun 2008 14:22:33 +0300 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> Message-ID: On 6/12/08, Olga Shern (Voltaire) wrote: > > > > On 6/12/08, Hal Rosenstock wrote: >> >> On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: >> > Basically this addresses the problem described by Al Chu in: >> > >> > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html >> > >> > When base lid paths become completely disbalanced on a fabrics with >> > lmc > 0. >> > >> > One feedback was from Yiftah Shahar: >> > >> > "I think that our requirements should be that even when you are working >> > with LMC>0 then the base LID routing should not be affected. >> >> > One way to achieve this goal is to first run the base-LID routing (so >> > all base LID improvement will be also in LMC>0) and then start with the >> > other LIDs as round-robbing starting from the base-lid-port + 1 >> > according current routing algorithm rules (keeping min-hop, >> up/down...)." >> > >> > We had some discussion with Al and Yiftah about this and considered that >> > in addition to "pure" base lid paths preservation (which is good thing >> by >> > itself) proposed method solves original lid disbalancing problem as >> well. >> >> Would you elaborate on the motivation behind the requirement to >> maintain/preserve the base LID routing ? >> >> -- Hal > > > LMC > 0 is mainly used by MPI, all other protocols / applications will use > only one LID (PATH), therefore we should make sure that performance of > these protocols / applications will not be effected if fabric is configured > with LMC > 0 > I meant "affected" and not "effected" of course _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tzachid at mellanox.co.il Thu Jun 12 05:46:05 2008 From: tzachid at mellanox.co.il (Tzachi Dar) Date: Thu, 12 Jun 2008 15:46:05 +0300 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com> References: <484FE673.4060406@bull.net><20080612100051.GI14494@sashak.voltaire.com><1213265829.14047.409.camel@hrosenstock-ws.xsigo.com><6C2C79E72C305246B504CBA17B5500C9043B16F5@mtlexch01.mtl.com> <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com> Message-ID: <6C2C79E72C305246B504CBA17B5500C9043B1855@mtlexch01.mtl.com> The surrounded can be bigger than the actual array (for example a whole file). Thanks Tzachi > -----Original Message----- > From: ofw-bounces at lists.openfabrics.org > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Hal Rosenstock > Sent: Thursday, June 12, 2008 2:43 PM > To: Leonid Keller > Cc: ofw at lists.openfabrics.org; Sasha Khapyorsky; Vincent > Ficet; general at lists.openfabrics.org > Subject: RE: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl > > On Thu, 2008-06-12 at 13:50 +0300, Leonid Keller wrote: > > One can use zero-sized arrays as a structure member, having > disabled > > the warning 4200. > > Here is a real example from hw\mlx4\kernel\bus\core\cache.c: > > > > #pragma warning( disable : 4200) > > struct ib_pkey_cache { > > int table_len; > > __be16 table[0]; > > }; > > #pragma warning( default : 4200) > > So to use this with Windows, any zero size array needs to be > surrounded by the pragma pair ? > > -- Hal > > > > -----Original Message----- > > > From: ofw-bounces at lists.openfabrics.org > > > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Hal > > > Rosenstock > > > Sent: Thursday, June 12, 2008 1:17 PM > > > To: Sasha Khapyorsky > > > Cc: ofw at lists.openfabrics.org; Vincent Ficet; > > > general at lists.openfabrics.org > > > Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl > > > > > > On Thu, 2008-06-12 at 13:00 +0300, Sasha Khapyorsky wrote: > > > > On 16:51 Wed 11 Jun , Vincent Ficet wrote: > > > > > > > > > > Also, could anyone explain why the osm_lin_fwd_tbl_t is > > > declared as > > > > > follows: > > > > > > > > > > typedef struct _osm_lin_fwd_tbl { > > > > > uint16_t size; > > > > > uint8_t port_tbl[1]; > > > > > } osm_lin_fwd_tbl_t; > > > > > > > > AFAIR the reason is a poor windows compiler which > doesn't support > > > > array[0] like definitions in structures. > > > > > > Yes, I too recall something being said along those lines quite a > > > while ago. > > > > > > Is this accurate ? If so, is this still a restriction of > the Windows > > > compilers in use (as I think the build environment and > perhaps the > > > compilers have changed quite a bit since > > > then) ? Can someone knowledgeable in Windows comment on this ? > > > > > > -- Hal > > > > > > > Sasha > > > > _______________________________________________ > > > > general mailing list > > > > general at lists.openfabrics.org > > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > > > To unsubscribe, please visit > > > > http://openib.org/mailman/listinfo/openib-general > > > > > > _______________________________________________ > > > ofw mailing list > > > ofw at lists.openfabrics.org > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > > > > > _______________________________________________ > ofw mailing list > ofw at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > -- Tzachi Dar Mellanox Technologies LTD. SW- Windows Phone: +972 (4) 909 7200 (ext 271) Mobile: +972 (57) 741 1269 E-mail: tzachid at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From sashak at voltaire.com Thu Jun 12 05:50:41 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 15:50:41 +0300 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <6C2C79E72C305246B504CBA17B5500C9043B1855@mtlexch01.mtl.com> References: <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com> <6C2C79E72C305246B504CBA17B5500C9043B1855@mtlexch01.mtl.com> Message-ID: <20080612125041.GO6256@sashak.voltaire.com> On 15:46 Thu 12 Jun , Tzachi Dar wrote: > The surrounded can be bigger than the actual array (for example a whole > file). And what about using compiler flags and not touching the code itself? I guess that if the code is supposed to be portable it should not have stuff like '#pragma warning( disable : 4200)'. Sasha From kliteyn at dev.mellanox.co.il Thu Jun 12 05:58:42 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 12 Jun 2008 15:58:42 +0300 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> Message-ID: <48511D82.1020203@dev.mellanox.co.il> Hal Rosenstock wrote: > On Thu, 2008-06-12 at 14:08 +0300, Olga Shern (Voltaire) wrote: >> >> On 6/12/08, Hal Rosenstock wrote: >> Hi Olga, >> >> On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: >> > Hi All, >> > >> > >> > >> > We have found something that seems like Infiniband Spec >> hole, >> >> What's the spec hole ? >> >> According to the Infiniband spec - partial member cannot "talk" with >> partial member only with full member. >> Therefore if partial member sending MC packet - all other partial >> members of this partition will generate BAD PKEY trap. >> It means that the behavior that we see is according to Infiniband >> Spec - but very problematic > > Originally, multicast groups were all full member only and more recently > was this extended to allow partial members and this was missed. A > comment should be filed against the spec on this. > >> > This issue is system issue that prevents from partial P_Key >> setup to >> > go into production. >> >> Indeed :-( >> >> > Short Setup & test description: >> > ------------------------------------------ >> > * Node A: P_Key XXX (full member) >> > * Node B, C, D, E, F: P_Key XXx (partial member) >> > >> > 1. Send ping from B -> A : ping is OK >> > 2. Send ping from C -> A : ping is OK >> > 3. Send ping from B -> C : no ping also OK >> > * Get traps Bad P_Key in SM - from all HCA in the fabric >> both for >> > test 1 & 2 (one time) and also for test 3 (all the time). > > What does all the time mean ? Does this mean with one test 3 ping, the > traps are repeated ? If so, at what rate ? Also, why do the HCAs issue these traps? Is the pkey enforcement on switch external ports is off? AFAIK, by default, OpenSM should configure pkeys on switch ports that are connected to these HCAs, so that partial member wouldn't get packet from another partial member. -- Yevgeny >> > Probably the ARP request that is MC traffic generate the >> trap in HCA, >> > for test 1 >> > & 2 we have only one ARP but for test 3 we send ARP all the >> time >> > because >> > we do not get any ARP reply. >> > >> > * The trap number SM get is 257 (HCA trap) if we will do >> P_Key >> > switch enforcement we will probably get 259 >> >> Is this with OpenSM or VSM ? >> >> We tested it with Voltaire SM but it should behave the same with >> OpenSM. > > That's likely but I'm not sure yet. > >> -- Hal >> >> > * We get trap also from the originator of the MC traffic >> even >> > though that receive switch relay error counter is increased >> (when out >> > port==in port), the switch does not drop the packet ? > > The implementation of that counter is broken and occurs "normally". The > increment of this counter is relatively meaningless :-( > >> > Additional questions/issues: >> > * Do we have a way to suppress port traps from SMA ?? i.e. >> that >> > the port will not generate traps that can "kill the SM" - as >> its look >> > this is bug in the spec where we can't send any mc traffic >> (even ARP) >> > when we have partial members and we do not have a way to >> suppress the >> > traps. > > All the SM can do is TrapRepress. > >> > * What will happen in the HCA when we get many traps (mc >> packets >> > from many nodes) and they need to keep all events until SM >> will >> > acknowledge? - Is there limitation in the number of on- >> going >> > traps (any HCA specific issues)? > > Assuming you mean events from which traps are generated, I think this is > left as an implementation dependent detail in terms of the spec. An > implementation needs to take care not to lose certain events; others > like this aren't critical but that's left to the specific SMA > implementation. > > -- Hal > >> > >> > >> > >> > >> > Best Regards >> > >> > Olga >> > >> > >> > _______________________________________________ >> > general mailing list >> > general at lists.openfabrics.org >> > http://lists.openfabrics.org/cgi- >> bin/mailman/listinfo/general >> > >> > To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hrosenstock at xsigo.com Thu Jun 12 06:03:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:03:29 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <20080612121102.GF6256@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <20080612113355.GB6256@sashak.voltaire.com> <1213271395.14047.463.camel@hrosenstock-ws.xsigo.com> <20080612121102.GF6256@sashak.voltaire.com> Message-ID: <1213275809.14047.479.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 15:11 +0300, Sasha Khapyorsky wrote: > On 04:49 Thu 12 Jun , Hal Rosenstock wrote: > > On Thu, 2008-06-12 at 14:33 +0300, Sasha Khapyorsky wrote: > > > On 03:59 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > > > Would you elaborate on the motivation behind the requirement to > > > > maintain/preserve the base LID routing ? > > > > > > I see couple advantages: > > > > > > 1. Application which works on base LIDs only will not be affected by LMC > > > > Yes; that's the obvious one. > > > > > 2. Changing LMC on a fabric will not change routing paths (when LIDs are > > > not reassigned) > > > > That's when LMC is reduced rather than increased. > Also when increased and LIDs are not reassigned. Is that possible for anything other than the last base LID assigned ? > > > 3. Finally it does better balancing for secondary LIDs ("port > > > offsetting") > > > > Isn't that accomodated in the patch but separate from the base LID > > preservation ? > > It is integrated in the patch - balancing for each LID starts from > its lower LID's port + 1. Understood (mostly) with comment below. > Not doing this would be really bad. Is the badness disrupting the base LID traffic or something else ? > > Preserving/maintaining base LIDs is a policy decision and perhaps this > > should be an option with this as the default. For some balancing all > > paths might be more important that not disrupting base LID traffic. > > This is the trick - by preserving base LID traffic and offsetting over > other LID ports we get better than before balancing. So right now it is > hard to me to see when proposed option would be useful. When all ULPs use all LIDs "equally" and it's not just MPI ? > But probably it would, I think we can add it then. Sure; this can be viewed a future thing. -- Hal > Sasha From hrosenstock at xsigo.com Thu Jun 12 06:05:54 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:05:54 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <20080612121603.GG6256@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> <20080612114618.GE6256@sashak.voltaire.com> <1213271452.14047.464.camel@hrosenstock-ws.xsigo.com> <20080612121603.GG6256@sashak.voltaire.com> Message-ID: <1213275954.14047.483.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 15:16 +0300, Sasha Khapyorsky wrote: > On 04:50 Thu 12 Jun , Hal Rosenstock wrote: > > On Thu, 2008-06-12 at 14:46 +0300, Sasha Khapyorsky wrote: > > > On 04:45 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > > > > > LMC > 0 is mainly used by MPI, all other protocols / > > > > > applications will use only one LID (PATH), > > > > > > > > That's today; what happens when more ULPs are able to take advantage of > > > > alternate LIDs ? > > > > > > Will be even better since those LIDs will be better balanced now. > > > > Right but why is OK to disrupt non base LID traffic but not base LID > > traffic ? Is it only because it's only being used by one ULP > > currrently ? > > How this disrupt non base LID traffic more than it was now? I didn't say that. I was asking why it's OK for non base LID traffic but not base LID traffic. Isn't it that's it's many ULPs rather than just MPI ? > Potentially it could be affected by only base LIDs balancing it is against all LIDs > balancing today. I can't parse that sentence. -- Hal > Sasha From hrosenstock at xsigo.com Thu Jun 12 06:08:50 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:08:50 -0700 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <48511D82.1020203@dev.mellanox.co.il> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> <48511D82.1020203@dev.mellanox.co.il> Message-ID: <1213276130.14047.485.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 15:58 +0300, Yevgeny Kliteynik wrote: > Hal Rosenstock wrote: > > On Thu, 2008-06-12 at 14:08 +0300, Olga Shern (Voltaire) wrote: > >> > >> On 6/12/08, Hal Rosenstock wrote: > >> Hi Olga, > >> > >> On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > >> > Hi All, > >> > > >> > > >> > > >> > We have found something that seems like Infiniband Spec > >> hole, > >> > >> What's the spec hole ? > >> > >> According to the Infiniband spec - partial member cannot "talk" with > >> partial member only with full member. > >> Therefore if partial member sending MC packet - all other partial > >> members of this partition will generate BAD PKEY trap. > >> It means that the behavior that we see is according to Infiniband > >> Spec - but very problematic > > > > Originally, multicast groups were all full member only and more recently > > was this extended to allow partial members and this was missed. A > > comment should be filed against the spec on this. > > > >> > This issue is system issue that prevents from partial P_Key > >> setup to > >> > go into production. > >> > >> Indeed :-( > >> > >> > Short Setup & test description: > >> > ------------------------------------------ > >> > * Node A: P_Key XXX (full member) > >> > * Node B, C, D, E, F: P_Key XXx (partial member) > >> > > >> > 1. Send ping from B -> A : ping is OK > >> > 2. Send ping from C -> A : ping is OK > >> > 3. Send ping from B -> C : no ping also OK > >> > * Get traps Bad P_Key in SM - from all HCA in the fabric > >> both for > >> > test 1 & 2 (one time) and also for test 3 (all the time). > > > > What does all the time mean ? Does this mean with one test 3 ping, the > > traps are repeated ? If so, at what rate ? > > Also, why do the HCAs issue these traps? Is the pkey enforcement > on switch external ports is off? I presume so but there was a claim about what would happen if ingress/egress filtering were on (about getting the switch rather than end port bad PKey traps). > AFAIK, by default, OpenSM should > configure pkeys on switch ports that are connected to these HCAs, > so that partial member wouldn't get packet from another partial > member. It was done using VSM not OpenSM. -- Hal > -- Yevgeny > > >> > Probably the ARP request that is MC traffic generate the > >> trap in HCA, > >> > for test 1 > >> > & 2 we have only one ARP but for test 3 we send ARP all the > >> time > >> > because > >> > we do not get any ARP reply. > >> > > >> > * The trap number SM get is 257 (HCA trap) if we will do > >> P_Key > >> > switch enforcement we will probably get 259 > >> > >> Is this with OpenSM or VSM ? > >> > >> We tested it with Voltaire SM but it should behave the same with > >> OpenSM. > > > > That's likely but I'm not sure yet. > > > >> -- Hal > >> > >> > * We get trap also from the originator of the MC traffic > >> even > >> > though that receive switch relay error counter is increased > >> (when out > >> > port==in port), the switch does not drop the packet ? > > > > The implementation of that counter is broken and occurs "normally". The > > increment of this counter is relatively meaningless :-( > > > >> > Additional questions/issues: > >> > * Do we have a way to suppress port traps from SMA ?? i.e. > >> that > >> > the port will not generate traps that can "kill the SM" - as > >> its look > >> > this is bug in the spec where we can't send any mc traffic > >> (even ARP) > >> > when we have partial members and we do not have a way to > >> suppress the > >> > traps. > > > > All the SM can do is TrapRepress. > > > >> > * What will happen in the HCA when we get many traps (mc > >> packets > >> > from many nodes) and they need to keep all events until SM > >> will > >> > acknowledge? - Is there limitation in the number of on- > >> going > >> > traps (any HCA specific issues)? > > > > Assuming you mean events from which traps are generated, I think this is > > left as an implementation dependent detail in terms of the spec. An > > implementation needs to take care not to lose certain events; others > > like this aren't critical but that's left to the specific SMA > > implementation. > > > > -- Hal > > > >> > > >> > > >> > > >> > > >> > Best Regards > >> > > >> > Olga > >> > > >> > > >> > _______________________________________________ > >> > general mailing list > >> > general at lists.openfabrics.org > >> > http://lists.openfabrics.org/cgi- > >> bin/mailman/listinfo/general > >> > > >> > To unsubscribe, please visit > >> http://openib.org/mailman/listinfo/openib-general > >> > >> _______________________________________________ > >> general mailing list > >> general at lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >> > >> To unsubscribe, please visit > >> http://openib.org/mailman/listinfo/openib-general > >> > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > From olga.shern at gmail.com Thu Jun 12 06:31:42 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Thu, 12 Jun 2008 16:31:42 +0300 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> Message-ID: On 6/12/08, Hal Rosenstock wrote: > > On Thu, 2008-06-12 at 14:08 +0300, Olga Shern (Voltaire) wrote: > > > > > > On 6/12/08, Hal Rosenstock wrote: > > Hi Olga, > > > > On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > > > Hi All, > > > > > > > > > > > > We have found something that seems like Infiniband Spec > > hole, > > > > What's the spec hole ? > > > > According to the Infiniband spec - partial member cannot "talk" with > > partial member only with full member. > > Therefore if partial member sending MC packet - all other partial > > members of this partition will generate BAD PKEY trap. > > It means that the behavior that we see is according to Infiniband > > Spec - but very problematic > > Originally, multicast groups were all full member only and more recently > was this extended to allow partial members and this was missed. A > comment should be filed against the spec on this. > > > > This issue is system issue that prevents from partial P_Key > > setup to > > > go into production. > > > > Indeed :-( > > > > > Short Setup & test description: > > > ------------------------------------------ > > > * Node A: P_Key XXX (full member) > > > * Node B, C, D, E, F: P_Key XXx (partial member) > > > > > > 1. Send ping from B -> A : ping is OK > > > 2. Send ping from C -> A : ping is OK > > > 3. Send ping from B -> C : no ping also OK > > > * Get traps Bad P_Key in SM - from all HCA in the fabric > > both for > > > test 1 & 2 (one time) and also for test 3 (all the time). > > What does all the time mean ? Does this mean with one test 3 ping, the > traps are repeated ? If so, at what rate ? every ping will generate ARP that will generate BAD PKEY trap > > Probably the ARP request that is MC traffic generate the > > trap in HCA, > > > for test 1 > > > & 2 we have only one ARP but for test 3 we send ARP all the > > time > > > because > > > we do not get any ARP reply. > > > > > > * The trap number SM get is 257 (HCA trap) if we will do > > P_Key > > > switch enforcement we will probably get 259 > > > > Is this with OpenSM or VSM ? > > > > We tested it with Voltaire SM but it should behave the same with > > OpenSM. > > That's likely but I'm not sure yet. > > > -- Hal > > > > > * We get trap also from the originator of the MC traffic > > even > > > though that receive switch relay error counter is increased > > (when out > > > port==in port), the switch does not drop the packet ? > > The implementation of that counter is broken and occurs "normally". The > increment of this counter is relatively meaningless :-( > > > > Additional questions/issues: > > > * Do we have a way to suppress port traps from SMA ?? i.e. > > that > > > the port will not generate traps that can "kill the SM" - as > > its look > > > this is bug in the spec where we can't send any mc traffic > > (even ARP) > > > when we have partial members and we do not have a way to > > suppress the > > > traps. > > All the SM can do is TrapRepress. > > > > * What will happen in the HCA when we get many traps (mc > > packets > > > from many nodes) and they need to keep all events until SM > > will > > > acknowledge? - Is there limitation in the number of on- > > going > > > traps (any HCA specific issues)? > > Assuming you mean events from which traps are generated, I think this is > left as an implementation dependent detail in terms of the spec. An > implementation needs to take care not to lose certain events; others > like this aren't critical but that's left to the specific SMA > implementation. > > -- Hal > > > > > > > > > > > > > > > > Best Regards > > > > > > Olga > > > > > > > > > _______________________________________________ > > > general mailing list > > > general at lists.openfabrics.org > > > http://lists.openfabrics.org/cgi- > > bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Thu Jun 12 06:35:56 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:35:56 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_sa_path_record.c: Break up some long OSM_LOG lines Message-ID: <1213277757.14047.494.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_path_record.c: Break up some long OSM_LOG lines Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index b2ce54d..c468ba2 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -1504,7 +1504,8 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, status = osm_get_mgrp_by_mgid(sa, &p_pr->dgid, pp_mgrp); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F09: " - "No MC group found for PathRecord destination GID 0x%016" PRIx64 " : " "0x%016" PRIx64 "\n", + "No MC group found for PathRecord destination " + "GID 0x%016" PRIx64 " : " "0x%016" PRIx64 "\n", cl_ntoh64(p_pr->dgid.unicast.prefix), cl_ntoh64(p_pr->dgid.unicast.interface_id)); goto Exit; @@ -1518,7 +1519,9 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, if ((*pp_mgrp)->mlid != p_pr->dlid) { /* Note: perhaps this might be better indicated as an invalid request */ OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F10: " - "MC group MLID 0x%x does not match PathRecord destination LID 0x%x\n", (*pp_mgrp)->mlid, p_pr->dlid); + "MC group MLID 0x%x does not match " + "PathRecord destination LID 0x%x\n", + (*pp_mgrp)->mlid, p_pr->dlid); *pp_mgrp = NULL; goto Exit; } @@ -1526,7 +1529,8 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, *pp_mgrp = __get_mgrp_by_mlid(sa, p_pr->dlid); if (*pp_mgrp == NULL) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F11: " - "No MC group found for PathRecord destination LID 0x%x\n", p_pr->dlid); + "No MC group found for PathRecord " + "destination LID 0x%x\n", p_pr->dlid); } } From hrosenstock at xsigo.com Thu Jun 12 06:35:58 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:35:58 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_trap_rcv.c: Break up some long comment lines Message-ID: <1213277758.14047.495.camel@hrosenstock-ws.xsigo.com> opensm/osm_trap_rcv.c: Break up some long comment lines Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index c562e9a..8210e7f 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -262,8 +262,8 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, if (osm_exit_flag) /* We got an exit flag - do nothing - Otherwise we start a sweep on the trap 144 caused by cleaning up - SM Cap bit... + Otherwise we start a sweep on the trap 144 caused by + cleaning up SM Cap bit... */ goto Exit; @@ -272,9 +272,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, p_madw->p_mad->mgmt_class == IB_MCLASS_SUBN_DIR) is_gsi = FALSE; - /* - No real need to grab the lock for this function. - */ + /* No real need to grab the lock for this function. */ memset(payload, 0, sizeof(payload)); memset(&tmp_madw, 0, sizeof(tmp_madw)); @@ -303,9 +301,9 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, */ if (p_madw->mad_addr.addr_type.smi.source_lid == 0) { - /* Check if the sm_base_lid is 0. If yes - this means that - the local lid wasn't configured yet. Don't send a response - to the trap. */ + /* Check if the sm_base_lid is 0. If yes - this means + that the local lid wasn't configured yet. Don't send + a response to the trap. */ if (sm->p_subn->sm_base_lid == 0) { OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Received SLID=0 Trap with local LID=0. Ignoring MAD\n"); From hrosenstock at xsigo.com Thu Jun 12 06:42:14 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:42:14 -0700 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213278134.14047.498.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 16:31 +0300, Olga Shern (Voltaire) wrote: > > > On 6/12/08, Hal Rosenstock wrote: > On Thu, 2008-06-12 at 14:08 +0300, Olga Shern (Voltaire) > wrote: > > > > > > On 6/12/08, Hal Rosenstock wrote: > > Hi Olga, > > > > On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > > > Hi All, > > > > > > > > > > > > We have found something that seems like Infiniband > Spec > > hole, > > > > What's the spec hole ? > > > > According to the Infiniband spec - partial member cannot > "talk" with > > partial member only with full member. > > Therefore if partial member sending MC packet - all other > partial > > members of this partition will generate BAD PKEY trap. > > It means that the behavior that we see is according to > Infiniband > > Spec - but very problematic > > Originally, multicast groups were all full member only and > more recently > was this extended to allow partial members and this was > missed. A > comment should be filed against the spec on this. > > > > This issue is system issue that prevents from > partial P_Key > > setup to > > > go into production. > > > > Indeed :-( > > > > > Short Setup & test description: > > > ------------------------------------------ > > > * Node A: P_Key XXX (full member) > > > * Node B, C, D, E, F: P_Key XXx (partial member) > > > > > > 1. Send ping from B -> A : ping is OK > > > 2. Send ping from C -> A : ping is OK > > > 3. Send ping from B -> C : no ping also OK > > > * Get traps Bad P_Key in SM - from all HCA in the > fabric > > both for > > > test 1 & 2 (one time) and also for test 3 (all the > time). > > What does all the time mean ? Does this mean with one test 3 > ping, the > traps are repeated ? If so, at what rate ? > > every ping will generate ARP that will generate BAD PKEY trap OK; so what do you mean by one time v. all the time ? Is that really the case ? > > > Probably the ARP request that is MC traffic > generate the > > trap in HCA, > > > for test 1 > > > & 2 we have only one ARP but for test 3 we send > ARP all the > > time > > > because > > > we do not get any ARP reply. > > > > > > * The trap number SM get is 257 (HCA trap) if we > will do > > P_Key > > > switch enforcement we will probably get 259 > > > > Is this with OpenSM or VSM ? > > > > We tested it with Voltaire SM but it should behave the same > with > > OpenSM. > > That's likely but I'm not sure yet. Would you try this with OpenSM (and validate your theory about getting switch bad PKey traps v. end port bad PKey traps) or does VSM have such a mode (ingress/egress partition filtering) ? -- Hal > > -- Hal > > > > > * We get trap also from the originator of the MC > traffic > > even > > > though that receive switch relay error counter is > increased > > (when out > > > port==in port), the switch does not drop the > packet ? > > The implementation of that counter is broken and occurs > "normally". The > increment of this counter is relatively meaningless :-( > > > > Additional questions/issues: > > > * Do we have a way to suppress port traps from > SMA ?? i.e. > > that > > > the port will not generate traps that can "kill > the SM" - as > > its look > > > this is bug in the spec where we can't send any mc > traffic > > (even ARP) > > > when we have partial members and we do not have a > way to > > suppress the > > > traps. > > All the SM can do is TrapRepress. > > > > * What will happen in the HCA when we get many > traps (mc > > packets > > > from many nodes) and they need to keep all events > until SM > > will > > > acknowledge? - Is there limitation in the number > of on- > > going > > > traps (any HCA specific issues)? > > Assuming you mean events from which traps are generated, I > think this is > left as an implementation dependent detail in terms of the > spec. An > implementation needs to take care not to lose certain events; > others > like this aren't critical but that's left to the specific SMA > implementation. > > -- Hal > > > > > > > > > > > > > > > > Best Regards > > > > > > Olga > > > > > > > > > _______________________________________________ > > > general mailing list > > > general at lists.openfabrics.org > > > http://lists.openfabrics.org/cgi- > > bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi- > bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > > From olga.shern at gmail.com Thu Jun 12 06:44:08 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Thu, 12 Jun 2008 16:44:08 +0300 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <1213278134.14047.498.camel@hrosenstock-ws.xsigo.com> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> <1213278134.14047.498.camel@hrosenstock-ws.xsigo.com> Message-ID: On 6/12/08, Hal Rosenstock wrote: > > On Thu, 2008-06-12 at 16:31 +0300, Olga Shern (Voltaire) wrote: > > > > > > On 6/12/08, Hal Rosenstock wrote: > > On Thu, 2008-06-12 at 14:08 +0300, Olga Shern (Voltaire) > > wrote: > > > > > > > > > On 6/12/08, Hal Rosenstock wrote: > > > Hi Olga, > > > > > > On Thu, 2008-06-12 at 09:46 +0300, Olga Shern wrote: > > > > Hi All, > > > > > > > > > > > > > > > > We have found something that seems like Infiniband > > Spec > > > hole, > > > > > > What's the spec hole ? > > > > > > According to the Infiniband spec - partial member cannot > > "talk" with > > > partial member only with full member. > > > Therefore if partial member sending MC packet - all other > > partial > > > members of this partition will generate BAD PKEY trap. > > > It means that the behavior that we see is according to > > Infiniband > > > Spec - but very problematic > > > > Originally, multicast groups were all full member only and > > more recently > > was this extended to allow partial members and this was > > missed. A > > comment should be filed against the spec on this. > > > > > > This issue is system issue that prevents from > > partial P_Key > > > setup to > > > > go into production. > > > > > > Indeed :-( > > > > > > > Short Setup & test description: > > > > ------------------------------------------ > > > > * Node A: P_Key XXX (full member) > > > > * Node B, C, D, E, F: P_Key XXx (partial member) > > > > > > > > 1. Send ping from B -> A : ping is OK > > > > 2. Send ping from C -> A : ping is OK > > > > 3. Send ping from B -> C : no ping also OK > > > > * Get traps Bad P_Key in SM - from all HCA in the > > fabric > > > both for > > > > test 1 & 2 (one time) and also for test 3 (all the > > time). > > > > What does all the time mean ? Does this mean with one test 3 > > ping, the > > traps are repeated ? If so, at what rate ? > > > > every ping will generate ARP that will generate BAD PKEY trap > > OK; so what do you mean by one time v. all the time ? Is that really the > case ? > > > > > Probably the ARP request that is MC traffic > > generate the > > > trap in HCA, > > > > for test 1 > > > > & 2 we have only one ARP but for test 3 we send > > ARP all the > > > time > > > > because > > > > we do not get any ARP reply. > > > > > > > > * The trap number SM get is 257 (HCA trap) if we > > will do > > > P_Key > > > > switch enforcement we will probably get 259 > > > > > > Is this with OpenSM or VSM ? > > > > > > We tested it with Voltaire SM but it should behave the same > > with > > > OpenSM. > > > > That's likely but I'm not sure yet. > > Would you try this with OpenSM (and validate your theory about getting > switch bad PKey traps v. end port bad PKey traps) or does VSM have such > a mode (ingress/egress partition filtering) ? > > -- Hal Yes, I will test it with OpenSM > > -- Hal > > > > > > > * We get trap also from the originator of the MC > > traffic > > > even > > > > though that receive switch relay error counter is > > increased > > > (when out > > > > port==in port), the switch does not drop the > > packet ? > > > > The implementation of that counter is broken and occurs > > "normally". The > > increment of this counter is relatively meaningless :-( > > > > > > Additional questions/issues: > > > > * Do we have a way to suppress port traps from > > SMA ?? i.e. > > > that > > > > the port will not generate traps that can "kill > > the SM" - as > > > its look > > > > this is bug in the spec where we can't send any mc > > traffic > > > (even ARP) > > > > when we have partial members and we do not have a > > way to > > > suppress the > > > > traps. > > > > All the SM can do is TrapRepress. > > > > > > * What will happen in the HCA when we get many > > traps (mc > > > packets > > > > from many nodes) and they need to keep all events > > until SM > > > will > > > > acknowledge? - Is there limitation in the number > > of on- > > > going > > > > traps (any HCA specific issues)? > > > > Assuming you mean events from which traps are generated, I > > think this is > > left as an implementation dependent detail in terms of the > > spec. An > > implementation needs to take care not to lose certain events; > > others > > like this aren't critical but that's left to the specific SMA > > implementation. > > > > -- Hal > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Olga > > > > > > > > > > > > _______________________________________________ > > > > general mailing list > > > > general at lists.openfabrics.org > > > > http://lists.openfabrics.org/cgi- > > > bin/mailman/listinfo/general > > > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > > > _______________________________________________ > > > general mailing list > > > general at lists.openfabrics.org > > > http://lists.openfabrics.org/cgi- > > bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Thu Jun 12 06:48:36 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 06:48:36 -0700 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <48511D82.1020203@dev.mellanox.co.il> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> <48511D82.1020203@dev.mellanox.co.il> Message-ID: <1213278516.14047.502.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 15:58 +0300, Yevgeny Kliteynik wrote: > AFAIK, by default, OpenSM should > configure pkeys on switch ports that are connected to these HCAs, > so that partial member wouldn't get packet from another partial > member. In one sense, this is a workaround. It still wastes the network bandwidth as it has to get all the way to the egress switch before dropping. So it's going to come down to how this scenario is decided to be dealt with in the spec and then in the implementations. -- Hal From roland.list at gmail.com Thu Jun 12 06:53:13 2008 From: roland.list at gmail.com (Roland Dreier) Date: Thu, 12 Jun 2008 06:53:13 -0700 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: <48511089.8050609@voltaire.com> References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> <48511089.8050609@voltaire.com> Message-ID: > For apps that use verbs directly this might be somehow simple, for example a > file/block I/O target that has a receiver process/thread per core/cpu and > this process is shielded (eg through CPU affinity mechanisms) to a specific > cpu. > > This process gives the index of the CPU it runs on as an input to the CQ > creation verb, done. ?? Nothing ties the CQ event vector number to a CPU number that I know of. From roland.list at gmail.com Thu Jun 12 06:55:55 2008 From: roland.list at gmail.com (Roland Dreier) Date: Thu, 12 Jun 2008 06:55:55 -0700 Subject: [ofa-general] unsing ibverbs 1.0 with rdmacm In-Reply-To: References: Message-ID: > This raises a question to this scenario: > 1) Commercial (binary) applications needs to support many > versions of OFED (ibverbs and rdmacm). > 2) ABI ibverbs 1.2 is released. > > Question: > What ibverbs ABI context will rdmacm ABI RDMACM_1.0 give > access to? Whatever ABI that librdmacm is linked against. This is a bit of a mess, but if you build your app against the same libibverbs that the librdmacm you link against, you should be fine. - R. From ogerlitz at voltaire.com Thu Jun 12 07:03:00 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Jun 2008 17:03:00 +0300 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> <48511089.8050609@voltaire.com> Message-ID: <48512C94.1040306@voltaire.com> Roland Dreier wrote: > ?? Nothing ties the CQ event vector number to a CPU number that I know of. > I am not sure to follow you. Do you want to say that the suggested implementation in this patch is such that the EQ is not tied to a specific CPU or that there is no proper mapping from the CPU this process is shielded on to the vector (=EQ) number this CQ would be associated with? Or. From kliteyn at dev.mellanox.co.il Thu Jun 12 07:22:31 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 12 Jun 2008 17:22:31 +0300 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: <1213278516.14047.502.camel@hrosenstock-ws.xsigo.com> References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> <48511D82.1020203@dev.mellanox.co.il> <1213278516.14047.502.camel@hrosenstock-ws.xsigo.com> Message-ID: <48513127.7050100@dev.mellanox.co.il> Hal Rosenstock wrote: > On Thu, 2008-06-12 at 15:58 +0300, Yevgeny Kliteynik wrote: >> AFAIK, by default, OpenSM should >> configure pkeys on switch ports that are connected to these HCAs, >> so that partial member wouldn't get packet from another partial >> member. > > In one sense, this is a workaround. It still wastes the network > bandwidth as it has to get all the way to the egress switch before > dropping. Right, I wasn't proposing solution, but trying to understand the setup. -- Yevgeny > So it's going to come down to how this scenario is decided to be dealt > with in the spec and then in the implementations. > > -- Hal > > From thomas.eggestad at gmail.com Thu Jun 12 07:51:33 2008 From: thomas.eggestad at gmail.com (Thomas Eggestad) Date: Thu, 12 Jun 2008 16:51:33 +0200 Subject: [ofa-general] unsing ibverbs 1.0 with rdmacm In-Reply-To: References: Message-ID: Thank you Ronald for taking the time. My comments are below On 6/12/08, Roland Dreier wrote: > > This raises a question to this scenario: > > 1) Commercial (binary) applications needs to support many > > versions of OFED (ibverbs and rdmacm). > > 2) ABI ibverbs 1.2 is released. > > > > Question: > > What ibverbs ABI context will rdmacm ABI RDMACM_1.0 give > > access to? > > > Whatever ABI that librdmacm is linked against. > > This is a bit of a mess, but if you build your app against the same > libibverbs that the librdmacm you link against, you should be fine. I do not believe this solves the scenario from my point of view. I was not clear about the fact that the app binary is not recompiled. The customer may be running older or newer OFED stack where the changes described in my original scenario may have happened. Do this work? If so there should be possible to work around the problem. We do not want or have the power to tell the customer what version of OFED that needs to be installed. Many of the customers do not run OFED at all. Regards, Thomas From amar.mudrankit at qlogic.com Thu Jun 12 08:13:40 2008 From: amar.mudrankit at qlogic.com (Amar Mudrankit) Date: Thu, 12 Jun 2008 20:43:40 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <484F751F.7030407@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> Message-ID: > >> +/sys/class/infiniband_qlgc_vnic/interfaces// >> + >> + vnic_state (0444) State of the VNIC interface. > > This I don't understand - you seem to be registering and > unregistering the net_device based some state machine, > in some cases even triggered by timers. > > Whats the idea behind this? > For a given VNIC interface, unless the VNIC host driver establishes a connection with EVIC and completes the control and data path communication sequence with the EVIC, the VNIC network device is not registered on the host. Establishing a connection with EVIC and going through the control and data path communication involves multiple packet exchange on the IB side with the EVIC and can take some time. The netpath statemachine is, hence, meant to PERIODICALLY check if connection is fully established with EVIC or not. Once connection is established, it takes care to register the VNIC netdevice. From kaber at trash.net Thu Jun 12 08:18:38 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 12 Jun 2008 17:18:38 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> Message-ID: <48513E4E.4040601@trash.net> Amar Mudrankit wrote: >>> +/sys/class/infiniband_qlgc_vnic/interfaces// >>> + >>> + vnic_state (0444) State of the VNIC interface. >> This I don't understand - you seem to be registering and >> unregistering the net_device based some state machine, >> in some cases even triggered by timers. >> >> Whats the idea behind this? > > For a given VNIC interface, unless the VNIC host driver establishes a > connection with EVIC > and completes the control and data path communication sequence with > the EVIC, the > VNIC network device is not registered on the host. Establishing a > connection with EVIC > and going through the control and data path communication involves > multiple packet > exchange on the IB side with the EVIC and can take some time. The > netpath statemachine is, hence, meant to PERIODICALLY check if > connection is fully established with EVIC or not. > Once connection is established, it takes care to register the VNIC netdevice. We have linkstate/operstate for this. How is a user supposed to configure the network device when it appears at a more or less random time from his perspective? From ramachandra.kuchimanchi at qlogic.com Thu Jun 12 08:21:31 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Thu, 12 Jun 2008 20:51:31 +0530 Subject: [ofa-general] [PATCH v4 00/14] QLogic VNIC Driver In-Reply-To: <20080610205633.11186.45499.stgit@dale> References: <20080610205633.11186.45499.stgit@dale> Message-ID: <71d336490806120821k3fe9f554l1a28dc55f204eadc@mail.gmail.com> Roland, On Wed, Jun 11, 2008 at 2:32 AM, Ramachandra K wrote: > Roland, > > This is the fourth round of QLogic Virtual NIC driver patch series for submission > to 2.6.27 kernel. The series has been tested against your for-2.6.27 branch. > > Based on comments received on third series of patches, following fixes have > been introduced in this series: > > - Single value per sysfs file created by QLogic VNIC Driver. > - Documentation/ABI/testing/sysfs-class-infiniband-qlgc-vnic > describing the sysfs interface of the QLogic VNIC Driver. > - Use of existing dev->stats instead of driver's private > vnic->stats. > - Changing more generic name completion_callback_cleanup to > the one more driver specific - vnic_completion_cleanup > - Removal of the field vnic->open > - Minor Modifications to linked list manipulations. Since the single value per sysfs file change has been implemented, is the driver ready for merging or are there any more changes required before the merge can happen ? Regards, Ram From sashak at voltaire.com Thu Jun 12 08:19:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 18:19:50 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213275809.14047.479.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <20080612113355.GB6256@sashak.voltaire.com> <1213271395.14047.463.camel@hrosenstock-ws.xsigo.com> <20080612121102.GF6256@sashak.voltaire.com> <1213275809.14047.479.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612151950.GS6256@sashak.voltaire.com> On 06:03 Thu 12 Jun , Hal Rosenstock wrote: > On Thu, 2008-06-12 at 15:11 +0300, Sasha Khapyorsky wrote: > > On 04:49 Thu 12 Jun , Hal Rosenstock wrote: > > > On Thu, 2008-06-12 at 14:33 +0300, Sasha Khapyorsky wrote: > > > > On 03:59 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > > > > > Would you elaborate on the motivation behind the requirement to > > > > > maintain/preserve the base LID routing ? > > > > > > > > I see couple advantages: > > > > > > > > 1. Application which works on base LIDs only will not be affected by LMC > > > > > > Yes; that's the obvious one. > > > > > > > 2. Changing LMC on a fabric will not change routing paths (when LIDs are > > > > not reassigned) > > > > > > That's when LMC is reduced rather than increased. > > > Also when increased and LIDs are not reassigned. > > Is that possible for anything other than the last base LID assigned ? We can set with LMC=2, decrease it to 0, increase it to 1, etc.. Normally LIDs will not be reassigned in this case. > > > > 3. Finally it does better balancing for secondary LIDs ("port > > > > offsetting") > > > > > > Isn't that accomodated in the patch but separate from the base LID > > > preservation ? > > > > It is integrated in the patch - balancing for each LID starts from > > its lower LID's port + 1. > > Understood (mostly) with comment below. > > > Not doing this would be really bad. > > Is the badness disrupting the base LID traffic or something else ? No, badness is bad balancing (see "port offseting" patchset thread). > > This is the trick - by preserving base LID traffic and offsetting over > > other LID ports we get better than before balancing. So right now it is > > hard to me to see when proposed option would be useful. > > When all ULPs use all LIDs "equally" and it's not just MPI ? Doesn't matter. LIDs still be "equial". Sasha From sashak at voltaire.com Thu Jun 12 08:27:20 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 18:27:20 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213275954.14047.483.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <1213271107.14047.456.camel@hrosenstock-ws.xsigo.com> <20080612114618.GE6256@sashak.voltaire.com> <1213271452.14047.464.camel@hrosenstock-ws.xsigo.com> <20080612121603.GG6256@sashak.voltaire.com> <1213275954.14047.483.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612152720.GT6256@sashak.voltaire.com> On 06:05 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > Right but why is OK to disrupt non base LID traffic but not base LID > > > traffic ? Is it only because it's only being used by one ULP > > > currrently ? > > > > How this disrupt non base LID traffic more than it was now? > > I didn't say that. I was asking why it's OK for non base LID traffic but > not base LID traffic. Then I likely don't understand your question. What are you asking "why base LID preservation is useful"? I think I answered this already. > Isn't it that's it's many ULPs rather than just > MPI ? I don't see how the whole issue is related to MPI/non-MPI. Sasha From ramachandra.kuchimanchi at qlogic.com Thu Jun 12 08:29:05 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Thu, 12 Jun 2008 20:59:05 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <48513E4E.4040601@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> Message-ID: <71d336490806120829o7cac67bes384c305e03b0d746@mail.gmail.com> On Thu, Jun 12, 2008 at 8:48 PM, Patrick McHardy wrote: > Amar Mudrankit wrote: >>>> >>>> +/sys/class/infiniband_qlgc_vnic/interfaces// >>>> + >>>> + vnic_state (0444) State of the VNIC interface. >>> >>> This I don't understand - you seem to be registering and >>> unregistering the net_device based some state machine, >>> in some cases even triggered by timers. >>> >>> Whats the idea behind this? >> >> For a given VNIC interface, unless the VNIC host driver establishes a >> connection with EVIC >> and completes the control and data path communication sequence with >> the EVIC, the >> VNIC network device is not registered on the host. Establishing a >> connection with EVIC >> and going through the control and data path communication involves >> multiple packet >> exchange on the IB side with the EVIC and can take some time. The >> netpath statemachine is, hence, meant to PERIODICALLY check if >> connection is fully established with EVIC or not. >> Once connection is established, it takes care to register the VNIC >> netdevice. > > > We have linkstate/operstate for this. How is a user supposed > to configure the network device when it appears at a more or > less random time from his perspective? > If you are referring to IP address configuration etc, users can configure the interfaces by setting up ifcfg files and the interfaces are automatically configured when they are registered. Regards, Ram From kaber at trash.net Thu Jun 12 08:34:53 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 12 Jun 2008 17:34:53 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <71d336490806120829o7cac67bes384c305e03b0d746@mail.gmail.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <71d336490806120829o7cac67bes384c305e03b0d746@mail.gmail.com> Message-ID: <4851421D.4030905@trash.net> Ramachandra K wrote: > On Thu, Jun 12, 2008 at 8:48 PM, Patrick McHardy wrote: >> Amar Mudrankit wrote: >>>>> +/sys/class/infiniband_qlgc_vnic/interfaces// >>>>> + >>>>> + vnic_state (0444) State of the VNIC interface. >>>> This I don't understand - you seem to be registering and >>>> unregistering the net_device based some state machine, >>>> in some cases even triggered by timers. >>>> >>>> Whats the idea behind this? >>> For a given VNIC interface, unless the VNIC host driver establishes a >>> connection with EVIC >>> and completes the control and data path communication sequence with >>> the EVIC, the >>> VNIC network device is not registered on the host. Establishing a >>> connection with EVIC >>> and going through the control and data path communication involves >>> multiple packet >>> exchange on the IB side with the EVIC and can take some time. The >>> netpath statemachine is, hence, meant to PERIODICALLY check if >>> connection is fully established with EVIC or not. >>> Once connection is established, it takes care to register the VNIC >>> netdevice. >> >> We have linkstate/operstate for this. How is a user supposed >> to configure the network device when it appears at a more or >> less random time from his perspective? >> > If you are referring to IP address configuration etc, users can configure > the interfaces by setting up ifcfg files and the interfaces are automatically > configured when they are registered. Maybe they can. It gets more complicated when daemons want to bind to that device etc. But that still leaves the main question, why is it not using the standard way and simply keeps the carrier turned off until the link is ready? From kaber at trash.net Thu Jun 12 08:35:32 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 12 Jun 2008 17:35:32 +0200 Subject: [ofa-general] [PATCH v4 00/14] QLogic VNIC Driver In-Reply-To: <71d336490806120821k3fe9f554l1a28dc55f204eadc@mail.gmail.com> References: <20080610205633.11186.45499.stgit@dale> <71d336490806120821k3fe9f554l1a28dc55f204eadc@mail.gmail.com> Message-ID: <48514244.7050500@trash.net> Ramachandra K wrote: > Since the single value per sysfs file change has been implemented, > is the driver ready for merging or are there any more changes required > before the merge can happen ? See my questions about this interface that you've skipped for the second time. From ariston at gmail.com Thu Jun 12 08:50:27 2008 From: ariston at gmail.com (Ramachandra K) Date: Thu, 12 Jun 2008 21:20:27 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <484F751F.7030407@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> Message-ID: <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> On Wed, Jun 11, 2008 at 12:17 PM, Patrick McHardy wrote: > Since I didn't receive anything but a "its too complex" in > response to my previous question, let me ask again: why are > you inventing a new sysfs interface (combined with ~15 module > parameters) instead of using the existing standard interfaces > for this? Can you please elaborate on the standard interfaces you are referring to ? Also as Roland mentioned in the other thread, echo-ing the parameters to sysfs files is a pretty straightforward way to configure the interfaces. Users can setup configuration files to configure QLogic VNIC interfaces specifying the name of the interface, various parameters of the QLogic EVIC gateway target they want to use and a simple user level script parses the configuration file and echoes the parameters to the driver. >> +# An interface can be deleted by echoing the name of the interface to be >> deleted >> +# to delete_vnic file >> +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic > > > This clearly shows the advantage of being able to use echo > for configuration. >From this statement, I thought you agreed with the advantage of echo for configuration. Regards, Ram From kaber at trash.net Thu Jun 12 09:03:38 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 12 Jun 2008 18:03:38 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> Message-ID: <485148DA.2060508@trash.net> Ramachandra K wrote: > On Wed, Jun 11, 2008 at 12:17 PM, Patrick McHardy wrote: >> Since I didn't receive anything but a "its too complex" in >> response to my previous question, let me ask again: why are >> you inventing a new sysfs interface (combined with ~15 module >> parameters) instead of using the existing standard interfaces >> for this? > > Can you please elaborate on the standard interfaces you are referring to ? > > Also as Roland mentioned in the other thread, echo-ing the parameters > to sysfs files > is a pretty straightforward way to configure the interfaces. Users > can setup configuration files to configure QLogic VNIC interfaces > specifying the name of the interface, various parameters of the > QLogic EVIC gateway target they want to use and a simple user level > script parses the configuration file and echoes the parameters to the driver. We introduced the rtnl_link API to avoid having each driver come up with its own method of creating virtual network devices. Besides the fact that it is the interface intended to be used for this kind of thing and the advantages mentioned in my response to Roland, looking at your interface, there are a few more advantages from integrating this cleanly in iproute: > +echo -n 00066a01de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_guid > +echo -n fe8000000000000100066a11de000037 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/dgid > +echo -n ffff > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/pkey I guess life would be easier for users if those parameters can be generated in userspace or at least looked up in something like the maps in /etc/iproute2/. > +echo -n 100 > /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/heartbeat I assume thats some kind of timing parameter. iproute would allow to specify the (undocumented) unit and convert it appropriately. In any case, things like tx_csum and rx_csum definitely do not belong in this interface. >>> +# An interface can be deleted by echoing the name of the interface to be >>> deleted >>> +# to delete_vnic file >>> +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic >> >> This clearly shows the advantage of being able to use echo >> for configuration. > >>From this statement, I thought you agreed with the advantage of echo for > configuration. Sorry, that wasn't meant seriously :) From shaeffer at neuralscape.com Thu Jun 12 09:04:39 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Thu, 12 Jun 2008 09:04:39 -0700 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> Message-ID: <20080612160439.GA13713@synapse.neuralscape.com> On Thu, Jun 12, 2008 at 09:20:27PM +0530, Ramachandra K wrote: > > > >> +# An interface can be deleted by echoing the name of the interface to be > >> deleted > >> +# to delete_vnic file > >> +echo -n veth1 > /sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic > > > > > > This clearly shows the advantage of being able to use echo > > for configuration. > > >From this statement, I thought you agreed with the advantage of echo for > configuration. > > Regards, > Ram Hi Ram, I believe that was Roland's comment, not Patrick's. Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer at neuralscape.com http://www.neuralscape.com From weiny2 at llnl.gov Thu Jun 12 09:45:06 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 12 Jun 2008 09:45:06 -0700 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <20080612125041.GO6256@sashak.voltaire.com> References: <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com> <6C2C79E72C305246B504CBA17B5500C9043B1855@mtlexch01.mtl.com> <20080612125041.GO6256@sashak.voltaire.com> Message-ID: <20080612094506.2ac3363b.weiny2@llnl.gov> Do we have a global header where we could put this in an #ifdef __WIN__ #endif ? Does it matter if the pragma is turned back on? I would assume not. Ira On Thu, 12 Jun 2008 15:50:41 +0300 Sasha Khapyorsky wrote: > On 15:46 Thu 12 Jun , Tzachi Dar wrote: > > The surrounded can be bigger than the actual array (for example a whole > > file). > > And what about using compiler flags and not touching the code itself? > I guess that if the code is supposed to be portable it should not have > stuff like '#pragma warning( disable : 4200)'. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From chu11 at llnl.gov Thu Jun 12 09:48:37 2008 From: chu11 at llnl.gov (Al Chu) Date: Thu, 12 Jun 2008 09:48:37 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <485000CC.8070502@nasa.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> Message-ID: <1213289317.5369.279.camel@cardanus.llnl.gov> Hey Jeff, On Wed, 2008-06-11 at 09:43 -0700, Jeff Becker wrote: > Basically, we have an Altix ICE cluster connected by a pair of hypercube > Infiniband fabrics. External to that, we have some Lustre nodes > connected into the cluster with Infiniband. Our goal is to keep Lustre > traffic separate from compute (MPI) traffic. Ideally, we'd have 2 > subnets and an IB router between the Lustre fabric and the compute > fabric to accomplish this. I see. In your environment, the lustre storage servers are on the same fabric as your compute nodes? > Barring that, I thought we could use partitions as follows: compute > HCA's and switch ports are on both partitions with full membership in > compute partition, and limited membership in I/O partition. The Lustre > nodes and switches would only be in the I/O partition (full > membership). That way, inter compute node (MPI) traffic would be > disallowed from using routes through the I/O fabric (by partition > membership), and I/O traffic could not interfere with compute (via > separate partitions). Is this scheme feasible? > > If that's not possible, the next idea is to modify OpenSM to assign > large weights to the links between the compute and I/O fabrics, so that > the MinHop algorithm would never consider using these links for > inter-compute node traffic. So dedicating (for example) X out of Y uplinks for MPI only and the remaining uplinks for lustre only? Al > Thoughts? Thanks. > > -jeff > > Al Chu wrote: > > Hey Jeff, > > > > Out of my curiosity, are you just trying to change the routing to > > improve job performance? i.e. lustre nodes get special routing vs. > > compute nodes? > > > > Al > > > > On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: > > > >> Hi all. I was looking into doing some subnet partitioning to separate > >> compute nodes from Lustre nodes, and I saw the following in > >> ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt > >> > >> OpenSM Partition Management > >> --------------------------- > >> > >> Roadmap: > >> Phase 1 - provide partition management at the EndPort (HCA, Router and Switch > >> Port 0) level with no routing affects. > >> Phase 2 - routing engine should take partitions into account. > >> ... > >> Phase 2 functionality: > >> > >> The partition policy should be considered during the routing such that > >> links are associated with particular partition or a set of > >> partitions. Policy should be enhanced to provide hints for how to do > >> that (correlating to QoS too). The exact algorithm is TBD. > >> > >> > >> What is the status of Pkey-aware routing? Thanks. > >> > >> -jeff > >> > >> _______________________________________________ > >> general mailing list > >> general at lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >> > >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > >> > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From tzachid at mellanox.co.il Thu Jun 12 09:49:17 2008 From: tzachid at mellanox.co.il (Tzachi Dar) Date: Thu, 12 Jun 2008 19:49:17 +0300 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <20080612094506.2ac3363b.weiny2@llnl.gov> References: <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com><6C2C79E72C305246B504CBA17B5500C9043B1855@mtlexch01.mtl.com><20080612125041.GO6256@sashak.voltaire.com> <20080612094506.2ac3363b.weiny2@llnl.gov> Message-ID: <6C2C79E72C305246B504CBA17B5500C9043B1AA4@mtlexch01.mtl.com> 1) I believe that we can live without this pragma being turned on again. 2) As far as I remember pragma are a way to tell a compiler what to do. If a compiler doesn't understand it, it just ignores it. Thanks Tzachi > -----Original Message----- > From: Ira Weiny [mailto:weiny2 at llnl.gov] > Sent: Thursday, June 12, 2008 7:45 PM > To: Sasha Khapyorsky > Cc: Tzachi Dar; ofw at lists.openfabrics.org; Leonid Keller; > hrosenstock at xsigo.com; general at lists.openfabrics.org > Subject: Re: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl > > Do we have a global header where we could put this in an > > #ifdef __WIN__ > #endif > > ? > > Does it matter if the pragma is turned back on? I would assume not. > > Ira > > On Thu, 12 Jun 2008 15:50:41 +0300 > Sasha Khapyorsky wrote: > > > On 15:46 Thu 12 Jun , Tzachi Dar wrote: > > > The surrounded can be bigger than the actual array (for example a > > > whole file). > > > > And what about using compiler flags and not touching the > code itself? > > I guess that if the code is supposed to be portable it > should not have > > stuff like '#pragma warning( disable : 4200)'. > > > > Sasha > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > From ftillier at windows.microsoft.com Thu Jun 12 10:02:58 2008 From: ftillier at windows.microsoft.com (Fab Tillier) Date: Thu, 12 Jun 2008 10:02:58 -0700 Subject: [ofw] Re: [ofa-general] Issues with _osm_lin_fwd_tbl In-Reply-To: <20080612094506.2ac3363b.weiny2@llnl.gov> References: <1213270989.14047.454.camel@hrosenstock-ws.xsigo.com> <6C2C79E72C305246B504CBA17B5500C9043B1855@mtlexch01.mtl.com> <20080612125041.GO6256@sashak.voltaire.com> <20080612094506.2ac3363b.weiny2@llnl.gov> Message-ID: <1D71F3D3B53304469F84F75958DABA8E0C436CF59A@TK5-EXMBX-W603v.wingroup.windeploy.ntdev.microsoft.com> > From: Ira Weiny > > Do we have a global header where we could put this in an > > #ifdef __WIN__ > #endif > > ? You can add /wd4200 in the SOURCES file for the components that use zero-length arrays. This would then only require the pragmas in public headers. -Fab From Jeffrey.C.Becker at nasa.gov Thu Jun 12 10:11:50 2008 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Thu, 12 Jun 2008 10:11:50 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213289317.5369.279.camel@cardanus.llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> Message-ID: <485158D6.4050201@nasa.gov> Hi Al Al Chu wrote: > Hey Jeff, > > On Wed, 2008-06-11 at 09:43 -0700, Jeff Becker wrote: > >> Basically, we have an Altix ICE cluster connected by a pair of hypercube >> Infiniband fabrics. External to that, we have some Lustre nodes >> connected into the cluster with Infiniband. Our goal is to keep Lustre >> traffic separate from compute (MPI) traffic. Ideally, we'd have 2 >> subnets and an IB router between the Lustre fabric and the compute >> fabric to accomplish this. >> > > I see. In your environment, the lustre storage servers are on the same > fabric as your compute nodes? > Right. > >> Barring that, I thought we could use partitions as follows: compute >> HCA's and switch ports are on both partitions with full membership in >> compute partition, and limited membership in I/O partition. The Lustre >> nodes and switches would only be in the I/O partition (full >> membership). That way, inter compute node (MPI) traffic would be >> disallowed from using routes through the I/O fabric (by partition >> membership), and I/O traffic could not interfere with compute (via >> separate partitions). Is this scheme feasible? >> >> If that's not possible, the next idea is to modify OpenSM to assign >> large weights to the links between the compute and I/O fabrics, so that >> the MinHop algorithm would never consider using these links for >> inter-compute node traffic. >> > > So dedicating (for example) X out of Y uplinks for MPI only and the > remaining uplinks for lustre only? > That works. The compute nodes need to talk to other compute nodes for MPI over one set of links, and they need to talk to the Lustre nodes for I/O, but over a different (disjoint) set of links. Thanks. -jeff > Al > > >> Thoughts? Thanks. >> >> -jeff >> >> Al Chu wrote: >> >>> Hey Jeff, >>> >>> Out of my curiosity, are you just trying to change the routing to >>> improve job performance? i.e. lustre nodes get special routing vs. >>> compute nodes? >>> >>> Al >>> >>> On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: >>> >>> >>>> Hi all. I was looking into doing some subnet partitioning to separate >>>> compute nodes from Lustre nodes, and I saw the following in >>>> ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt >>>> >>>> OpenSM Partition Management >>>> --------------------------- >>>> >>>> Roadmap: >>>> Phase 1 - provide partition management at the EndPort (HCA, Router and Switch >>>> Port 0) level with no routing affects. >>>> Phase 2 - routing engine should take partitions into account. >>>> ... >>>> Phase 2 functionality: >>>> >>>> The partition policy should be considered during the routing such that >>>> links are associated with particular partition or a set of >>>> partitions. Policy should be enhanced to provide hints for how to do >>>> that (correlating to QoS too). The exact algorithm is TBD. >>>> >>>> >>>> What is the status of Pkey-aware routing? Thanks. >>>> >>>> -jeff >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>>> >>>> From 20aurore.sergeant at pum.fr Thu Jun 12 08:57:05 2008 From: 20aurore.sergeant at pum.fr (averill sotiris) Date: Thu, 12 Jun 2008 15:57:05 +0000 Subject: [ofa-general] Best of Bally, Paul Smith, UGG Message-ID: <000601c8ccb3$07aa9cce$b99bfeac@riwouvcd> The world's broad luxury store for shoes and bags is just one click away. Recommended by hundred thousands of satisfied customers all around the world, we carry dozens of famous brands including: Prada Adidas Coach Dsquared Louis Vuitton Here you willc find tens thousands of owsome designs for shoes, and leather products, at rock bottom pri'c'ing. Sale close this week, so visit our store today and start pampering yourself and your loved second half. - Visit our site: www.rapaette[DOT]com (copy this link and replace "[DOT]" to ".") From chu11 at llnl.gov Thu Jun 12 10:49:44 2008 From: chu11 at llnl.gov (Al Chu) Date: Thu, 12 Jun 2008 10:49:44 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <485158D6.4050201@nasa.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> Message-ID: <1213292984.5369.292.camel@cardanus.llnl.gov> Hey Jeff, > That works. The compute nodes need to talk to other compute nodes for > MPI over one set of links, and they need to talk to the Lustre nodes for > I/O, but over a different (disjoint) set of links. Thanks. Is there a strong belief that a different/disjoint set of links would be beneficial? Sometime ago, Sasha and I iterated on a patch in which I found out sometimes not all switch ports would be used. In this particular case, a chunk of leaf switches were sometimes using only 11 out of 12 uplinks. After the fix, mpigraph showed about 20% improvement in MPI bandwidth. It obviously depends on your cluster/environment/apps/user usage pattern/etc. Livermore Lab's usage patterns will probably be different. Al On Thu, 2008-06-12 at 10:11 -0700, Jeff Becker wrote: > Hi Al > > Al Chu wrote: > > Hey Jeff, > > > > On Wed, 2008-06-11 at 09:43 -0700, Jeff Becker wrote: > > > >> Basically, we have an Altix ICE cluster connected by a pair of hypercube > >> Infiniband fabrics. External to that, we have some Lustre nodes > >> connected into the cluster with Infiniband. Our goal is to keep Lustre > >> traffic separate from compute (MPI) traffic. Ideally, we'd have 2 > >> subnets and an IB router between the Lustre fabric and the compute > >> fabric to accomplish this. > >> > > > > I see. In your environment, the lustre storage servers are on the same > > fabric as your compute nodes? > > > Right. > > > >> Barring that, I thought we could use partitions as follows: compute > >> HCA's and switch ports are on both partitions with full membership in > >> compute partition, and limited membership in I/O partition. The Lustre > >> nodes and switches would only be in the I/O partition (full > >> membership). That way, inter compute node (MPI) traffic would be > >> disallowed from using routes through the I/O fabric (by partition > >> membership), and I/O traffic could not interfere with compute (via > >> separate partitions). Is this scheme feasible? > >> > >> If that's not possible, the next idea is to modify OpenSM to assign > >> large weights to the links between the compute and I/O fabrics, so that > >> the MinHop algorithm would never consider using these links for > >> inter-compute node traffic. > >> > > > > So dedicating (for example) X out of Y uplinks for MPI only and the > > remaining uplinks for lustre only? > > > That works. The compute nodes need to talk to other compute nodes for > MPI over one set of links, and they need to talk to the Lustre nodes for > I/O, but over a different (disjoint) set of links. Thanks. > > -jeff > > Al > > > > > >> Thoughts? Thanks. > >> > >> -jeff > >> > >> Al Chu wrote: > >> > >>> Hey Jeff, > >>> > >>> Out of my curiosity, are you just trying to change the routing to > >>> improve job performance? i.e. lustre nodes get special routing vs. > >>> compute nodes? > >>> > >>> Al > >>> > >>> On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: > >>> > >>> > >>>> Hi all. I was looking into doing some subnet partitioning to separate > >>>> compute nodes from Lustre nodes, and I saw the following in > >>>> ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt > >>>> > >>>> OpenSM Partition Management > >>>> --------------------------- > >>>> > >>>> Roadmap: > >>>> Phase 1 - provide partition management at the EndPort (HCA, Router and Switch > >>>> Port 0) level with no routing affects. > >>>> Phase 2 - routing engine should take partitions into account. > >>>> ... > >>>> Phase 2 functionality: > >>>> > >>>> The partition policy should be considered during the routing such that > >>>> links are associated with particular partition or a set of > >>>> partitions. Policy should be enhanced to provide hints for how to do > >>>> that (correlating to QoS too). The exact algorithm is TBD. > >>>> > >>>> > >>>> What is the status of Pkey-aware routing? Thanks. > >>>> > >>>> -jeff > >>>> > >>>> _______________________________________________ > >>>> general mailing list > >>>> general at lists.openfabrics.org > >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >>>> > >>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > >>>> > >>>> > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From hrosenstock at xsigo.com Thu Jun 12 11:02:48 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 11:02:48 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_sa_mcmember_record.c: Breakup some long lines Message-ID: <1213293768.14047.522.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Breakup some long lines Also, fix commentary typo Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 0168021..de174c5 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -197,7 +197,8 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) if (cl_ntoh16(p_mgrp->mlid) > sa->p_subn->max_multicast_lid_ho) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B27: " - "Found mgrp with mlid:0x%04X > max allowed mlid:0x%04X\n", + "Found mgrp with mlid:0x%04X > " + "max allowed mlid:0x%04X\n", cl_ntoh16(p_mgrp->mlid), max_num_mlids + IB_LID_MCAST_START_HO); } else { @@ -286,7 +287,8 @@ __add_new_mgrp_port(IN osm_sa_t * sa, } else { /* The port is not the one specified in PortGID. The check that the requester is in the same partition as - the PortGID is done before - just need to update the proxy_join. */ + the PortGID is done before - just need to update + the proxy_join. */ proxy_join = TRUE; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Create new port with proxy_join TRUE\n"); @@ -389,7 +391,8 @@ __validate_more_comp_fields(osm_log_t * p_log, case 0: /* Greater than MTU specified */ if (mtu_mgrp <= mtu_required) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Requested mcast group has MTU %x, which is not greater than %x\n", + "Requested mcast group has MTU %x, " + "which is not greater than %x\n", mtu_mgrp, mtu_required); return FALSE; } @@ -397,7 +400,8 @@ __validate_more_comp_fields(osm_log_t * p_log, case 1: /* Less than MTU specified */ if (mtu_mgrp >= mtu_required) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Requested mcast group has MTU %x, which is not less than %x\n", + "Requested mcast group has MTU %x, " + "which is not less than %x\n", mtu_mgrp, mtu_required); return FALSE; } @@ -405,7 +409,8 @@ __validate_more_comp_fields(osm_log_t * p_log, case 2: /* Exactly MTU specified */ if (mtu_mgrp != mtu_required) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Requested mcast group has MTU %x, which is not equal to %x\n", + "Requested mcast group has MTU %x, " + "which is not equal to %x\n", mtu_mgrp, mtu_required); return FALSE; } @@ -425,7 +430,8 @@ __validate_more_comp_fields(osm_log_t * p_log, case 0: /* Greater than RATE specified */ if (rate_mgrp <= rate_required) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Requested mcast group has RATE %x, which is not greater than %x\n", + "Requested mcast group has RATE %x, " + "which is not greater than %x\n", rate_mgrp, rate_required); return FALSE; } @@ -433,7 +439,8 @@ __validate_more_comp_fields(osm_log_t * p_log, case 1: /* Less than RATE specified */ if (rate_mgrp >= rate_required) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Requested mcast group has RATE %x, which is not less than %x\n", + "Requested mcast group has RATE %x, " + "which is not less than %x\n", rate_mgrp, rate_required); return FALSE; } @@ -441,7 +448,8 @@ __validate_more_comp_fields(osm_log_t * p_log, case 2: /* Exactly RATE specified */ if (rate_mgrp != rate_required) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Requested mcast group has RATE %x, which is not equal to %x\n", + "Requested mcast group has RATE %x, " + "which is not equal to %x\n", rate_mgrp, rate_required); return FALSE; } @@ -540,8 +548,8 @@ __validate_modify(IN osm_sa_t * sa, if (memcmp(&((*pp_mcm_port)->port_gid), &request_gid, sizeof(ib_gid_t))) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "No ProxyJoin but different ports: stored:0x%016" - PRIx64 " request:0x%016" PRIx64 "\n", + "No ProxyJoin but different ports: stored:" + "0x%016" PRIx64 " request:0x%016" PRIx64 "\n", cl_ntoh64((*pp_mcm_port)->port_gid.unicast. interface_id), cl_ntoh64(p_mad_addr->addr_type.gsi.grh_info. @@ -561,8 +569,8 @@ __validate_modify(IN osm_sa_t * sa, p_request_physp)) { /* the request port is not part of the partition for this mgrp */ OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "ProxyJoin but port not in partition. stored:0x%016" - PRIx64 " request:0x%016" PRIx64 "\n", + "ProxyJoin but port not in partition. stored:" + "0x%016" PRIx64 " request:0x%016" PRIx64 "\n", cl_ntoh64((*pp_mcm_port)->port_gid.unicast. interface_id), cl_ntoh64(p_mad_addr->addr_type.gsi.grh_info. @@ -619,7 +627,8 @@ __validate_delete(IN osm_sa_t * sa, if (!(p_recvd_mcmember_rec->scope_state & 0x0F & (*pp_mcm_port)->scope_state)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Could not find any matching bits in the stored and requested JoinStates\n"); + "Could not find any matching bits in the stored " + "and requested JoinStates\n"); return FALSE; } @@ -628,7 +637,8 @@ __validate_delete(IN osm_sa_t * sa, (0x0F & (*pp_mcm_port)->scope_state)) != (0x0F & (*pp_mcm_port)->scope_state)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Some bits in the request JoinState (0x%X) are not set in the stored port (0x%X)\n", + "Some bits in the request JoinState (0x%X) are not " + "set in the stored port (0x%X)\n", (p_recvd_mcmember_rec->scope_state & 0x0F), (0x0F & (*pp_mcm_port)->scope_state)); return FALSE; @@ -1028,7 +1038,8 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, if (p_prev_mgrp != (osm_mgrp_t *) cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Found previous group for mlid:0x%04x - Need to destroy it\n", + "Found previous group for mlid:0x%04x - " + "Need to destroy it\n", cl_ntoh16(mlid)); cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, (cl_map_item_t *) p_prev_mgrp); @@ -1038,7 +1049,7 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, cl_qmap_insert(&sa->p_subn->mgrp_mlid_tbl, mlid, &(*pp_mgrp)->map_item); - /* Send a Report to any InformInfo registerd for + /* Send a Report to any InformInfo registered for Trap 66: MCGroup create */ osm_mgrp_send_create_notice(sa->p_subn, sa->p_log, *pp_mgrp); @@ -1098,8 +1109,9 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) (g_interface_id & INT_ID_MASK) == (rcv_interface_id & INT_ID_MASK)) { OSM_LOG(sa->p_log, OSM_LOG_INFO, - "Special Case Solicited Node Mcast Join " - " for MGID 0x%016"PRIx64" : 0x%016"PRIx64"\n", + "Special Case Solicited Node Mcast " + " Join for MGID 0x%016" PRIx64 + " : 0x%016" PRIx64 "\n", rcv_prefix, rcv_interface_id); } else return; @@ -1204,9 +1216,9 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, if (valid) { /* - * according to the same o15-0.1.14 we get the stored JoinState and the - * request JoinState and they must be opposite to leave - - * otherwise just update it + * according to the same o15-0.1.14 we get the stored + * JoinState and the request JoinState and they must be + * opposite to leave - otherwise just update it */ port_join_state = p_mcm_port->scope_state & 0x0F; new_join_state = @@ -1224,7 +1236,8 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, CL_PLOCK_RELEASE(sa->p_lock); OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "After update JoinState != 0. Updating from 0x%X to 0x%X\n", + "After update JoinState != 0. " + "Updating from 0x%X to 0x%X\n", port_join_state, new_join_state); } else { /* we need to return the stored scope state */ @@ -1238,7 +1251,8 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, osm_sm_mcgrp_leave(sa->sm, mlid, portguid); if (status != IB_SUCCESS) { - OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B09: " + OSM_LOG(sa->p_log, OSM_LOG_ERROR, + "ERR 1B09: " "osm_sm_mcgrp_leave failed\n"); } } @@ -1361,7 +1375,8 @@ __osm_mcmr_rcv_join_mgrp(IN osm_sa_t * sa, if ((join_state & 0x01) != 0x01) { CL_PLOCK_RELEASE(sa->p_lock); OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B10: " - "Provided Join State != FullMember - required for create, " + "Provided Join State != FullMember - " + "required for create, " "MGID: 0x%016" PRIx64 " : " "0x%016" PRIx64 " from port 0x%016" PRIx64 " (%s)\n", From Jeffrey.C.Becker at nasa.gov Thu Jun 12 11:14:43 2008 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Thu, 12 Jun 2008 11:14:43 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213292984.5369.292.camel@cardanus.llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> Message-ID: <48516793.3090708@nasa.gov> Hi Al Al Chu wrote: > Hey Jeff, > > >> That works. The compute nodes need to talk to other compute nodes for >> MPI over one set of links, and they need to talk to the Lustre nodes for >> I/O, but over a different (disjoint) set of links. Thanks. >> > > Is there a strong belief that a different/disjoint set of links would be > beneficial? Sometime ago, Sasha and I iterated on a patch in which I > found out sometimes not all switch ports would be used. In this > particular case, a chunk of leaf switches were sometimes using only 11 > out of 12 uplinks. After the fix, mpigraph showed about 20% improvement > in MPI bandwidth. > Basically, we want to avoid situations where I/O and MPI contend for the same links, and get in each other's way. -jeff > It obviously depends on your cluster/environment/apps/user usage > pattern/etc. Livermore Lab's usage patterns will probably be different. > > Al > > On Thu, 2008-06-12 at 10:11 -0700, Jeff Becker wrote: > >> Hi Al >> >> Al Chu wrote: >> >>> Hey Jeff, >>> >>> On Wed, 2008-06-11 at 09:43 -0700, Jeff Becker wrote: >>> >>> >>>> Basically, we have an Altix ICE cluster connected by a pair of hypercube >>>> Infiniband fabrics. External to that, we have some Lustre nodes >>>> connected into the cluster with Infiniband. Our goal is to keep Lustre >>>> traffic separate from compute (MPI) traffic. Ideally, we'd have 2 >>>> subnets and an IB router between the Lustre fabric and the compute >>>> fabric to accomplish this. >>>> >>>> >>> I see. In your environment, the lustre storage servers are on the same >>> fabric as your compute nodes? >>> >>> >> Right. >> >>> >>> >>>> Barring that, I thought we could use partitions as follows: compute >>>> HCA's and switch ports are on both partitions with full membership in >>>> compute partition, and limited membership in I/O partition. The Lustre >>>> nodes and switches would only be in the I/O partition (full >>>> membership). That way, inter compute node (MPI) traffic would be >>>> disallowed from using routes through the I/O fabric (by partition >>>> membership), and I/O traffic could not interfere with compute (via >>>> separate partitions). Is this scheme feasible? >>>> >>>> If that's not possible, the next idea is to modify OpenSM to assign >>>> large weights to the links between the compute and I/O fabrics, so that >>>> the MinHop algorithm would never consider using these links for >>>> inter-compute node traffic. >>>> >>>> >>> So dedicating (for example) X out of Y uplinks for MPI only and the >>> remaining uplinks for lustre only? >>> >>> >> That works. The compute nodes need to talk to other compute nodes for >> MPI over one set of links, and they need to talk to the Lustre nodes for >> I/O, but over a different (disjoint) set of links. Thanks. >> >> -jeff >> >>> Al >>> >>> >>> >>>> Thoughts? Thanks. >>>> >>>> -jeff >>>> >>>> Al Chu wrote: >>>> >>>> >>>>> Hey Jeff, >>>>> >>>>> Out of my curiosity, are you just trying to change the routing to >>>>> improve job performance? i.e. lustre nodes get special routing vs. >>>>> compute nodes? >>>>> >>>>> Al >>>>> >>>>> On Tue, 2008-06-10 at 15:08 -0700, Jeff Becker wrote: >>>>> >>>>> >>>>> >>>>>> Hi all. I was looking into doing some subnet partitioning to separate >>>>>> compute nodes from Lustre nodes, and I saw the following in >>>>>> ~sashak/management.git on the OFA server, in opensm/doc/OpenSM_PKey_Mgr.txt >>>>>> >>>>>> OpenSM Partition Management >>>>>> --------------------------- >>>>>> >>>>>> Roadmap: >>>>>> Phase 1 - provide partition management at the EndPort (HCA, Router and Switch >>>>>> Port 0) level with no routing affects. >>>>>> Phase 2 - routing engine should take partitions into account. >>>>>> ... >>>>>> Phase 2 functionality: >>>>>> >>>>>> The partition policy should be considered during the routing such that >>>>>> links are associated with particular partition or a set of >>>>>> partitions. Policy should be enhanced to provide hints for how to do >>>>>> that (correlating to QoS too). The exact algorithm is TBD. >>>>>> >>>>>> >>>>>> What is the status of Pkey-aware routing? Thanks. >>>>>> >>>>>> -jeff >>>>>> >>>>>> _______________________________________________ >>>>>> general mailing list >>>>>> general at lists.openfabrics.org >>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>>>> >>>>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>>>>> >>>>>> >>>>>> From dledford at redhat.com Thu Jun 12 12:08:25 2008 From: dledford at redhat.com (Doug Ledford) Date: Thu, 12 Jun 2008 15:08:25 -0400 Subject: [ofa-general] Missing tarballs from downloads area of ofa website Message-ID: <48517429.7000509@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've been trying to submit various packages to the Fedora review process so the packages can be included in the next Fedora release. Unfortunately, the reviewers are very much sticklers over the issue of the existence of a valid upstream tarball release that can be downloaded and verified against the package. This has prevented me from submitting several packages for review. Amongst them: libipathverbs (there is a release up on the site, but I happen to know that the OFED-1.3 distro include a tarball libipathverbs tarball with the exact same version number but a totally different set of files, so until there is a definitive release tarball, I can't submit this one) libehca (no tarball exists...of course, this isn't too big a deal for Fedora since it doesn't really support big iron ppc hardware anyway) libnes (no tarball) Most of the other important packages have tarballs up for download, thanks much for that. Several of the less important packages are still missing tarballs for download (perftest, qlvnictools, mstflint, stuff like that) and will be skipped until they are available. - -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iEYEARECAAYFAkhRdCkACgkQTzP9PM7pjw8TeACeMs0uhGoZ/f+YZwmIKf+GdEsP BB0AniovOGjym2kjBSm7tuNy2aexvlCP =MD/G -----END PGP SIGNATURE----- From vu at mellanox.com Thu Jun 12 12:36:13 2008 From: vu at mellanox.com (Vu Pham) Date: Thu, 12 Jun 2008 12:36:13 -0700 Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, Message-ID: <48517AAD.5070303@mellanox.com> Hi James/Tom, MLX4 transport (connectX HCA) need 2 extra S/G entries per work request. One is for the header and one for the checksum at the end. This patch will fix rpc rdma transport to create qp with max_send_sge = device_cap.max_sge - 2 if low-level device hca is connectX thanks, -vu -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: nfsrdma_max_send_sge_for_mlx4.patch URL: From sashak at voltaire.com Thu Jun 12 12:43:48 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 22:43:48 +0300 Subject: [ofa-general] Re: [PATCH] infiniband-diags/saquery.c: In print_multicast_group_records, only query NodeRecords when needed In-Reply-To: <1213121551.14047.265.camel@hrosenstock-ws.xsigo.com> References: <1213121551.14047.265.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612194348.GV6256@sashak.voltaire.com> On 11:12 Tue 10 Jun , Hal Rosenstock wrote: > infiniband-diags/saquery.c: In print_multicast_group_records, only query > NodeRecords when needed. This is a minor optimization. > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From jlentini at netapp.com Thu Jun 12 12:45:53 2008 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 Jun 2008 15:45:53 -0400 (EDT) Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, In-Reply-To: <48517AAD.5070303@mellanox.com> References: <48517AAD.5070303@mellanox.com> Message-ID: On Thu, 12 Jun 2008, Vu Pham wrote: > Hi James/Tom, > > MLX4 transport (connectX HCA) need 2 extra S/G entries per work > request. One is for the header and one for the checksum at the end. > This patch will fix rpc rdma transport to create qp with > max_send_sge = device_cap.max_sge - 2 if low-level device hca is > connectX > > thanks, > -vu > > Vu, Many ULPs use the device's advertized maximum S/G count to size their S/G entries. Did you consider changing the driver to advertize 2 less S/G entries? james From sashak at voltaire.com Thu Jun 12 12:45:58 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 22:45:58 +0300 Subject: [ofa-general] [PATCH] infiniband-diags/src/saquery.c: fix potential core dump (Was: Re: infiniband-diags/saquery.c:print_multicast_member_record question) In-Reply-To: <20080610131803.34c7a889.weiny2@llnl.gov> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> <20080610112617.3563f3ed.weiny2@llnl.gov> <1213122919.14047.280.camel@hrosenstock-ws.xsigo.com> <20080610131803.34c7a889.weiny2@llnl.gov> Message-ID: <20080612194558.GW6256@sashak.voltaire.com> On 13:18 Tue 10 Jun , Ira Weiny wrote: > > >From ac10f52fd65b0b34b409ca2aa266a7363ae32e2c Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Tue, 10 Jun 2008 13:07:43 -0700 > Subject: [PATCH] infiniband-diags/src/saquery.c: fix potential core dump and/or incorrect node > descriptions from being printed if a corresponding node_record is not found for > a multicast member record. > > Signed-off-by: Ira K. Weiny Applied. Thanks. Guys, didn't you think that it is time to split this code into to different functions? Patch shortly... Sasha From sashak at voltaire.com Thu Jun 12 12:48:10 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 22:48:10 +0300 Subject: [ofa-general] [PATCH] infiniband-diags/saquery: split print_multicast_group_records() function In-Reply-To: <20080612194558.GW6256@sashak.voltaire.com> References: <1213109186.14047.239.camel@hrosenstock-ws.xsigo.com> <20080610090534.3197494c.weiny2@llnl.gov> <1213118604.14047.256.camel@hrosenstock-ws.xsigo.com> <20080610111131.039f399e.weiny2@llnl.gov> <1213121784.14047.269.camel@hrosenstock-ws.xsigo.com> <20080610112617.3563f3ed.weiny2@llnl.gov> <1213122919.14047.280.camel@hrosenstock-ws.xsigo.com> <20080610131803.34c7a889.weiny2@llnl.gov> <20080612194558.GW6256@sashak.voltaire.com> Message-ID: <20080612194810.GX6256@sashak.voltaire.com> Split logic overflowed function print_multicast_group_records() to two - print_multicast_group_records() and print_multicast_member_records(). Simplify the code as for another print_*_records() functions. Signed-off-by: Sasha Khapyorsky --- infiniband-diags/src/saquery.c | 60 ++++++++++++++++++++++------------------ 1 files changed, 33 insertions(+), 27 deletions(-) diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c index f1780fa..1594cad 100644 --- a/infiniband-diags/src/saquery.c +++ b/infiniband-diags/src/saquery.c @@ -324,9 +324,9 @@ static void dump_portinfo_record(void *data) ); } -static void -print_multicast_group_record(ib_member_rec_t *p_mcmr) +static void dump_multicast_group_record(void *data) { + ib_member_rec_t *p_mcmr = data; uint8_t sl; ib_member_get_sl_flow_hop(p_mcmr->sl_flow_hop, &sl, NULL, NULL); printf("MCMemberRecord group dump:\n" @@ -348,9 +348,9 @@ print_multicast_group_record(ib_member_rec_t *p_mcmr) ); } -static void -print_multicast_member_record(ib_member_rec_t *p_mcmr) +static void dump_multicast_member_record(void *data) { + ib_member_rec_t *p_mcmr = data; uint64_t gid_prefix = cl_ntoh64( p_mcmr->port_gid.unicast.prefix ); uint64_t gid_interface_id = cl_ntoh64( p_mcmr->port_gid.unicast.interface_id ); uint16_t mlid = cl_ntoh16( p_mcmr->mlid ); @@ -1039,37 +1039,25 @@ print_portinfo_records(osm_bind_handle_t bind_handle) } static ib_api_status_t -print_multicast_group_records(osm_bind_handle_t bind_handle, int members) +print_multicast_member_records(osm_bind_handle_t bind_handle) { - int i = 0; - ib_member_rec_t *mcast_record = NULL; - ib_net16_t mc_group_attr_offset = ib_get_attr_offset(sizeof(*mcast_record)); osmv_query_res_t mc_group_result; - ib_net16_t node_attr_offset = ib_get_attr_offset(sizeof(ib_node_record_t)); ib_api_status_t status; - status = get_all_records(bind_handle, IB_MAD_ATTR_MCMEMBER_RECORD, mc_group_attr_offset, members); + status = get_all_records(bind_handle, IB_MAD_ATTR_MCMEMBER_RECORD, + ib_get_attr_offset(sizeof(ib_member_rec_t)), 1); if (status != IB_SUCCESS) return (status); - mc_group_result = result; - if (members) { - status = get_all_records(bind_handle, IB_MAD_ATTR_NODE_RECORD, - node_attr_offset, 0); - if (status != IB_SUCCESS) - goto return_mc; - } + mc_group_result = result; - for (i = 0; i < mc_group_result.result_cnt; i++) { - mcast_record = osmv_get_query_mc_rec(mc_group_result.p_result_madw, i); - if (members == 0) - print_multicast_group_record(mcast_record); - else - print_multicast_member_record(mcast_record); - } + status = get_all_records(bind_handle, IB_MAD_ATTR_NODE_RECORD, + ib_get_attr_offset(sizeof(ib_node_record_t)), 0); + if (status != IB_SUCCESS) + goto return_mc; - if (members) - return_mad(); + dump_results(&mc_group_result, dump_multicast_member_record); + return_mad(); return_mc: /* return_mad for the mc_group_result */ @@ -1082,6 +1070,21 @@ return_mc: } static ib_api_status_t +print_multicast_group_records(osm_bind_handle_t bind_handle) +{ + ib_api_status_t status; + + status = get_all_records(bind_handle, IB_MAD_ATTR_MCMEMBER_RECORD, + ib_get_attr_offset(sizeof(ib_member_rec_t)), 0); + if (status != IB_SUCCESS) + return (status); + + dump_results(&result, dump_multicast_group_record); + return_mad(); + return (status); +} + +static ib_api_status_t print_service_records(osm_bind_handle_t bind_handle) { ib_net16_t attr_offset = ib_get_attr_offset(sizeof(ib_service_record_t)); @@ -1693,7 +1696,10 @@ main(int argc, char **argv) status = print_portinfo_records(bind_handle); break; case IB_MAD_ATTR_MCMEMBER_RECORD: - status = print_multicast_group_records(bind_handle, members); + if (members) + status = print_multicast_member_records(bind_handle); + else + status = print_multicast_group_records(bind_handle); break; case IB_MAD_ATTR_SERVICE_RECORD: status = print_service_records(bind_handle); -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Thu Jun 12 12:57:24 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 22:57:24 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_sa_mcmember_record.c: Breakup some long lines In-Reply-To: <1213293768.14047.522.camel@hrosenstock-ws.xsigo.com> References: <1213293768.14047.522.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612195724.GY6256@sashak.voltaire.com> On 11:02 Thu 12 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: Breakup some long lines > Also, fix commentary typo > > Signed-off-by: Hal Rosenstock All three are applied. Thanks. Sasha From jlentini at netapp.com Thu Jun 12 13:05:41 2008 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 Jun 2008 16:05:41 -0400 (EDT) Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, In-Reply-To: References: <48517AAD.5070303@mellanox.com> Message-ID: On Thu, 12 Jun 2008, James Lentini wrote: > > > On Thu, 12 Jun 2008, Vu Pham wrote: > > > Hi James/Tom, > > > > MLX4 transport (connectX HCA) need 2 extra S/G entries per work > > request. One is for the header and one for the checksum at the end. > > This patch will fix rpc rdma transport to create qp with > > max_send_sge = device_cap.max_sge - 2 if low-level device hca is > > connectX > > > > thanks, > > -vu > > > > > > Vu, > > Many ULPs use the device's advertized maximum S/G count to size > their S/G entries. > > Did you consider changing the driver to advertize 2 less S/G > entries? > > james > Vu, Is this already fixed: http://lists.openfabrics.org/pipermail/general/2008-May/050256.html From chu11 at llnl.gov Thu Jun 12 13:20:06 2008 From: chu11 at llnl.gov (Al Chu) Date: Thu, 12 Jun 2008 13:20:06 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213275809.14047.479.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213268351.14047.429.camel@hrosenstock-ws.xsigo.com> <20080612113355.GB6256@sashak.voltaire.com> <1213271395.14047.463.camel@hrosenstock-ws.xsigo.com> <20080612121102.GF6256@sashak.voltaire.com> <1213275809.14047.479.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213302006.5369.305.camel@cardanus.llnl.gov> Hey Hal, On Thu, 2008-06-12 at 06:03 -0700, Hal Rosenstock wrote: > On Thu, 2008-06-12 at 15:11 +0300, Sasha Khapyorsky wrote: > > On 04:49 Thu 12 Jun , Hal Rosenstock wrote: > > > On Thu, 2008-06-12 at 14:33 +0300, Sasha Khapyorsky wrote: > > > > On 03:59 Thu 12 Jun , Hal Rosenstock wrote: > > > > > > > > > > Would you elaborate on the motivation behind the requirement to > > > > > maintain/preserve the base LID routing ? > > > > > > > > I see couple advantages: > > > > > > > > 1. Application which works on base LIDs only will not be affected by LMC > > > > > > Yes; that's the obvious one. > > > > > > > 2. Changing LMC on a fabric will not change routing paths (when LIDs are > > > > not reassigned) > > > > > > That's when LMC is reduced rather than increased. > > > Also when increased and LIDs are not reassigned. > > Is that possible for anything other than the last base LID assigned ? > > > > > 3. Finally it does better balancing for secondary LIDs ("port > > > > offsetting") > > > > > > Isn't that accomodated in the patch but separate from the base LID > > > preservation ? > > > > It is integrated in the patch - balancing for each LID starts from > > its lower LID's port + 1. > > Understood (mostly) with comment below. > > > Not doing this would be really bad. > > Is the badness disrupting the base LID traffic or something else ? Just thought I'd comment on this since much discussion between Sasha, myself, and Yiftah (Sasha's coworker) was off list. As Sasha stated, the bad balancing was the worst part. That's what led me to develop my port-offsetting patch series (reasons for its development can be seen in that thread). Yiftah, made the case that disrupting base lid traffic was also part of the problem. If lmc is changed to > 0, it should not affect any user's code's performance due to base lid routing changes. This was reported by some other Voltaire user, but I can see the "principle" behind this. Needless to say, the lmc change to > 0 affected routing greatly due to bad balancing before. But even minor basic-lid routing changes could affect users jobs. Al > > > Preserving/maintaining base LIDs is a policy decision and perhaps this > > > should be an option with this as the default. For some balancing all > > > paths might be more important that not disrupting base LID traffic. > > > > This is the trick - by preserving base LID traffic and offsetting over > > other LID ports we get better than before balancing. So right now it is > > hard to me to see when proposed option would be useful. > > When all ULPs use all LIDs "equally" and it's not just MPI ? > > > But probably it would, I think we can add it then. > > Sure; this can be viewed a future thing. > > -- Hal > > > Sasha > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From jeff at garzik.org Thu Jun 12 13:22:24 2008 From: jeff at garzik.org (Jeff Garzik) Date: Thu, 12 Jun 2008 16:22:24 -0400 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <71d336490806120829o7cac67bes384c305e03b0d746@mail.gmail.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <71d336490806120829o7cac67bes384c305e03b0d746@mail.gmail.com> Message-ID: <48518580.7010206@garzik.org> Ramachandra K wrote: > If you are referring to IP address configuration etc, users can configure > the interfaces by setting up ifcfg files and the interfaces are automatically > configured when they are registered. That makes a lot of assumptions about the Linux distribution and userspace setup, assumptions that may not be true for all Linux. Jeff From sashak at voltaire.com Thu Jun 12 13:51:57 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 23:51:57 +0300 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080612205157.GZ6256@sashak.voltaire.com> On 06:53 Wed 11 Jun , Hal Rosenstock wrote: > ibsim: Support for short RMPP packets (up to 256 bytes total) > > If this is acceptable, I'll follow this up with long packet support. RMPP support is acceptable and needed :). I would suggest to discuss design aspects. I had some experimental "RMPP support" (actually - "long packets support") for ibsim, for doing this workable I switched to stream sockets. Finally I saw bad performance numbers with huge queries. Some comments about this patch are below. [snip...] > diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c > index 675d95b..eef0bab > --- a/ibsim/sim_mad.c > +++ b/ibsim/sim_mad.c > @@ -1,5 +1,6 @@ > /* > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This file is part of ibsim. > * > @@ -1128,14 +1129,14 @@ Smpfn *get_handle_fn(ib_rpc_t rpc, int response) > return fn; > } > > - return 0; // No MGTCLASS matched . > + return 0; // No MGTCLASS matched. > } > > int process_packet(Client * cl, void *p, int size, Client ** dcl) > { > struct sim_request *r = p; > Port *port; > - uint8_t data[256]; > + uint8_t data[MAD_BLOCK_SIZE]; > int status, tlid, tqp; > int response; > Smpfn *fn; > @@ -1144,9 +1145,10 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > > *dcl = cl; > > - DEBUG("client %d, size %d", cl->id, size); > - if (size != sizeof(*r)) { > - IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); > + DEBUG("client %d size %d", cl->id, size); > + if (size > sizeof(*r) + MAD_BLOCK_SIZE) { What about partial write? The socket is non-blocked datagram. > + IBWARN("unsupported packet size %d (> %zu)", size, > + sizeof(*r) + MAD_BLOCK_SIZE); > return -1; > } > > @@ -1183,7 +1185,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > VERB("forward pkt to client %d pid %d attr %d", > (*dcl)->id, (*dcl)->pid, rpc.attr.id); > forward_MAD(r->mad, &rpc, &path); > - return sizeof(*r); // forward only > + return size; // forward only > } > > if (port->errrate && (random() % 100) < port->errrate) { > @@ -1214,12 +1216,12 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > VERB("PKT roll back did not succeed"); > goto _dropped; > } > - return sizeof(*r); > + return sizeof(*r) + MAD_BLOCK_SIZE; > > _dropped: > r->status = htonl(110); > *dcl = cl; > - return sizeof(*r); > + return sizeof(*r) + MAD_BLOCK_SIZE; > } > > static int encode_trap128(Port * port, char *data) > @@ -1279,7 +1281,7 @@ static int encode_trap_header(char *buf) > > int send_trap(Port * port, int trapnum) > { > - struct sim_request req; > + struct sim_req256 req; > Client *cl; > int ret, lid = port->lid; > char *data = req.mad + 64; /* data offset */ > diff --git a/include/ibsim.h b/include/ibsim.h > index 84568e6..14a3f90 100644 > --- a/include/ibsim.h > +++ b/include/ibsim.h > @@ -1,5 +1,6 @@ > /* > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This file is part of ibsim. > * > @@ -61,6 +62,8 @@ struct sim_port { > #define SIM_MAGIC 0xdeadbeef > #define SIM_CTL_MAX_DATA 64 > > +#define MAD_BLOCK_SIZE 256 > + > struct sim_request { > uint32_t dlid; > uint32_t slid; > @@ -68,7 +71,17 @@ struct sim_request { > uint32_t sqp; > uint32_t status; > uint64_t context; > - char mad[256]; > + char mad[0]; > +}; > + > +struct sim_req256 { > + uint32_t dlid; > + uint32_t slid; > + uint32_t dqp; > + uint32_t sqp; > + uint32_t status; > + uint64_t context; > + char mad[MAD_BLOCK_SIZE]; > }; Why do we need two separate structures? > > enum SIM_CTL_TYPES { > diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c > index 4cbf8da..9c69fb5 100644 > --- a/umad2sim/umad2sim.c > +++ b/umad2sim/umad2sim.c > @@ -1,5 +1,6 @@ > /* > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This file is part of ibsim. > * > @@ -376,7 +377,7 @@ static int dev_sysfs_create(struct umad2sim_dev *dev) > > static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > { > - struct sim_request req; > + struct sim_req256 req; > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > unsigned mgmt_class; > int cnt; > @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); > DEBUG("umad2sim_read: got %d...\n", cnt); > - if (cnt < sizeof(req)) { > - ERROR("umad2sim_read: partial request - skip.\n"); > - umad->status = EAGAIN; > - return umad_size(); > - } > > mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); This checked that we got at least req header before decode this. Why is it removed? > @@ -411,7 +407,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > umad->status = ntohl(req.status); > umad->timeout_ms = 0; > umad->retries = 0; > - umad->length = umad_size() + sizeof(req.mad); > + umad->length = umad_size() + cnt; 'cnt' is normally actual MAD length + sim_request header size. Is it correct to put it as is in umad->length (shouldn't be umad_size() + cnt - sizeof(struct sim_request))? > > umad->addr.qpn = req.sqp; > umad->addr.qkey = 0; // agent->qkey; > @@ -431,9 +427,9 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > static ssize_t umad2sim_write(struct umad2sim_dev *dev, > const void *buf, size_t count) > { > - struct sim_request req; > + struct sim_request *req; > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > - int cnt; > + int cnt, ocnt; > > #ifdef SIMULATE_SEND_ERRORS > { static int err_count; > @@ -464,25 +460,32 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, > mad_get_field(umad_get_mad(umad), 0, IB_MAD_ATTRMOD_F) > ); > > - req.dlid = umad->addr.lid; > - req.slid = req.dlid == 0xffff ? 0xffff : 0; /* 0 - means auto > + cnt = count - umad_size(); > + if (cnt > MAD_BLOCK_SIZE) > + cnt = MAD_BLOCK_SIZE; > + req = malloc(sizeof(*req) + cnt); So why malloc() if size is limited by struct sim_request anyway? What was a reason to change 'struct sim_request r' to 'struct sim_request *r'? Sasha > + if (!req) { > + ERROR("umad2sim_write: no mem for sim req: %m"); > + return -1; > + } > + > + req->dlid = umad->addr.lid; > + req->slid = req->dlid == 0xffff ? 0xffff : 0; /* 0 - means auto > (supported by ibsim) */ ; > - req.dqp = umad->addr.qpn; > - req.sqp = htonl(dev->agents[umad->agent_id].qpn); > - req.status = 0; > - req.context = 0; > + req->dqp = umad->addr.qpn; > + req->sqp = htonl(dev->agents[umad->agent_id].qpn); > + req->status = 0; > + req->context = 0; > > - cnt = count - umad_size(); > - if (cnt > sizeof(req.mad)) > - cnt = sizeof(req.mad); > - memcpy(req.mad, umad_get_mad(umad), cnt); > + memcpy(req->mad, umad_get_mad(umad), cnt); > > - cnt = write(dev->sim_client.fd_pktout, (void *)&req, sizeof(req)); > - if (cnt < 0) { > + ocnt = write(dev->sim_client.fd_pktout, req, sizeof(*req) + cnt); > + free(req); > + if (ocnt < 0) { > ERROR("umad2sim_write: cannot write\n"); > return -1; > } > - if (cnt < sizeof(req)) { > + if (ocnt < sizeof(*req) + cnt) { > ERROR("umad2sim_write: partial write\n"); > } > > > From sashak at voltaire.com Thu Jun 12 13:57:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Jun 2008 23:57:50 +0300 Subject: [ofa-general] Re: [Announce] libopensmskummeeplugin; OpenSM/PerfMgr to MySQL plugin In-Reply-To: <20080515133456.7b1e5c04.weiny2@llnl.gov> References: <20080515133456.7b1e5c04.weiny2@llnl.gov> Message-ID: <20080612205750.GA6256@sashak.voltaire.com> Hi Ira, On 13:34 Thu 15 May , Ira Weiny wrote: > Announcing, libopensmskummeeplugin. > > https://computing.llnl.gov/linux/skummeeplugin.html Does it use SCM (git :))? What is a patch acception procedure? Sasha From amar.mudrankit at qlogic.com Thu Jun 12 14:09:51 2008 From: amar.mudrankit at qlogic.com (Amar Mudrankit) Date: Fri, 13 Jun 2008 02:39:51 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <485148DA.2060508@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> <485148DA.2060508@trash.net> Message-ID: On Thu, Jun 12, 2008 at 9:33 PM, Patrick McHardy wrote: > Ramachandra K wrote: >> >> On Wed, Jun 11, 2008 at 12:17 PM, Patrick McHardy wrote: >>> >>> Since I didn't receive anything but a "its too complex" in >>> response to my previous question, let me ask again: why are >>> you inventing a new sysfs interface (combined with ~15 module >>> parameters) instead of using the existing standard interfaces >>> for this? >> >> Can you please elaborate on the standard interfaces you are referring to ? >> >> Also as Roland mentioned in the other thread, echo-ing the parameters >> to sysfs files >> is a pretty straightforward way to configure the interfaces. Users >> can setup configuration files to configure QLogic VNIC interfaces >> specifying the name of the interface, various parameters of the >> QLogic EVIC gateway target they want to use and a simple user level >> script parses the configuration file and echoes the parameters to the >> driver. > > > We introduced the rtnl_link API to avoid having each driver > come up with its own method of creating virtual network devices. > > Besides the fact that it is the interface intended to be used > for this kind of thing and the advantages mentioned in my > response to Roland, looking at your interface, there are a > few more advantages from integrating this cleanly in iproute: > I have just started on getting into the rtnl_link API. There are few queries on it, plz correct me if I am wrong. 1. If I understand it correctly, this interface is not text based. Hence, it looks like this interface needs a separate user program which actually sends the parameters to the driver. A separate user program for each driver, as their configuration parameters might be different. Is it right? 2. I am not sure if there exists a standardized (netlink?) interface through which parameters can be passed to kernel. If so, there would be standard set of parameters that can be configured and standard set of statistics values that can be retrieved from driver. Is it so? Actually, I tried googling around rtnl_link, but couldn't collect sufficient information. 3. Is there any kind of documentation available on this? I tried searching on it in Documentation/ directory as well as git log but could not locate enough of it. 4. Can anybody point to an ethernet driver as an example of implementation of rtnl_link API along with an equivalent user space program to configure it? That would be of immense help. >> +echo -n 00066a01de000037 > >> /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_guid >> +echo -n fe8000000000000100066a11de000037 > >> /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/dgid >> +echo -n ffff > >> /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/pkey > > I guess life would be easier for users if those parameters can > be generated in userspace or at least looked up in something > like the maps in /etc/iproute2/. > Is it the answer of question no 2 above? If so, what about statistics of the driver then? > In any case, things like tx_csum and rx_csum definitely do > not belong in this interface. > We are planning to give ethtool support for the driver in future, in which case these 2 parameters will be configured through ethtool. Thanks and Regards, Amar From swise at opengridcomputing.com Thu Jun 12 14:13:02 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 12 Jun 2008 16:13:02 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. Message-ID: <20080612211302.25120.41155.stgit@dell3.ogc.int> - Add a new rdma ctl command called RDMA_GET_MIB to the low level driver to obtain the protocol mib from the hardware. - Add a sysfs entry to allow dumping these stats. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 56 +++++++++++++++++++++++++++ drivers/net/cxgb3/cxgb3_ctl_defs.h | 48 +++++++++++++++++++++++ drivers/net/cxgb3/cxgb3_offload.c | 7 +++ 3 files changed, 110 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 480a2da..f71a103 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1192,17 +1192,71 @@ static ssize_t show_board(struct device *dev, struct device_attribute *attr, return sprintf(buf, "%x.%x\n", iwch_dev->rdev.rnic_info.pdev->vendor, iwch_dev->rdev.rnic_info.pdev->device); } +static ssize_t show_stats(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct iwch_dev *iwch_dev = container_of(dev, struct iwch_dev, + ibdev.dev); + struct rdma_mib_stats m; + int ret; + ssize_t len = 0; + + PDBG("%s dev 0x%p\n", __func__, dev); + ret = iwch_dev->rdev.t3cdev_p->ctl(iwch_dev->rdev.t3cdev_p, + RDMA_GET_MIB, &m); + if (ret) + return sprintf(buf, "Not Available\n"); + +#define MIB32(s, field) len += sprintf(buf + len, "%-18s %u\n", s, m.field) +#define MIB64(s, hi, lo) \ + len += sprintf(buf + len, "%-18s %llu\n", s, \ + (unsigned long long)m.hi + m.lo) + + MIB64("IPInReceives:", ipInReceive_hi, ipInReceive_lo); + MIB64("IPInHdrErrors:", ipInHdrErrors_hi, ipInHdrErrors_lo); + MIB64("IPInAddrErrors:", ipInAddrErrors_hi, ipInAddrErrors_lo); + MIB64("IPInUnknownProtos:", ipInUnknownProtos_hi, + ipInUnknownProtos_lo); + MIB64("IPInDiscards:", ipInDiscards_hi, ipInDiscards_lo); + MIB64("IPInDelivers:", ipInDelivers_hi, ipInDelivers_lo); + MIB64("IPOutRequests:", ipOutRequests_hi, ipOutRequests_lo); + MIB64("IPOutDiscards:", ipOutDiscards_hi, ipOutDiscards_lo); + MIB64("IPOutNoRoutes:", ipOutNoRoutes_hi, ipOutNoRoutes_lo); + MIB32("IPReasmTimeout:", ipReasmTimeout); + MIB32("IPReasmReqds:", ipReasmReqds); + MIB32("IPReasmOKs:", ipReasmOKs); + MIB32("IPReasmFails:", ipReasmFails); + MIB32("TCPActiveOpens:", tcpActiveOpens); + MIB32("TCPPassiveOpens:", tcpPassiveOpens); + MIB32("TCPAttemptFails:", tcpAttemptFails); + MIB32("TCPEstabResets:", tcpEstabResets); + MIB32("TCPOutRsts:", tcpOutRsts); + MIB32("TCPCurrEstab:", tcpCurrEstab); + MIB64("TCPInSegs:", tcpInSegs_hi, tcpInSegs_lo); + MIB64("TCPOutSegs:", tcpOutSegs_hi, tcpOutSegs_lo); + MIB64("TCPRetransSeg:", tcpRetransSeg_hi, tcpRetransSeg_lo); + MIB64("TCPInErrs:", tcpInErrs_hi, tcpInErrs_lo); + MIB32("TCPRtoMin:", tcpRtoMin); + MIB32("TCPRtoMax:", tcpRtoMax); + +#undef MIB32 +#undef MIB64 + + return len; +} static DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL); static DEVICE_ATTR(fw_ver, S_IRUGO, show_fw_ver, NULL); static DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL); static DEVICE_ATTR(board_id, S_IRUGO, show_board, NULL); +static DEVICE_ATTR(stats, S_IRUGO, show_stats, NULL); static struct device_attribute *iwch_class_attributes[] = { &dev_attr_hw_rev, &dev_attr_fw_ver, &dev_attr_hca_type, - &dev_attr_board_id + &dev_attr_board_id, + &dev_attr_stats }; int iwch_register_device(struct iwch_dev *dev) diff --git a/drivers/net/cxgb3/cxgb3_ctl_defs.h b/drivers/net/cxgb3/cxgb3_ctl_defs.h index 6c4f320..2dffc2e 100644 --- a/drivers/net/cxgb3/cxgb3_ctl_defs.h +++ b/drivers/net/cxgb3/cxgb3_ctl_defs.h @@ -54,6 +54,7 @@ enum { RDMA_CQ_DISABLE = 16, RDMA_CTRL_QP_SETUP = 17, RDMA_GET_MEM = 18, + RDMA_GET_MIB = 19, GET_RX_PAGE_INFO = 50, }; @@ -171,4 +172,51 @@ struct ofld_page_info { unsigned int page_size; /* Page size, should be a power of 2 */ unsigned int num; /* Number of pages */ }; + +/* + * Offload RDMA protocol stats. + */ +struct rdma_mib_stats { + u32 ipInReceive_hi; + u32 ipInReceive_lo; + u32 ipInHdrErrors_hi; + u32 ipInHdrErrors_lo; + u32 ipInAddrErrors_hi; + u32 ipInAddrErrors_lo; + u32 ipInUnknownProtos_hi; + u32 ipInUnknownProtos_lo; + u32 ipInDiscards_hi; + u32 ipInDiscards_lo; + u32 ipInDelivers_hi; + u32 ipInDelivers_lo; + u32 ipOutRequests_hi; + u32 ipOutRequests_lo; + u32 ipOutDiscards_hi; + u32 ipOutDiscards_lo; + u32 ipOutNoRoutes_hi; + u32 ipOutNoRoutes_lo; + u32 ipReasmTimeout; + u32 ipReasmReqds; + u32 ipReasmOKs; + u32 ipReasmFails; + + u32 reserved[8]; + + u32 tcpActiveOpens; + u32 tcpPassiveOpens; + u32 tcpAttemptFails; + u32 tcpEstabResets; + u32 tcpOutRsts; + u32 tcpCurrEstab; + u32 tcpInSegs_hi; + u32 tcpInSegs_lo; + u32 tcpOutSegs_hi; + u32 tcpOutSegs_lo; + u32 tcpRetransSeg_hi; + u32 tcpRetransSeg_lo; + u32 tcpInErrs_hi; + u32 tcpInErrs_lo; + u32 tcpRtoMin; + u32 tcpRtoMax; +}; #endif /* _CXGB3_OFFLOAD_CTL_DEFS_H */ diff --git a/drivers/net/cxgb3/cxgb3_offload.c b/drivers/net/cxgb3/cxgb3_offload.c index ff9c013..1884238 100644 --- a/drivers/net/cxgb3/cxgb3_offload.c +++ b/drivers/net/cxgb3/cxgb3_offload.c @@ -303,6 +303,12 @@ static int cxgb_rdma_ctl(struct adapter *adapter, unsigned int req, void *data) spin_unlock_irq(&adapter->sge.reg_lock); break; } + case RDMA_GET_MIB: { + spin_lock(&adapter->stats_lock); + t3_tp_get_mib_stats(adapter, (struct rdma_mib_stats *)data); + spin_unlock(&adapter->stats_lock); + break; + } default: ret = -EOPNOTSUPP; } @@ -381,6 +387,7 @@ static int cxgb_offload_ctl(struct t3cdev *tdev, unsigned int req, void *data) case RDMA_CQ_DISABLE: case RDMA_CTRL_QP_SETUP: case RDMA_GET_MEM: + case RDMA_GET_MIB: if (!offload_running(adapter)) return -EAGAIN; return cxgb_rdma_ctl(adapter, req, data); From or.gerlitz at gmail.com Thu Jun 12 14:15:30 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Fri, 13 Jun 2008 00:15:30 +0300 Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, In-Reply-To: <48517AAD.5070303@mellanox.com> References: <48517AAD.5070303@mellanox.com> Message-ID: <15ddcffd0806121415t6351850dmd6cc6984391301ff@mail.gmail.com> On 6/12/08, Vu Pham wrote: > MLX4 transport (connectX HCA) need 2 extra S/G entries per work request. > One is for the header and one for the checksum at the end. Hi Vu, Can you please clarify what header you refer to, is it one of the IB L4/L3/L2 ones? also what checksum is this? Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From or.gerlitz at gmail.com Thu Jun 12 14:16:02 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Fri, 13 Jun 2008 00:16:02 +0300 Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, In-Reply-To: References: <48517AAD.5070303@mellanox.com> Message-ID: <15ddcffd0806121416p7cfd5db1yca2714a6822dd7d0@mail.gmail.com> On 6/12/08, James Lentini wrote: > Is this already fixed: > http://lists.openfabrics.org/pipermail/general/2008-May/050256.html this patch is to mthca not mlx4 Or. From xma at us.ibm.com Thu Jun 12 14:20:21 2008 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 12 Jun 2008 14:20:21 -0700 Subject: [ofa-general] mlx4_core CQ overrun In-Reply-To: Message-ID: Hello, Anybody saw mlx4_core CQ overrun before? The test is based on OFED-1.3. FW version is 2.3.0. Please let me know any more info is needed. c955mgrs1:~ # dsh -av "grep 'CQ overrun' /var/log/messages" | sort dsh: c955c2s1.ppd.pok.ibm.com Host is not responding. No command will be issued to this host c955c1s11.ppd.pok.ibm.com: Jun 10 07:18:15 c955c1s11 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 000098 c955c1s13.ppd.pok.ibm.com: Jun 9 21:38:12 c955c1s13 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 00008f c955c1s14.ppd.pok.ibm.com: Jun 9 04:44:07 c955c1s14 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 00008a c955c1s2.ppd.pok.ibm.com: Jun 5 10:48:30 c955c1s2 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 000086 c955c1s5.ppd.pok.ibm.com: Jun 4 08:45:53 c955c1s5 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 00008a c955c2s12.ppd.pok.ibm.com: Jun 4 17:54:31 c955c2s12 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 000088 c955c2s12.ppd.pok.ibm.com: Jun 9 10:48:43 c955c2s12 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 0000a3 c955c2s13.ppd.pok.ibm.com: Jun 4 14:29:01 c955c2s13 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 000089 c955c2s2.ppd.pok.ibm.com: Jun 6 17:23:48 c955c2s2 kernel: mlx4_core 0003:01:00.0: CQ overrun on CQN 00008c Thanks Shirley From jlentini at netapp.com Thu Jun 12 14:34:34 2008 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 Jun 2008 17:34:34 -0400 (EDT) Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, In-Reply-To: <15ddcffd0806121416p7cfd5db1yca2714a6822dd7d0@mail.gmail.com> References: <48517AAD.5070303@mellanox.com> <15ddcffd0806121416p7cfd5db1yca2714a6822dd7d0@mail.gmail.com> Message-ID: On Fri, 13 Jun 2008, Or Gerlitz wrote: > On 6/12/08, James Lentini wrote: > > Is this already fixed: > > http://lists.openfabrics.org/pipermail/general/2008-May/050256.html > > this patch is to mthca not mlx4 > > Or. > Keep reading through the thread. Tom Talpey reported the same problem with mlx4 and Roland created a patch to fix it. From hrosenstock at xsigo.com Thu Jun 12 14:40:15 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 12 Jun 2008 14:40:15 -0700 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <20080612205157.GZ6256@sashak.voltaire.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> Message-ID: <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 23:51 +0300, Sasha Khapyorsky wrote: > On 06:53 Wed 11 Jun , Hal Rosenstock wrote: > > ibsim: Support for short RMPP packets (up to 256 bytes total) > > > > If this is acceptable, I'll follow this up with long packet support. > > RMPP support is acceptable and needed :). I would suggest to discuss > design aspects. I had some experimental "RMPP support" (actually - "long > packets support") for ibsim, for doing this workable I switched to > stream sockets. Finally I saw bad performance numbers with huge queries. > > Some comments about this patch are below. > > [snip...] > > diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c > > index 675d95b..eef0bab > > --- a/ibsim/sim_mad.c > > +++ b/ibsim/sim_mad.c > > @@ -1,5 +1,6 @@ > > /* > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This file is part of ibsim. > > * > > @@ -1128,14 +1129,14 @@ Smpfn *get_handle_fn(ib_rpc_t rpc, int response) > > return fn; > > } > > > > - return 0; // No MGTCLASS matched . > > + return 0; // No MGTCLASS matched. > > } > > > > int process_packet(Client * cl, void *p, int size, Client ** dcl) > > { > > struct sim_request *r = p; > > Port *port; > > - uint8_t data[256]; > > + uint8_t data[MAD_BLOCK_SIZE]; > > int status, tlid, tqp; > > int response; > > Smpfn *fn; > > @@ -1144,9 +1145,10 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > > > > *dcl = cl; > > > > - DEBUG("client %d, size %d", cl->id, size); > > - if (size != sizeof(*r)) { > > - IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); > > + DEBUG("client %d size %d", cl->id, size); > > + if (size > sizeof(*r) + MAD_BLOCK_SIZE) { > > What about partial write? This change was needed for short RMPP. > The socket is non-blocked datagram. Is a framing layer needed ? > > + IBWARN("unsupported packet size %d (> %zu)", size, > > + sizeof(*r) + MAD_BLOCK_SIZE); > > return -1; > > } > > > > @@ -1183,7 +1185,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > > VERB("forward pkt to client %d pid %d attr %d", > > (*dcl)->id, (*dcl)->pid, rpc.attr.id); > > forward_MAD(r->mad, &rpc, &path); > > - return sizeof(*r); // forward only > > + return size; // forward only > > } > > > > if (port->errrate && (random() % 100) < port->errrate) { > > @@ -1214,12 +1216,12 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > > VERB("PKT roll back did not succeed"); > > goto _dropped; > > } > > - return sizeof(*r); > > + return sizeof(*r) + MAD_BLOCK_SIZE; > > > > _dropped: > > r->status = htonl(110); > > *dcl = cl; > > - return sizeof(*r); > > + return sizeof(*r) + MAD_BLOCK_SIZE; > > } > > > > static int encode_trap128(Port * port, char *data) > > @@ -1279,7 +1281,7 @@ static int encode_trap_header(char *buf) > > > > int send_trap(Port * port, int trapnum) > > { > > - struct sim_request req; > > + struct sim_req256 req; > > Client *cl; > > int ret, lid = port->lid; > > char *data = req.mad + 64; /* data offset */ > > diff --git a/include/ibsim.h b/include/ibsim.h > > index 84568e6..14a3f90 100644 > > --- a/include/ibsim.h > > +++ b/include/ibsim.h > > @@ -1,5 +1,6 @@ > > /* > > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This file is part of ibsim. > > * > > @@ -61,6 +62,8 @@ struct sim_port { > > #define SIM_MAGIC 0xdeadbeef > > #define SIM_CTL_MAX_DATA 64 > > > > +#define MAD_BLOCK_SIZE 256 > > + > > struct sim_request { > > uint32_t dlid; > > uint32_t slid; > > @@ -68,7 +71,17 @@ struct sim_request { > > uint32_t sqp; > > uint32_t status; > > uint64_t context; > > - char mad[256]; > > + char mad[0]; > > +}; > > + > > +struct sim_req256 { > > + uint32_t dlid; > > + uint32_t slid; > > + uint32_t dqp; > > + uint32_t sqp; > > + uint32_t status; > > + uint64_t context; > > + char mad[MAD_BLOCK_SIZE]; > > }; > > Why do we need two separate structures? The second one was just to eliminate the memory allocation (and use the stack). Only forwarded packets have the variability in size. > > enum SIM_CTL_TYPES { > > diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c > > index 4cbf8da..9c69fb5 100644 > > --- a/umad2sim/umad2sim.c > > +++ b/umad2sim/umad2sim.c > > @@ -1,5 +1,6 @@ > > /* > > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This file is part of ibsim. > > * > > @@ -376,7 +377,7 @@ static int dev_sysfs_create(struct umad2sim_dev *dev) > > > > static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > { > > - struct sim_request req; > > + struct sim_req256 req; > > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > > unsigned mgmt_class; > > int cnt; > > @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); > > DEBUG("umad2sim_read: got %d...\n", cnt); > > - if (cnt < sizeof(req)) { > > - ERROR("umad2sim_read: partial request - skip.\n"); > > - umad->status = EAGAIN; > > - return umad_size(); > > - } > > > > mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); > > This checked that we got at least req header before decode this. No; I think it checked that the entire packet existed as req was the entire packet not just the header. > Why is it removed? Because of short RMPP. It can add back at least the header check part but I think the partial write (long writes) is the fundamental issue. > > @@ -411,7 +407,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > umad->status = ntohl(req.status); > > umad->timeout_ms = 0; > > umad->retries = 0; > > - umad->length = umad_size() + sizeof(req.mad); > > + umad->length = umad_size() + cnt; > > 'cnt' is normally actual MAD length + sim_request header size. Is it > correct to put it as is in umad->length (shouldn't be umad_size() + cnt > - sizeof(struct sim_request))? Not sure about this. I'll need to get back on this. > > > > umad->addr.qpn = req.sqp; > > umad->addr.qkey = 0; // agent->qkey; > > @@ -431,9 +427,9 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > static ssize_t umad2sim_write(struct umad2sim_dev *dev, > > const void *buf, size_t count) > > { > > - struct sim_request req; > > + struct sim_request *req; > > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > > - int cnt; > > + int cnt, ocnt; > > > > #ifdef SIMULATE_SEND_ERRORS > > { static int err_count; > > @@ -464,25 +460,32 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, > > mad_get_field(umad_get_mad(umad), 0, IB_MAD_ATTRMOD_F) > > ); > > > > - req.dlid = umad->addr.lid; > > - req.slid = req.dlid == 0xffff ? 0xffff : 0; /* 0 - means auto > > + cnt = count - umad_size(); > > + if (cnt > MAD_BLOCK_SIZE) > > + cnt = MAD_BLOCK_SIZE; > > + req = malloc(sizeof(*req) + cnt); > > So why malloc() if size is limited by struct sim_request anyway? Because long RMPP isn't handled yet. > What was a reason to change 'struct sim_request r' to 'struct sim_request > *r'? Because sim_request is now only the header and the memory needs to be malloc'd. It's to get ready for more flexible packet sizes (long RMPP) as only the cnt limiting would be removed here. Also, short RMPP cnts are smaller than MAD_BLOCK_SIZE. -- Hal > Sasha > > > + if (!req) { > > + ERROR("umad2sim_write: no mem for sim req: %m"); > > + return -1; > > + } > > + > > + req->dlid = umad->addr.lid; > > + req->slid = req->dlid == 0xffff ? 0xffff : 0; /* 0 - means auto > > (supported by ibsim) */ ; > > - req.dqp = umad->addr.qpn; > > - req.sqp = htonl(dev->agents[umad->agent_id].qpn); > > - req.status = 0; > > - req.context = 0; > > + req->dqp = umad->addr.qpn; > > + req->sqp = htonl(dev->agents[umad->agent_id].qpn); > > + req->status = 0; > > + req->context = 0; > > > > - cnt = count - umad_size(); > > - if (cnt > sizeof(req.mad)) > > - cnt = sizeof(req.mad); > > - memcpy(req.mad, umad_get_mad(umad), cnt); > > + memcpy(req->mad, umad_get_mad(umad), cnt); > > > > - cnt = write(dev->sim_client.fd_pktout, (void *)&req, sizeof(req)); > > - if (cnt < 0) { > > + ocnt = write(dev->sim_client.fd_pktout, req, sizeof(*req) + cnt); > > + free(req); > > + if (ocnt < 0) { > > ERROR("umad2sim_write: cannot write\n"); > > return -1; > > } > > - if (cnt < sizeof(req)) { > > + if (ocnt < sizeof(*req) + cnt) { > > ERROR("umad2sim_write: partial write\n"); > > } > > > > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From vu at mellanox.com Thu Jun 12 14:56:55 2008 From: vu at mellanox.com (Vu Pham) Date: Thu, 12 Jun 2008 14:56:55 -0700 Subject: [ofa-general] [PATCH] nfsrdma: fix connection/qp creation for mlx4, In-Reply-To: References: <48517AAD.5070303@mellanox.com> <15ddcffd0806121416p7cfd5db1yca2714a6822dd7d0@mail.gmail.com> Message-ID: <48519BA7.9000906@mellanox.com> > On Fri, 13 Jun 2008, Or Gerlitz wrote: > > >> On 6/12/08, James Lentini wrote: >> >>> Is this already fixed: >>> http://lists.openfabrics.org/pipermail/general/2008-May/050256.html >>> >> this patch is to mthca not mlx4 >> >> Or. >> >> > > Keep reading through the thread. Tom Talpey reported the same problem > with mlx4 and Roland created a patch to fix it. > Roland post this fix for mlx4 - http://lists.openfabrics.org/pipermail/general/2008-May/050570.html Thanks James! Please ignore my original patch From swise at opengridcomputing.com Thu Jun 12 15:04:43 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 12 Jun 2008 17:04:43 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support Message-ID: <20080612220443.3491.22057.stgit@dell3.ogc.int> - set IB_DEVICE_MEM_MGT_EXTENSIONS capability bit if fw supports it. - set max_fast_reg_page_list_len device attribute. - add iwch_alloc_fast_reg_mr function. - add iwch_alloc_fastreg_pbl - add iwch_free_fastreg_pbl - adjust the WQ depth for kernel mode work queues to account for fastreg possibly taking 2 WR slots. - add fastreg_mr work request support. - add local_inv work request support. - add send_with_inv and send_with_se_inv work request support. - removed useless duplicate enums/defines for TPT/MW/MR stuff. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 13 ++- drivers/infiniband/hw/cxgb3/cxio_hal.h | 1 drivers/infiniband/hw/cxgb3/cxio_wr.h | 60 +++++++++---- drivers/infiniband/hw/cxgb3/iwch_cq.c | 14 ++- drivers/infiniband/hw/cxgb3/iwch_provider.c | 98 ++++++++++++++++++++- drivers/infiniband/hw/cxgb3/iwch_provider.h | 8 -- drivers/infiniband/hw/cxgb3/iwch_qp.c | 129 ++++++++++++++++++--------- 7 files changed, 248 insertions(+), 75 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 3f441fc..4c0e73a 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -145,7 +145,7 @@ static int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev_p, u32 qpid) } wqe = (struct t3_modify_qp_wr *) skb_put(skb, sizeof(*wqe)); memset(wqe, 0, sizeof(*wqe)); - build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 3, 0, qpid, 7); + build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 3, 0, qpid, 7, 3); wqe->flags = cpu_to_be32(MODQP_WRITE_EC); sge_cmd = qpid << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); @@ -558,7 +558,7 @@ static int cxio_hal_init_ctrl_qp(struct cxio_rdev *rdev_p) wqe = (struct t3_modify_qp_wr *) skb_put(skb, sizeof(*wqe)); memset(wqe, 0, sizeof(*wqe)); build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 0, 0, - T3_CTL_QP_TID, 7); + T3_CTL_QP_TID, 7, 3); wqe->flags = cpu_to_be32(MODQP_WRITE_EC); sge_cmd = (3ULL << 56) | FW_RI_SGEEC_START << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); @@ -674,7 +674,7 @@ static int cxio_hal_ctrl_qp_write_mem(struct cxio_rdev *rdev_p, u32 addr, build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_BP, flag, Q_GENBIT(rdev_p->ctrl_qp.wptr, T3_CTRL_QP_SIZE_LOG2), T3_CTRL_QP_ID, - wr_len); + wr_len, 3); if (flag == T3_COMPLETION_FLAG) ring_doorbell(rdev_p->ctrl_qp.doorbell, T3_CTRL_QP_ID); len -= 96; @@ -816,6 +816,13 @@ int cxio_deallocate_window(struct cxio_rdev *rdev_p, u32 stag) 0, 0); } +int cxio_allocate_stag(struct cxio_rdev *rdev_p, u32 * stag, u32 pdid, u32 pbl_size, u32 pbl_addr) +{ + *stag = T3_STAG_UNSET; + return __cxio_tpt_op(rdev_p, 0, stag, 0, pdid, TPT_NON_SHARED_MR, + 0, 0, 0ULL, 0, 0, pbl_size, pbl_addr); +} + int cxio_rdma_init(struct cxio_rdev *rdev_p, struct t3_rdma_init_attr *attr) { struct t3_rdma_init_wr *wqe; diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 6e128f6..4bb5926 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -165,6 +165,7 @@ int cxio_reregister_phys_mem(struct cxio_rdev *rdev, u32 * stag, u32 pdid, int cxio_dereg_mem(struct cxio_rdev *rdev, u32 stag, u32 pbl_size, u32 pbl_addr); int cxio_allocate_window(struct cxio_rdev *rdev, u32 * stag, u32 pdid); +int cxio_allocate_stag(struct cxio_rdev *rdev, u32 * stag, u32 pdid, u32 pbl_size, u32 pbl_addr); int cxio_deallocate_window(struct cxio_rdev *rdev, u32 stag); int cxio_rdma_init(struct cxio_rdev *rdev, struct t3_rdma_init_attr *attr); void cxio_register_ev_cb(cxio_hal_ev_callback_func_t ev_cb); diff --git a/drivers/infiniband/hw/cxgb3/cxio_wr.h b/drivers/infiniband/hw/cxgb3/cxio_wr.h index f1a25a8..85421ff 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_wr.h +++ b/drivers/infiniband/hw/cxgb3/cxio_wr.h @@ -72,7 +72,8 @@ enum t3_wr_opcode { T3_WR_BIND = FW_WROPCODE_RI_BIND_MW, T3_WR_RCV = FW_WROPCODE_RI_RECEIVE, T3_WR_INIT = FW_WROPCODE_RI_RDMA_INIT, - T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP + T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP, + T3_WR_FASTREG = FW_WROPCODE_RI_FASTREGISTER_MR } __attribute__ ((packed)); enum t3_rdma_opcode { @@ -89,7 +90,8 @@ enum t3_rdma_opcode { T3_FAST_REGISTER, T3_LOCAL_INV, T3_QP_MOD, - T3_BYPASS + T3_BYPASS, + T3_RDMA_READ_REQ_WITH_INV, } __attribute__ ((packed)); static inline enum t3_rdma_opcode wr2opcode(enum t3_wr_opcode wrop) @@ -170,11 +172,45 @@ struct t3_send_wr { struct t3_sge sgl[T3_MAX_SGE]; /* 4+ */ }; +#define T3_MAX_FASTREG_DEPTH 24 + +struct t3_fastreg_wr { + struct fw_riwrh wrh; /* 0 */ + union t3_wrid wrid; /* 1 */ + __be32 stag; /* 2 */ + __be32 len; + __be32 va_base_hi; /* 3 */ + __be32 va_base_lo_fbo; + __be32 page_type_perms; /* 4 */ + __be32 reserved1; + __be64 pbl_addrs[0]; /* 5+ */ +}; + +#define S_FR_PAGE_COUNT 24 +#define M_FR_PAGE_COUNT 0xff +#define V_FR_PAGE_COUNT(x) ((x) << S_FR_PAGE_COUNT) +#define G_FR_PAGE_COUNT(x) ((((x) >> S_FR_PAGE_COUNT)) & M_FR_PAGE_COUNT) + +#define S_FR_PAGE_SIZE 16 +#define M_FR_PAGE_SIZE 0x1f +#define V_FR_PAGE_SIZE(x) ((x) << S_FR_PAGE_SIZE) +#define G_FR_PAGE_SIZE(x) ((((x) >> S_FR_PAGE_SIZE)) & M_FR_PAGE_SIZE) + +#define S_FR_TYPE 8 +#define M_FR_TYPE 0x1 +#define V_FR_TYPE(x) ((x) << S_FR_TYPE) +#define G_FR_TYPE(x) ((((x) >> S_FR_TYPE)) & M_FR_TYPE) + +#define S_FR_PERMS 0 +#define M_FR_PERMS 0xff +#define V_FR_PERMS(x) ((x) << S_FR_PERMS) +#define G_FR_PERMS(x) ((((x) >> S_FR_PERMS)) & M_FR_PERMS) + struct t3_local_inv_wr { struct fw_riwrh wrh; /* 0 */ union t3_wrid wrid; /* 1 */ __be32 stag; /* 2 */ - __be32 reserved3; + __be32 reserved; }; struct t3_rdma_write_wr { @@ -201,18 +237,6 @@ struct t3_rdma_read_wr { __be64 local_to; /* 5 */ }; -enum t3_addr_type { - T3_VA_BASED_TO = 0x0, - T3_ZERO_BASED_TO = 0x1 -} __attribute__ ((packed)); - -enum t3_mem_perms { - T3_MEM_ACCESS_LOCAL_READ = 0x1, - T3_MEM_ACCESS_LOCAL_WRITE = 0x2, - T3_MEM_ACCESS_REM_READ = 0x4, - T3_MEM_ACCESS_REM_WRITE = 0x8 -} __attribute__ ((packed)); - struct t3_bind_mw_wr { struct fw_riwrh wrh; /* 0 */ union t3_wrid wrid; /* 1 */ @@ -346,6 +370,7 @@ union t3_wr { struct t3_rdma_write_wr write; struct t3_rdma_read_wr read; struct t3_receive_wr recv; + struct t3_fastreg_wr fastreg; struct t3_local_inv_wr local_inv; struct t3_bind_mw_wr bind; struct t3_bypass_wr bypass; @@ -368,10 +393,10 @@ static inline enum t3_wr_opcode fw_riwrh_opcode(struct fw_riwrh *wqe) static inline void build_fw_riwrh(struct fw_riwrh *wqe, enum t3_wr_opcode op, enum t3_wr_flags flags, u8 genbit, u32 tid, - u8 len) + u8 len, u8 sopeop) { wqe->op_seop_flags = cpu_to_be32(V_FW_RIWR_OP(op) | - V_FW_RIWR_SOPEOP(M_FW_RIWR_SOPEOP) | + V_FW_RIWR_SOPEOP(sopeop) | V_FW_RIWR_FLAGS(flags)); wmb(); wqe->gen_tid_len = cpu_to_be32(V_FW_RIWR_GEN(genbit) | @@ -404,6 +429,7 @@ enum tpt_addr_type { }; enum tpt_mem_perm { + TPT_MW_BIND = 0x10, TPT_LOCAL_READ = 0x8, TPT_LOCAL_WRITE = 0x4, TPT_REMOTE_READ = 0x2, diff --git a/drivers/infiniband/hw/cxgb3/iwch_cq.c b/drivers/infiniband/hw/cxgb3/iwch_cq.c index 4ee8ccd..3d7a8e8 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cq.c +++ b/drivers/infiniband/hw/cxgb3/iwch_cq.c @@ -81,6 +81,7 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, wc->wr_id = cookie; wc->qp = &qhp->ibqp; wc->vendor_err = CQE_STATUS(cqe); + wc->wc_flags = 0; PDBG("%s qpid 0x%x type %d opcode %d status 0x%x wrid hi 0x%x " "lo 0x%x cookie 0x%llx\n", __func__, @@ -94,6 +95,10 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, else wc->byte_len = 0; wc->opcode = IB_WC_RECV; + if (CQE_OPCODE(cqe) == T3_SEND_WITH_INV || CQE_OPCODE(cqe) == T3_SEND_WITH_SE_INV) { + wc->ex.invalidate_rkey = CQE_WRID_STAG(cqe); + wc->wc_flags |= IB_WC_WITH_INVALIDATE; + } } else { switch (CQE_OPCODE(cqe)) { case T3_RDMA_WRITE: @@ -105,17 +110,20 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, break; case T3_SEND: case T3_SEND_WITH_SE: + case T3_SEND_WITH_INV: + case T3_SEND_WITH_SE_INV: wc->opcode = IB_WC_SEND; break; case T3_BIND_MW: wc->opcode = IB_WC_BIND_MW; break; - /* these aren't supported yet */ - case T3_SEND_WITH_INV: - case T3_SEND_WITH_SE_INV: case T3_LOCAL_INV: + wc->opcode = IB_WC_LOCAL_INV; + break; case T3_FAST_REGISTER: + wc->opcode = IB_WC_FAST_REG_MR; + break; default: printk(KERN_ERR MOD "Unexpected opcode %d " "in the CQE received for QPID=0x%0x\n", diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 8934178..193dfe7 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -768,6 +768,67 @@ static int iwch_dealloc_mw(struct ib_mw *mw) return 0; } +static struct ib_mr *iwch_alloc_fast_reg_mr(struct ib_pd *pd, int pbl_depth) +{ + struct iwch_dev *rhp; + struct iwch_pd *php; + struct iwch_mr *mhp; + u32 mmid; + u32 stag = 0; + int ret; + + php = to_iwch_pd(pd); + rhp = php->rhp; + mhp = kzalloc(sizeof(*mhp), GFP_KERNEL); + if (!mhp) + return ERR_PTR(-ENOMEM); + + mhp->rhp = rhp; + ret = iwch_alloc_pbl(mhp, pbl_depth); + if (ret) { + kfree(mhp); + return ERR_PTR(ret); + } + mhp->attr.pbl_size = pbl_depth; + ret = cxio_allocate_stag(&rhp->rdev, &stag, php->pdid, mhp->attr.pbl_size, mhp->attr.pbl_addr); + if (ret) { + iwch_free_pbl(mhp); + kfree(mhp); + return ERR_PTR(ret); + } + mhp->attr.pdid = php->pdid; + mhp->attr.type = TPT_NON_SHARED_MR; + mhp->attr.stag = stag; + mhp->attr.state = 1; + mmid = (stag) >> 8; + mhp->ibmr.rkey = mhp->ibmr.lkey = stag; + insert_handle(rhp, &rhp->mmidr, mhp, mmid); + PDBG("%s mmid 0x%x mhp %p stag 0x%x\n", __func__, mmid, mhp, stag); + return &(mhp->ibmr); +} + +static struct ib_fast_reg_page_list *iwch_alloc_fastreg_pbl( + struct ib_device *device, + int page_list_len) +{ + struct ib_fast_reg_page_list *page_list; + + page_list = kmalloc(sizeof *page_list + page_list_len * sizeof(u64), + GFP_KERNEL); + if (!page_list) + return ERR_PTR(-ENOMEM); + + page_list->page_list = (u64 *)(page_list + 1); + page_list->max_page_list_len = page_list_len; + + return page_list; +} + +static void iwch_free_fastreg_pbl(struct ib_fast_reg_page_list *page_list) +{ + kfree(page_list); +} + static int iwch_destroy_qp(struct ib_qp *ib_qp) { struct iwch_dev *rhp; @@ -843,6 +904,15 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, */ sqsize = roundup_pow_of_two(attrs->cap.max_send_wr); wqsize = roundup_pow_of_two(rqsize + sqsize); + + /* + * Kernel users need more wq space for fastreg WRs which can take + * 2 WR fragments. + */ + ucontext = pd->uobject ? to_iwch_ucontext(pd->uobject->context) : NULL; + if (!ucontext && wqsize < (rqsize + (2 * sqsize))) + wqsize = roundup_pow_of_two(rqsize + + roundup_pow_of_two(attrs->cap.max_send_wr * 2)); PDBG("%s wqsize %d sqsize %d rqsize %d\n", __func__, wqsize, sqsize, rqsize); qhp = kzalloc(sizeof(*qhp), GFP_KERNEL); @@ -851,7 +921,6 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, qhp->wq.size_log2 = ilog2(wqsize); qhp->wq.rq_size_log2 = ilog2(rqsize); qhp->wq.sq_size_log2 = ilog2(sqsize); - ucontext = pd->uobject ? to_iwch_ucontext(pd->uobject->context) : NULL; if (cxio_create_qp(&rhp->rdev, !udata, &qhp->wq, ucontext ? &ucontext->uctx : &rhp->rdev.uctx)) { kfree(qhp); @@ -1048,6 +1117,7 @@ static int iwch_query_device(struct ib_device *ibdev, props->max_mr = dev->attr.max_mem_regs; props->max_pd = dev->attr.max_pds; props->local_ca_ack_delay = 0; + props->max_fast_reg_page_list_len = T3_MAX_FASTREG_DEPTH; return 0; } @@ -1088,6 +1158,23 @@ static ssize_t show_rev(struct device *dev, struct device_attribute *attr, return sprintf(buf, "%d\n", iwch_dev->rdev.t3cdev_p->type); } +static int fw_supports_fastreg(struct iwch_dev *iwch_dev) +{ + struct ethtool_drvinfo info; + struct net_device *lldev = iwch_dev->rdev.t3cdev_p->lldev; + char *cp, *next; + unsigned fw_maj, fw_min; + + lldev->ethtool_ops->get_drvinfo(lldev, &info); + next = info.fw_version+1; + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_maj); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_min); + printk("%s maj %u min %u\n", __func__, fw_maj, fw_min); + return (fw_maj >= 6 && fw_min > 0); +} + static ssize_t show_fw_ver(struct device *dev, struct device_attribute *attr, char *buf) { struct iwch_dev *iwch_dev = container_of(dev, struct iwch_dev, @@ -1145,8 +1232,10 @@ int iwch_register_device(struct iwch_dev *dev) memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid)); memcpy(&dev->ibdev.node_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); dev->ibdev.owner = THIS_MODULE; - dev->device_cap_flags = - (IB_DEVICE_ZERO_STAG | IB_DEVICE_MEM_WINDOW); + dev->device_cap_flags = IB_DEVICE_ZERO_STAG | + IB_DEVICE_MEM_WINDOW; + if (fw_supports_fastreg(dev)) + dev->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; dev->ibdev.uverbs_cmd_mask = (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) | @@ -1198,6 +1287,9 @@ int iwch_register_device(struct iwch_dev *dev) dev->ibdev.alloc_mw = iwch_alloc_mw; dev->ibdev.bind_mw = iwch_bind_mw; dev->ibdev.dealloc_mw = iwch_dealloc_mw; + dev->ibdev.alloc_fast_reg_mr = iwch_alloc_fast_reg_mr; + dev->ibdev.alloc_fast_reg_page_list = iwch_alloc_fastreg_pbl; + dev->ibdev.free_fast_reg_page_list = iwch_free_fastreg_pbl; dev->ibdev.attach_mcast = iwch_multicast_attach; dev->ibdev.detach_mcast = iwch_multicast_detach; diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.h b/drivers/infiniband/hw/cxgb3/iwch_provider.h index 836163f..f5ceca0 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.h +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.h @@ -296,14 +296,6 @@ static inline u32 iwch_ib_to_tpt_access(int acc) TPT_LOCAL_READ; } -static inline u32 iwch_ib_to_mwbind_access(int acc) -{ - return (acc & IB_ACCESS_REMOTE_WRITE ? T3_MEM_ACCESS_REM_WRITE : 0) | - (acc & IB_ACCESS_REMOTE_READ ? T3_MEM_ACCESS_REM_READ : 0) | - (acc & IB_ACCESS_LOCAL_WRITE ? T3_MEM_ACCESS_LOCAL_WRITE : 0) | - T3_MEM_ACCESS_LOCAL_READ; -} - enum iwch_mmid_state { IWCH_STAG_STATE_VALID, IWCH_STAG_STATE_INVALID diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 9926137..1f543b9 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -44,54 +44,39 @@ static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, switch (wr->opcode) { case IB_WR_SEND: - case IB_WR_SEND_WITH_IMM: if (wr->send_flags & IB_SEND_SOLICITED) wqe->send.rdmaop = T3_SEND_WITH_SE; else wqe->send.rdmaop = T3_SEND; wqe->send.rem_stag = 0; break; -#if 0 /* Not currently supported */ - case TYPE_SEND_INVALIDATE: - case TYPE_SEND_INVALIDATE_IMMEDIATE: - wqe->send.rdmaop = T3_SEND_WITH_INV; - wqe->send.rem_stag = cpu_to_be32(wr->wr.rdma.rkey); - break; - case TYPE_SEND_SE_INVALIDATE: - wqe->send.rdmaop = T3_SEND_WITH_SE_INV; - wqe->send.rem_stag = cpu_to_be32(wr->wr.rdma.rkey); + case IB_WR_SEND_WITH_INV: + if (wr->send_flags & IB_SEND_SOLICITED) + wqe->send.rdmaop = T3_SEND_WITH_SE_INV; + else + wqe->send.rdmaop = T3_SEND_WITH_INV; + wqe->send.rem_stag = cpu_to_be32(wr->ex.invalidate_rkey); break; -#endif default: - break; + return -EINVAL; } if (wr->num_sge > T3_MAX_SGE) return -EINVAL; wqe->send.reserved[0] = 0; wqe->send.reserved[1] = 0; wqe->send.reserved[2] = 0; - if (wr->opcode == IB_WR_SEND_WITH_IMM) { - plen = 4; - wqe->send.sgl[0].stag = wr->ex.imm_data; - wqe->send.sgl[0].len = __constant_cpu_to_be32(0); - wqe->send.num_sgle = __constant_cpu_to_be32(0); - *flit_cnt = 5; - } else { - plen = 0; - for (i = 0; i < wr->num_sge; i++) { - if ((plen + wr->sg_list[i].length) < plen) { - return -EMSGSIZE; - } - plen += wr->sg_list[i].length; - wqe->send.sgl[i].stag = - cpu_to_be32(wr->sg_list[i].lkey); - wqe->send.sgl[i].len = - cpu_to_be32(wr->sg_list[i].length); - wqe->send.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); + plen = 0; + for (i = 0; i < wr->num_sge; i++) { + if ((plen + wr->sg_list[i].length) < plen) { + return -EMSGSIZE; } - wqe->send.num_sgle = cpu_to_be32(wr->num_sge); - *flit_cnt = 4 + ((wr->num_sge) << 1); + plen += wr->sg_list[i].length; + wqe->send.sgl[i].stag = cpu_to_be32(wr->sg_list[i].lkey); + wqe->send.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); + wqe->send.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); } + wqe->send.num_sgle = cpu_to_be32(wr->num_sge); + *flit_cnt = 4 + ((wr->num_sge) << 1); wqe->send.plen = cpu_to_be32(plen); return 0; } @@ -155,6 +140,56 @@ static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } +static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, + u8 *flit_cnt, int *wr_cnt, struct t3_wq *wq) +{ + int i; + u64 *p; + + if (wr->wr.fast_reg.page_list_len > T3_MAX_FASTREG_DEPTH) + return -EINVAL; + *wr_cnt = 1; + wqe->fastreg.stag = cpu_to_be32(wr->wr.fast_reg.rkey); + wqe->fastreg.len = cpu_to_be32(wr->wr.fast_reg.length); + wqe->fastreg.va_base_hi = cpu_to_be32(wr->wr.fast_reg.iova_start>>32); + wqe->fastreg.va_base_lo_fbo = + cpu_to_be32(wr->wr.fast_reg.iova_start&0xffffffff); + wqe->fastreg.page_type_perms = cpu_to_be32( + V_FR_PAGE_COUNT(wr->wr.fast_reg.page_list_len) | + V_FR_PAGE_SIZE(wr->wr.fast_reg.page_shift-12) | + V_FR_TYPE(TPT_VATO) | + V_FR_PERMS(iwch_ib_to_tpt_access(wr->wr.fast_reg.access_flags))); + p = &wqe->fastreg.pbl_addrs[0]; + for (i=0; iwr.fast_reg.page_list_len; i++, p++) { + + /* If we need a 2nd WR, then set it up */ + if (i == 10) { + *wr_cnt = 2; + wqe = (union t3_wr *)(wq->queue + + Q_PTR2IDX((wq->wptr+1), wq->size_log2)); + build_fw_riwrh((void *)wqe, T3_WR_FASTREG, 0, + Q_GENBIT(wq->wptr, wq->size_log2), + 0, 1 + wr->wr.fast_reg.page_list_len - 10, 1); + + p = &wqe->flit[1]; + } + *p = cpu_to_be64((u64)wr->wr.fast_reg.page_list->page_list[i]); + } + *flit_cnt = 5 + wr->wr.fast_reg.page_list_len; + if (*flit_cnt > 15) + *flit_cnt = 15; + return 0; +} + +static int iwch_build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, + u8 *flit_cnt) +{ + wqe->local_inv.stag = cpu_to_be32(wr->ex.invalidate_rkey); + wqe->local_inv.reserved = 0; + *flit_cnt = sizeof(struct t3_local_inv_wr) >> 3; + return 0; +} + /* * TBD: this is going to be moved to firmware. Missing pdid/qpid check for now. */ @@ -238,6 +273,7 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, u32 num_wrs; unsigned long flag; struct t3_swsq *sqp; + int wr_cnt = 1; qhp = to_iwch_qp(ibqp); spin_lock_irqsave(&qhp->lock, flag); @@ -262,15 +298,15 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, t3_wr_flags = 0; if (wr->send_flags & IB_SEND_SOLICITED) t3_wr_flags |= T3_SOLICITED_EVENT_FLAG; - if (wr->send_flags & IB_SEND_FENCE) - t3_wr_flags |= T3_READ_FENCE_FLAG; if (wr->send_flags & IB_SEND_SIGNALED) t3_wr_flags |= T3_COMPLETION_FLAG; sqp = qhp->wq.sq + Q_PTR2IDX(qhp->wq.sq_wptr, qhp->wq.sq_size_log2); switch (wr->opcode) { case IB_WR_SEND: - case IB_WR_SEND_WITH_IMM: + case IB_WR_SEND_WITH_INV: + if (wr->send_flags & IB_SEND_FENCE) + t3_wr_flags |= T3_READ_FENCE_FLAG; t3_wr_opcode = T3_WR_SEND; err = iwch_build_rdma_send(wqe, wr, &t3_wr_flit_cnt); break; @@ -289,6 +325,17 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, if (!qhp->wq.oldest_read) qhp->wq.oldest_read = sqp; break; + case IB_WR_FAST_REG_MR: + t3_wr_opcode = T3_WR_FASTREG; + err = iwch_build_fastreg(wqe, wr, &t3_wr_flit_cnt, + &wr_cnt, &qhp->wq); + break; + case IB_WR_LOCAL_INV: + if (wr->send_flags & IB_SEND_FENCE) + t3_wr_flags |= T3_LOCAL_FENCE_FLAG; + t3_wr_opcode = T3_WR_INV_STAG; + err = iwch_build_inv_stag(wqe, wr, &t3_wr_flit_cnt); + break; default: PDBG("%s post of type=%d TBD!\n", __func__, wr->opcode); @@ -307,14 +354,14 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, build_fw_riwrh((void *) wqe, t3_wr_opcode, t3_wr_flags, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), - 0, t3_wr_flit_cnt); + 0, t3_wr_flit_cnt, (wr_cnt == 1) ? 3 : 2); PDBG("%s cookie 0x%llx wq idx 0x%x swsq idx %ld opcode %d\n", __func__, (unsigned long long) wr->wr_id, idx, Q_PTR2IDX(qhp->wq.sq_wptr, qhp->wq.sq_size_log2), sqp->opcode); wr = wr->next; num_wrs--; - ++(qhp->wq.wptr); + qhp->wq.wptr += wr_cnt; ++(qhp->wq.sq_wptr); } spin_unlock_irqrestore(&qhp->lock, flag); @@ -359,7 +406,7 @@ int iwch_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, wr->wr_id; build_fw_riwrh((void *) wqe, T3_WR_RCV, T3_COMPLETION_FLAG, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), - 0, sizeof(struct t3_receive_wr) >> 3); + 0, sizeof(struct t3_receive_wr) >> 3, 3); PDBG("%s cookie 0x%llx idx 0x%x rq_wptr 0x%x rw_rptr 0x%x " "wqe %p \n", __func__, (unsigned long long) wr->wr_id, idx, qhp->wq.rq_wptr, qhp->wq.rq_rptr, wqe); @@ -419,10 +466,10 @@ int iwch_bind_mw(struct ib_qp *qp, sgl.lkey = mw_bind->mr->lkey; sgl.length = mw_bind->length; wqe->bind.reserved = 0; - wqe->bind.type = T3_VA_BASED_TO; + wqe->bind.type = TPT_VATO; /* TBD: check perms */ - wqe->bind.perms = iwch_ib_to_mwbind_access(mw_bind->mw_access_flags); + wqe->bind.perms = iwch_ib_to_tpt_access(mw_bind->mw_access_flags); wqe->bind.mr_stag = cpu_to_be32(mw_bind->mr->lkey); wqe->bind.mw_stag = cpu_to_be32(mw->rkey); wqe->bind.mw_len = cpu_to_be32(mw_bind->length); @@ -444,7 +491,7 @@ int iwch_bind_mw(struct ib_qp *qp, wqe->flit[T3_SQ_COOKIE_FLIT] = mw_bind->wr_id; build_fw_riwrh((void *)wqe, T3_WR_BIND, t3_wr_flags, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), 0, - sizeof(struct t3_bind_mw_wr) >> 3); + sizeof(struct t3_bind_mw_wr) >> 3, 3); ++(qhp->wq.wptr); ++(qhp->wq.sq_wptr); spin_unlock_irqrestore(&qhp->lock, flag); From weiny2 at llnl.gov Thu Jun 12 15:17:15 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 12 Jun 2008 15:17:15 -0700 Subject: [ofa-general] Re: [Announce] libopensmskummeeplugin; OpenSM/PerfMgr to MySQL plugin In-Reply-To: <20080612205750.GA6256@sashak.voltaire.com> References: <20080515133456.7b1e5c04.weiny2@llnl.gov> <20080612205750.GA6256@sashak.voltaire.com> Message-ID: <20080612151715.5b32c786.weiny2@llnl.gov> I am using SVN internally. I think I am going to use: http://savannah.gnu.org/ to host a git tree, but for now I would accept patches if you have any. Ira On Thu, 12 Jun 2008 23:57:50 +0300 Sasha Khapyorsky wrote: > Hi Ira, > > On 13:34 Thu 15 May , Ira Weiny wrote: > > Announcing, libopensmskummeeplugin. > > > > https://computing.llnl.gov/linux/skummeeplugin.html > > Does it use SCM (git :))? What is a patch acception procedure? > > Sasha From roland.list at gmail.com Thu Jun 12 17:47:07 2008 From: roland.list at gmail.com (Roland Dreier) Date: Thu, 12 Jun 2008 17:47:07 -0700 Subject: [ofa-general] mlx4_core CQ overrun In-Reply-To: References: Message-ID: > Anybody saw mlx4_core CQ overrun before? The test is based on OFED-1.3. FW > version is 2.3.0. Please let me know any more info is needed. Yes, I've seen CQ overrun -- when a CQ is overrun... > c955mgrs1:~ # dsh -av "grep 'CQ overrun' /var/log/messages" | sort > dsh: c955c2s1.ppd.pok.ibm.com Host is not responding. No command will be > issued to this host > c955c1s11.ppd.pok.ibm.com: Jun 10 07:18:15 c955c1s11 kernel: mlx4_core > 0003:01:00.0: CQ overrun on CQN 000098 What test are you running to get this error? My first guess would be a bug in the test that overruns a CQ. - R. From roland.list at gmail.com Thu Jun 12 17:53:28 2008 From: roland.list at gmail.com (Roland Dreier) Date: Thu, 12 Jun 2008 17:53:28 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: <20080612211302.25120.41155.stgit@dell3.ogc.int> References: <20080612211302.25120.41155.stgit@dell3.ogc.int> Message-ID: > - Add a new rdma ctl command called RDMA_GET_MIB to the low level driver > to obtain the protocol mib from the hardware. > > - Add a sysfs entry to allow dumping these stats. This looks really useful for debugging/tuning etc... what MIBs are being dumped here? The standard TCP and IPv4 MIBs? Looking at this it seems that maybe we want to add this to the core so all iWARP devices can (optionally) expose this in a common way -- I would assume that sane iWARP hardware gives a way to get offload engine stats. So maybe a new ib_device or iw_cm method -- i don't know the right name but something like struct ib_device.get_iw_stats struct iw_cm_verbs.get_stats? And then add something similar to the perf counter stuff we have in sysfs.c for IB devices? Then userspace tools can count on stuff being exposed in the same way rather than being vendor dependent. - R. From xma at us.ibm.com Thu Jun 12 18:10:08 2008 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 12 Jun 2008 18:10:08 -0700 Subject: [ofa-general] mlx4_core CQ overrun In-Reply-To: Message-ID: "Roland Dreier" wrote on 06/12/2008 05:47:07 PM: > > Anybody saw mlx4_core CQ overrun before? The test is based on OFED-1.3. FW > > version is 2.3.0. Please let me know any more info is needed. > > Yes, I've seen CQ overrun -- when a CQ is overrun... Thanks for your prompt response. So it is not possible a driver or FW bug? We will recheck our test. > > c955mgrs1:~ # dsh -av "grep 'CQ overrun' /var/log/messages" | sort > > dsh: c955c2s1.ppd.pok.ibm.com Host is not responding. No command will be > > issued to this host > > c955c1s11.ppd.pok.ibm.com: Jun 10 07:18:15 c955c1s11 kernel: mlx4_core > > 0003:01:00.0: CQ overrun on CQN 000098 > > What test are you running to get this error? My first guess would be > a bug in the > test that overruns a CQ. > > - R. Some vendor specific MPI stress test. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Thu Jun 12 18:20:27 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 12 Jun 2008 20:20:27 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: References: <20080612211302.25120.41155.stgit@dell3.ogc.int> Message-ID: <4851CB5B.1060804@opengridcomputing.com> Roland Dreier wrote: >> - Add a new rdma ctl command called RDMA_GET_MIB to the low level driver >> to obtain the protocol mib from the hardware. >> >> - Add a sysfs entry to allow dumping these stats. > > This looks really useful for debugging/tuning etc... what MIBs are being dumped > here? The standard TCP and IPv4 MIBs? > Yes, this is a standard TCP MIB. > Looking at this it seems that maybe we want to add this to the core so > all iWARP devices can (optionally) expose this in a common way -- I > would assume that sane > iWARP hardware gives a way to get offload engine stats. > > So maybe a new ib_device or iw_cm method -- i don't know the right name but > something like struct ib_device.get_iw_stats struct iw_cm_verbs.get_stats? > And then add something similar to the perf counter stuff we have in sysfs.c for > IB devices? > > Then userspace tools can count on stuff being exposed in the same way rather > than being vendor dependent. > Ok, lemme chew on that and come up with something. Steve. From tom at opengridcomputing.com Thu Jun 12 18:44:43 2008 From: tom at opengridcomputing.com (Tom Tucker) Date: Thu, 12 Jun 2008 20:44:43 -0500 Subject: [ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support In-Reply-To: <483AB626.8060404@opengridcomputing.com> References: <20080516223256.27221.34568.stgit@dell3.ogc.int> <20080516223419.27221.49014.stgit@dell3.ogc.int> <483AB626.8060404@opengridcomputing.com> Message-ID: <1213321483.30988.45.camel@trinity.ogc.int> On Mon, 2008-05-26 at 08:07 -0500, Steve Wise wrote: > > > Roland Dreier wrote: > > > - device-specific alloc/free of physical buffer lists for use in fast > > > register work requests. This allows devices to allocate this memory as > > > needed (like via dma_alloc_coherent). > > > > I'm looking at how one would implement the MM extensions for mlx4, and > > it turns out that in addition to needing to allocate these fastreg page > > lists in coherent memory, mlx4 is even going to need to write to the > > memory (basically set the lsb of each address for internal device > > reasons). So I think we just need to update the documentation of the > > interface so that not only does the page list belong to the device > > driver between posting the fastreg work request and completing the > > request, but also the device driver is allowed to change the page list > > as part of the work request processing. > > > > I don't see any real reason why this would cause problems for consumers; > > does this seem OK to other people? > > Tom, > > Does this affect how you plan to implement NFSRDMA MEM_MGT_EXTENTIONS > support? I think this is ok. From vlad at lists.openfabrics.org Fri Jun 13 02:15:44 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 13 Jun 2008 02:15:44 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080613-0200 daily build status Message-ID: <20080613091544.93697E619EC@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.21.1 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 1 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 2 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: error: too many arguments to function 'ip_route_output_key' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080613-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From swise at opengridcomputing.com Fri Jun 13 07:24:45 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 13 Jun 2008 09:24:45 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: References: <20080612211302.25120.41155.stgit@dell3.ogc.int> Message-ID: <4852832D.4000404@opengridcomputing.com> Roland Dreier wrote: >> - Add a new rdma ctl command called RDMA_GET_MIB to the low level driver >> to obtain the protocol mib from the hardware. >> >> - Add a sysfs entry to allow dumping these stats. >> > > This looks really useful for debugging/tuning etc... what MIBs are being dumped > here? The standard TCP and IPv4 MIBs? > > Looking at this it seems that maybe we want to add this to the core so > all iWARP devices can (optionally) expose this in a common way -- I > would assume that sane > iWARP hardware gives a way to get offload engine stats. > > So maybe a new ib_device or iw_cm method -- i don't know the right name but > something like struct ib_device.get_iw_stats struct iw_cm_verbs.get_stats? > And then add something similar to the perf counter stuff we have in sysfs.c for > IB devices? > I don't see any counter stuff in /sys on my mthca system. Where should it be and do I need to load a specific module to get these stats? Steve. From ossrosch at linux.vnet.ibm.com Fri Jun 13 07:39:40 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Fri, 13 Jun 2008 16:39:40 +0200 Subject: [ofa-general] Re: [PATCH REPOST] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: References: <200806101644.34693.ossrosch@linux.vnet.ibm.com> <200806101701.47090.ossrosch@linux.vnet.ibm.com> Message-ID: <200806131639.43173.ossrosch@linux.vnet.ibm.com> Hi Roland, On Tuesday 10 June 2008 18:18:50 Roland Dreier wrote: > > > So just to be clear: this is a workaround for a hardware/firmware bug? > > > Yes it is. > > OK, so paulus et al... does it seem like a good approach to call H_EOI > from driver code (given that this driver makes tons of other hcalls)? > > How critical is this? Since you said "corner case testing" I suspect we > can defer this to 2.6.27 and maybe get it into -stable later? No, it's ok with me if you pick this for 2.6.27. > > Also, out of curiousity: > > > +u64 hipz_h_eoi(int irq) > > +{ > > + int value; > > + unsigned long xirr; > > + > > + iosync(); > > what is the iosync() required for here? It's the same sequence as the interrupt handler for powerpc is implemented. > > > + value = (0xff << 24) | irq; > > + xirr = value & 0xffffffff; > > given that irq and value are ints, is there any possible way value could > have bits outside of the low 32 set? If you're worried about sign > extension isn't it simpler to just make value unsigned? > > > + return plpar_hcall_norets(H_EOI, xirr); > > +} > > ie why not: > > u64 hipz_h_eoi(int irq) > { > unsigned xirr; > > iosync(); > xirr = (0xff << 24) | irq; > return plpar_hcall_norets(H_EOI, xirr); > } > Yeah, you are rigth I will change that with the final patch. I will send the final patch soon. regards Stefan From hrosenstock at xsigo.com Fri Jun 13 07:53:10 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 13 Jun 2008 07:53:10 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_sa_mcmember_record.c: Better message on failure due to MC group not found Message-ID: <1213368790.14047.551.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Better message on failure due to MC group not found Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index de174c5..8b8cfa1 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1278,8 +1278,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, } } else { CL_PLOCK_RELEASE(sa->p_lock); - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Failed since multicast group not present\n"); + OSM_LOG(sa->p_log, OSM_LOG_ERROR, + "Failed since multicast group 0x%16" + PRIx64 " : 0x%016" PRIx64 " not present\n", + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix), + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id)); osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); goto Exit; } From ossrosch at linux.vnet.ibm.com Fri Jun 13 07:55:41 2008 From: ossrosch at linux.vnet.ibm.com (Stefan Roscher) Date: Fri, 13 Jun 2008 16:55:41 +0200 Subject: [ofa-general] [PATCH REPOST #2] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts Message-ID: <200806131655.43410.ossrosch@linux.vnet.ibm.com> During corner case testing, we noticed that some versions of ehca do not properly transition to interrupt done in special load situations. This can be resolved by periodically triggering EOI through H_EOI, if eqes are pending. Signed-off-by: Stefan Roscher --- As firmware team suggested I moved the call of the EOI h_call into the handler function, this ensures that we will call EOI only when we find a valid eqe on the event queue. Additionally I changed the calculation of the xirr value as Roland suggested. drivers/infiniband/hw/ehca/ehca_irq.c | 9 +++++++-- drivers/infiniband/hw/ehca/hcp_if.c | 10 ++++++++++ drivers/infiniband/hw/ehca/hcp_if.h | 1 + 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_irq.c b/drivers/infiniband/hw/ehca/ehca_irq.c index ce1ab05..0792d93 100644 --- a/drivers/infiniband/hw/ehca/ehca_irq.c +++ b/drivers/infiniband/hw/ehca/ehca_irq.c @@ -531,7 +531,7 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq) { struct ehca_eq *eq = &shca->eq; struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache; - u64 eqe_value; + u64 eqe_value, ret; unsigned long flags; int eqe_cnt, i; int eq_empty = 0; @@ -583,8 +583,13 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq) ehca_dbg(&shca->ib_device, "No eqe found for irq event"); goto unlock_irq_spinlock; - } else if (!is_irq) + } else if (!is_irq) { + ret = hipz_h_eoi(eq->ist); + if (ret != H_SUCCESS) + ehca_err(&shca->ib_device, + "bad return code EOI -rc = %ld\n", ret); ehca_dbg(&shca->ib_device, "deadman found %x eqe", eqe_cnt); + } if (unlikely(eqe_cnt == EHCA_EQE_CACHE_SIZE)) ehca_dbg(&shca->ib_device, "too many eqes for one irq event"); /* enable irq for new packets */ diff --git a/drivers/infiniband/hw/ehca/hcp_if.c b/drivers/infiniband/hw/ehca/hcp_if.c index 5245e13..415d3a4 100644 --- a/drivers/infiniband/hw/ehca/hcp_if.c +++ b/drivers/infiniband/hw/ehca/hcp_if.c @@ -933,3 +933,13 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, r_cb, 0, 0, 0, 0); } + +u64 hipz_h_eoi(int irq) +{ + unsigned long xirr; + + iosync(); + xirr = (0xffULL << 24) | irq; + + return plpar_hcall_norets(H_EOI, xirr); +} diff --git a/drivers/infiniband/hw/ehca/hcp_if.h b/drivers/infiniband/hw/ehca/hcp_if.h index 60ce02b..2c3c6e0 100644 --- a/drivers/infiniband/hw/ehca/hcp_if.h +++ b/drivers/infiniband/hw/ehca/hcp_if.h @@ -260,5 +260,6 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, const u64 ressource_handle, void *rblock, unsigned long *byte_count); +u64 hipz_h_eoi(int irq); #endif /* __HCP_IF_H__ */ -- 1.5.5 From hrosenstock at xsigo.com Fri Jun 13 08:00:20 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 13 Jun 2008 08:00:20 -0700 Subject: [ofa-general] [PATCHv2][TRIVIAL] opensm/osm_sa_mcmember_record.c: Better message on failure due to MC group not found Message-ID: <1213369220.14047.553.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Better message on failure due to MC group not found Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index de174c5..0690d52 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1278,8 +1278,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, } } else { CL_PLOCK_RELEASE(sa->p_lock); - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Failed since multicast group not present\n"); + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " + "Failed since multicast group 0x%16" + PRIx64 " : 0x%016" PRIx64 " not present\n", + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix), + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id)); osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); goto Exit; } From kaber at trash.net Fri Jun 13 08:20:07 2008 From: kaber at trash.net (Patrick McHardy) Date: Fri, 13 Jun 2008 17:20:07 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> <485148DA.2060508@trash.net> Message-ID: <48529027.1090309@trash.net> Amar Mudrankit wrote: > I have just started on getting into the rtnl_link API. There are few > queries on it, plz correct me if I am wrong. > > 1. If I understand it correctly, this interface is not text based. > Hence, it looks like this interface needs a separate user program > which actually sends the parameters to the driver. A separate user > program for each driver, as their configuration parameters might be > different. Is it right? Not necessarily an entire program, you can integrate it in iproute2 or libnl. Look at ip/iplink_vlan.c in iproute for an example. > 2. I am not sure if there exists a standardized (netlink?) interface > through which parameters can be passed to kernel. For virtual network drivers, its rtnl_link, or more generally, rtnetlink. > If so, there would > be standard set of parameters that can be configured and standard set > of statistics values that can be retrieved from driver. Is it so? Yes. > Actually, I tried googling around rtnl_link, but couldn't collect > sufficient information. > > 3. Is there any kind of documentation available on this? I tried > searching on it in Documentation/ directory as well as git log but > could not locate enough of it. My fault, I have some unfinished documentation that I never completed. There are a few examples in the tree: drivers/net/dummy.c drivers/net/macvlan.c drivers/net/ifb.c drivers/net/veth.c net/8021q/vlan_netlink.c macvlan and vlan are probably best suited as an example, dummy and ifb are too simple and veth is kind of special because it registers two devices. If you have questions, feel free to ask. > 4. Can anybody point to an ethernet driver as an example of > implementation of rtnl_link API along with an equivalent user space > program to configure it? > > That would be of immense help. See above. >>> +echo -n 00066a01de000037 > >>> /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/ioc_guid >>> +echo -n fe8000000000000100066a11de000037 > >>> /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/dgid >>> +echo -n ffff > >>> /sys/class/infiniband_qlgc_vnic/interfaces/veth1/primary_path/pkey >> I guess life would be easier for users if those parameters can >> be generated in userspace or at least looked up in something >> like the maps in /etc/iproute2/. >> > > Is it the answer of question no 2 above? If so, what about statistics > of the driver then? You can dump driver private statistics in the ->fill_xstats function. >> In any case, things like tx_csum and rx_csum definitely do >> not belong in this interface. >> > > We are planning to give ethtool support for the driver in future, in > which case these 2 parameters will be configured through ethtool. Yes, thats where it belongs. From hrosenstock at xsigo.com Fri Jun 13 08:30:41 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 13 Jun 2008 08:30:41 -0700 Subject: [ofa-general] [PATCHv3][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements Message-ID: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Some error message improvements Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index de174c5..df4eca3 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1121,7 +1121,10 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) if (p_ctxt->p_mgrp) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B30: " - "Multiple MC groups for same MGID\n"); + "Multiple MC groups for MGID " + "0x%016" PRIx64 " : 0x%016" PRIx64 "\n", + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix), + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id)); return; } p_ctxt->p_mgrp = p_mgrp; @@ -1278,8 +1281,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, } } else { CL_PLOCK_RELEASE(sa->p_lock); - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Failed since multicast group not present\n"); + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " + "Failed since multicast group 0x%16" + PRIx64 " : 0x%016" PRIx64 " not present\n", + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix), + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id)); osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); goto Exit; } From hrosenstock at xsigo.com Fri Jun 13 10:28:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 13 Jun 2008 10:28:29 -0700 Subject: [ofa-general] [PATCH] opensm/osm_sa_mcmember_record.c: When consolidating SNM, need separate group per PKey Message-ID: <1213378109.14047.563.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: When consolidating SNM, need separate group per PKey If don't do this, leaves on one group can cause the entire group to be prematurely removed Also, fix consolidation issue introduced at commit a20dad09d2dbaecff6c05d4566fe8fac358b614e Also, cosmetic change to log message Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index de174c5..1a9bc54 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1093,7 +1093,8 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) /* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ /* Where Z is the scope, XXXX is the P_Key, and * YYYYYY is the last 24 bits of the port guid */ -#define PREFIX_MASK (0xff10ffff0000ffffULL) +#define PREFIX_NOPKEY_MASK (0xff10ffff0000ffffULL) +#define PREFIX_MASK (0xff10ffffffffffffULL) #define PREFIX_SIGNATURE (0xff10601b00000000ULL) #define INT_ID_MASK (0xfffffff1ff000000ULL) #define INT_ID_SIGNATURE (0x00000001ff000000ULL) @@ -1102,15 +1103,15 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) uint64_t rcv_prefix = cl_ntoh64(p_recvd_mgid->unicast.prefix); uint64_t rcv_interface_id = cl_ntoh64(p_recvd_mgid->unicast.interface_id); - if ((rcv_prefix & PREFIX_MASK) == PREFIX_SIGNATURE && + if ((rcv_prefix & PREFIX_NOPKEY_MASK) == PREFIX_SIGNATURE && (rcv_interface_id & INT_ID_MASK) == INT_ID_SIGNATURE && (g_prefix & PREFIX_MASK) == - (rcv_prefix && PREFIX_MASK) && + (rcv_prefix & PREFIX_MASK) && (g_interface_id & INT_ID_MASK) == (rcv_interface_id & INT_ID_MASK)) { OSM_LOG(sa->p_log, OSM_LOG_INFO, "Special Case Solicited Node Mcast " - " Join for MGID 0x%016" PRIx64 + "Join for MGID 0x%016" PRIx64 " : 0x%016" PRIx64 "\n", rcv_prefix, rcv_interface_id); } else From hrosenstock at xsigo.com Fri Jun 13 10:35:55 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 13 Jun 2008 10:35:55 -0700 Subject: [ofa-general] [PATCH] opensm/man/opensm.8.in: Update consolidate_ipv6_snm_req description Message-ID: <1213378555.14047.568.camel@hrosenstock-ws.xsigo.com> opensm/man/opensm.8.in: Update consolidate_ipv6_snm_req description Signed-off-by: Hal Rosenstock diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index b6a9fc7..d34b9c8 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -1,4 +1,4 @@ -.TH OPENSM 8 "Feb 27, 2008" "OpenIB" "OpenIB Management" +.TH OPENSM 8 "June 13, 2008" "OpenIB" "OpenIB Management" .SH NAME opensm \- InfiniBand subnet manager and administration (SM/SA) @@ -253,9 +253,9 @@ Specify the sweep time for the performance manager in seconds (default is 180 seconds). Only takes effect if --enable-perfmgr was specified at configure time. .TP -.BI --consolidate_ipv6_snm_reqests -Consolidate IPv6 Solicited Node Multicast group joins into 1 IB multicast -group. +.BI --consolidate_ipv6_snm_req +Consolidate IPv6 Solicited Node Multicast group join requests into one +multicast group per MGID PKey. .TP \fB\-v\fR, \fB\-\-verbose\fR This option increases the log verbosity level. From ralph.campbell at qlogic.com Fri Jun 13 11:28:57 2008 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Fri, 13 Jun 2008 11:28:57 -0700 Subject: [ofa-general] Missing tarballs from downloads area of ofa website In-Reply-To: <48517429.7000509@redhat.com> References: <48517429.7000509@redhat.com> Message-ID: <1213381737.5452.115.camel@chromite.mv.qlogic.com> On Thu, 2008-06-12 at 15:08 -0400, Doug Ledford wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I've been trying to submit various packages to the Fedora review process > so the packages can be included in the next Fedora release. > Unfortunately, the reviewers are very much sticklers over the issue of > the existence of a valid upstream tarball release that can be downloaded > and verified against the package. This has prevented me from submitting > several packages for review. Amongst them: > > libipathverbs (there is a release up on the site, but I happen to know > that the OFED-1.3 distro include a tarball libipathverbs tarball with > the exact same version number but a totally different set of files, so > until there is a definitive release tarball, I can't submit this one) I updated the libipathverbs-1.1.tar.gz that is in http://www.openfabrics.org/downloads/ipath/ to be the one from the official OFED-1.3.1.tgz release. This is also the same as in OFED-1.3.tgz. Although many files were "totally different", these were all due to a different version of automake to generate the configure script. The actual source files for the library were the same. From ctung at NetEffect.com Fri Jun 13 12:38:07 2008 From: ctung at NetEffect.com (Chien Tung) Date: Fri, 13 Jun 2008 14:38:07 -0500 Subject: [ofa-general] Missing tarballs from downloads area of ofa website In-Reply-To: <48517429.7000509@redhat.com> References: <48517429.7000509@redhat.com> Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC081D95B4@venom2> Doug Ledford wrote: > I've been trying to submit various packages to the Fedora > review process so the packages can be included in the next > Fedora release. > Unfortunately, the reviewers are very much sticklers over the > issue of the existence of a valid upstream tarball release > that can be downloaded and verified against the package. > This has prevented me from submitting several packages for > review. Amongst them: > > libipathverbs (there is a release up on the site, but I > happen to know that the OFED-1.3 distro include a tarball > libipathverbs tarball with the exact same version number but > a totally different set of files, so until there is a > definitive release tarball, I can't submit this one) > > libehca (no tarball exists...of course, this isn't too big a > deal for Fedora since it doesn't really support big iron ppc > hardware anyway) > > libnes (no tarball) I've placed libnes tarball in the OFED download directory: http://www.openfabrics.org/downloads/nes/libnes-0.5.tar.gz. I built it from OFED 1.3.1 codebase. Chien From amar.mudrankit at qlogic.com Fri Jun 13 14:47:08 2008 From: amar.mudrankit at qlogic.com (Amar Mudrankit) Date: Sat, 14 Jun 2008 03:17:08 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <48529027.1090309@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> <485148DA.2060508@trash.net> <48529027.1090309@trash.net> Message-ID: > > If you have questions, feel free to ask. > I was trying to guess the use case for the vlan which has implemented rtnl_link api : For sending parameters to driver: # ip link set up multicast ...... type vlan id reorder_hdr on ingress-qos-map <...> egress-qos-map <...> In such case, I think the link_util's private parse_opt function will be invoked to parse link's private configurable parameters. These parameters along with the standard set of parameters (name, multicast, promisc etc) are then sent to driver through rtnl_talk function . Similarly, for reading values from driver: # ip addr show dev does it require "type" field similar to "ip link set"? Is this understanding correct? How does a link type corresponding to VLAN driver gets added into linkutil_list, so that ip will correctly call functions specific to VLAN driver? > > You can dump driver private statistics in the ->fill_xstats function. > I was looking at latest release of iproute2, in which I could not locate fill_xstats. Can you help me to locate this? Thanks and Regards, Amar From chu11 at llnl.gov Fri Jun 13 15:48:32 2008 From: chu11 at llnl.gov (Al Chu) Date: Fri, 13 Jun 2008 15:48:32 -0700 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing Message-ID: <1213397313.5369.356.camel@cardanus.llnl.gov> Hey Sasha, This is a conceptually simple option I've developed for updn routing. Currently in updn routing, nodes/guids are routed on switches in a seemingly-random order, which I believe is due to internal data structure organization (i.e. cl_qmap_apply_func is called on port_guid_tbl) as well as how the fabric is scanned (it is logically scanned from a port perspective, but it may not be logical from a node perspective). I had a hypothesis that this was leading to increased contention in the network for MPI. For example, suppose we have 12 uplinks from a leaf switch to a spine switch. If we want to send data from this leaf switch to node[13-24], the up links we will send on are pretty random. It's because: A) node[13-24] are individually routed at seemingly-random points based on when they are called by cl_qmap_apply_func(). B) the ports chosen for routing are based on least used port usage. C) least used port usage is based on whatever was routed earlier on. So I developed this patch series, which supports an option called "guid_routing_order_file" which allows the user to input a file with a list of port_guids which will indicate the order in which guids are routed instead (naturally, those guids not listed are routed last). I list the port guids of the nodes of the cluster from node0 to nodeN, one per line in the file. By listing the nodes in this order, I believe we could get less contention in the network. In the example above, sending to node[13-24] should use all of the 12 uplinks, b/c the ports will be equally used b/c nodes[1-12] were routed beforehand in order. The results from some tests are pretty impressive when I do this. LMC=0 average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s when I use guid_routing_order. A variety of other positive performance increases were found when doing other tests, other MPIs, and other LMCs if anyone is interested. BTW, I developed this patch series before your preserve-base-lid patch series. It will 100% conflict with the preserve-base-lid patch series. I will fix this patch series once the preserve-base-lids patch series is committed to git. I'm just looking for comments right now. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From chu11 at llnl.gov Fri Jun 13 15:48:47 2008 From: chu11 at llnl.gov (Al Chu) Date: Fri, 13 Jun 2008 15:48:47 -0700 Subject: [ofa-general] [OPENSM PATCH 1/5]: add guid_routing_order_file option Message-ID: <1213397327.5369.357.camel@cardanus.llnl.gov> Nothing too fancy in this patch. Parse command line options, add manpage entries, etc. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-add-guid_routing_order_file-option.patch Type: text/x-patch Size: 5794 bytes Desc: not available URL: From chu11 at llnl.gov Fri Jun 13 15:49:02 2008 From: chu11 at llnl.gov (Al Chu) Date: Fri, 13 Jun 2008 15:49:02 -0700 Subject: [ofa-general] [OPENSM PATCH 2/5]: make osm_ucast_mgr_process_port public Message-ID: <1213397342.5369.358.camel@cardanus.llnl.gov> To avoid code copying, need to make this function accessible by other files. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-make-osm_ucast_mgr_process_port-public.patch Type: text/x-patch Size: 3048 bytes Desc: not available URL: From chu11 at llnl.gov Fri Jun 13 15:49:10 2008 From: chu11 at llnl.gov (Al Chu) Date: Fri, 13 Jun 2008 15:49:10 -0700 Subject: [ofa-general] [OPENSM PATCH 3/5]: make osm_ucast_mgr_process_tbl public. Message-ID: <1213397350.5369.359.camel@cardanus.llnl.gov> To avoid code copying, need to make this function accessible by other files. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-make-osm_ucast_mgr_process_tbl-public.patch Type: text/x-patch Size: 3659 bytes Desc: not available URL: From chu11 at llnl.gov Fri Jun 13 15:49:53 2008 From: chu11 at llnl.gov (Al Chu) Date: Fri, 13 Jun 2008 15:49:53 -0700 Subject: [ofa-general] [OPENSM PATCH 4/5]: add updn ucast_build_fwd_tables call Message-ID: <1213397394.5369.361.camel@cardanus.llnl.gov> Add a routing-engine ucast_build_fwd_tables call to call internal to osm_ucast_updn.c functions rather than have the default __osm_ucast_build_fwd_tables() be called in osm_ucast_mgr_process(). Necessary for this option to be exclusive for updn routing. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0004-add-updn-ucast_build_fwd_tables-call.patch Type: text/x-patch Size: 2139 bytes Desc: not available URL: From chu11 at llnl.gov Fri Jun 13 15:50:15 2008 From: chu11 at llnl.gov (Al Chu) Date: Fri, 13 Jun 2008 15:50:15 -0700 Subject: [ofa-general] [OPENSM PATCH 5/5]: support guid_routing_order_file option Message-ID: <1213397415.5369.363.camel@cardanus.llnl.gov> This implements the real stuff in this patch series. Loads and parses the guid_routing_order_file. Finds ports that aren't specified in the file and puts them at the end of the list for sorting. I do add a flag variable into osm_port_t for convenience. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0005-support-guid_routing_order_file.patch Type: text/x-patch Size: 8628 bytes Desc: not available URL: From supervive at ixtaccihuatl.com Sat Jun 14 00:09:23 2008 From: supervive at ixtaccihuatl.com (Jackye Wallace) Date: Sat, 14 Jun 2008 16:09:23 +0900 Subject: [ofa-general] `Acro bat 8 Pro` Message-ID: <000701c8cdec$717c3300$0100007f@fajjr> ` Adobe CS3 Master Collection for PC or MAC includes: ` InDesign CS3 ` Photoshop CS3 ` Illustrator CS3 ` Acrobat 8 Professional ` Flash CS3 Professional ` Dreamweaver CS3 ` Fireworks CS3 ` Contribute CS3 ` After Effects CS3 Professional ` Premiere Pro CS3 ` Encore DVD CS3 ` Soundbooth CS3 ` getsoftxp .com in your Internet Exp1orer ` System Requirements ` For PC: ` Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core ` Duo (or compatible) processor; SSE2-enabled processor required for AMD systems ` Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) ` 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ` 38GB of available hard-disk space (additional free space required during installation) ` Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ` Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) ` 1,280x1,024 monitor resolution with 32-bit color adapter ` DVD-ROM drive ` For MAC: ` PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) ` Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server ` 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ` 36GB of available hard-disk space (additional free space required during installation) ` Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ` Core Audio compatible sound card ` 1,280x1,024 monitor resolution with 32-bit color adapter ` DVD-ROM drive` DVD+-R burner required for DVD creation China Tightens Control on Corruption, Quake Stories Where a Hospital Has No Doctors From kaber at trash.net Sat Jun 14 01:08:08 2008 From: kaber at trash.net (Patrick McHardy) Date: Sat, 14 Jun 2008 10:08:08 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> <485148DA.2060508@trash.net> <48529027.1090309@trash.net> Message-ID: <48537C68.1010207@trash.net> Amar Mudrankit wrote: >> If you have questions, feel free to ask. >> > I was trying to guess the use case for the vlan which has implemented > rtnl_link api : > > For sending parameters to driver: > # ip link set up multicast ...... type vlan id > reorder_hdr on > > ingress-qos-map <...> egress-qos-map <...> > > In such case, I think the link_util's private parse_opt function will > be invoked to parse link's private configurable parameters. These > parameters along with the standard set of parameters (name, multicast, > promisc etc) are then sent to driver through rtnl_talk function . > > Similarly, for reading values from driver: > > # ip addr show dev > > does it require "type" field similar to "ip link set"? No. Thats only required for parsing device specific parameters. > Is this understanding correct? Yes. > How does a link type corresponding to VLAN driver gets added into > linkutil_list, so that ip will correctly call functions specific to > VLAN driver? iproute dlopens itself and looks for the struct "_link_util". >> You can dump driver private statistics in the ->fill_xstats function. >> > > I was looking at latest release of iproute2, in which I could not > locate fill_xstats. Can you help me to locate this? ->fill_xstats is in the kernel, in iproute its ->print_xstats. From vlad at lists.openfabrics.org Sat Jun 14 02:16:04 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 14 Jun 2008 02:16:04 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080614-0200 daily build status Message-ID: <20080614091604.33B5DE2841C@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 1 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 2 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: error: too many arguments to function 'ip_route_output_key' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080614-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From yaltid at bramonline.com Sat Jun 14 02:16:09 2008 From: yaltid at bramonline.com (Jed Maurer) Date: Sat, 14 Jun 2008 13:16:09 +0400 Subject: [ofa-general] Re: Message-ID: <01c8ce20$cf2a7280$fe2983d5@yaltid> Get your corn really big! Be the center of the world! Be the MAN! http://www.rioew.cn/a/ From sashak at voltaire.com Sat Jun 14 05:50:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 14 Jun 2008 15:50:50 +0300 Subject: [ofa-general] [PATCH] opensm: consolidate config parser logging code Message-ID: <20080614125050.GA24365@sashak.voltaire.com> Consolidate config parser logging code Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_subnet.c | 71 +++++++++++++++---------------------------- 1 files changed, 25 insertions(+), 46 deletions(-) diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 94e98c3..b1486ca 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -46,6 +46,7 @@ #include #include +#include #include #include #include @@ -468,21 +469,28 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * const p_opt) /********************************************************************** **********************************************************************/ +static void log_config_value(char *name, const char *fmt, ...) +{ + char buf[128]; + va_list args; + unsigned n; + va_start(args, fmt); + n = snprintf(buf, sizeof(buf), " Loading Cached Option:%s = ", name); + n += vsnprintf(buf + n, sizeof(buf) - n, fmt, args); + snprintf(buf + n, sizeof(buf) - n, "\n"); + va_end(args); + printf(buf); + cl_log_event("OpenSM", CL_LOG_INFO, buf, NULL, 0); +} + static void opts_unpack_net64(IN char *p_req_key, IN char *p_key, IN char *p_val_str, IN uint64_t * p_val) { - uint64_t val; - if (!strcmp(p_req_key, p_key)) { - val = strtoull(p_val_str, NULL, 0); + uint64_t val = strtoull(p_val_str, NULL, 0); if (cl_hton64(val) != *p_val) { - char buff[128]; - sprintf(buff, - " Loading Cached Option:%s = 0x%016" PRIx64 - "\n", p_key, val); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); + log_config_value(p_key, "0x%016" PRIx64, val); *p_val = cl_ntoh64(val); } } @@ -494,16 +502,10 @@ static void opts_unpack_uint32(IN char *p_req_key, IN char *p_key, IN char *p_val_str, IN uint32_t * p_val) { - uint32_t val; - if (!strcmp(p_req_key, p_key)) { - val = strtoul(p_val_str, NULL, 0); + uint32_t val = strtoul(p_val_str, NULL, 0); if (val != *p_val) { - char buff[128]; - sprintf(buff, " Loading Cached Option:%s = %u\n", - p_key, val); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); + log_config_value(p_key, "%u", val); *p_val = val; } } @@ -515,16 +517,10 @@ static void opts_unpack_uint16(IN char *p_req_key, IN char *p_key, IN char *p_val_str, IN uint16_t * p_val) { - uint16_t val; - if (!strcmp(p_req_key, p_key)) { - val = (uint16_t) strtoul(p_val_str, NULL, 0); + uint16_t val = (uint16_t) strtoul(p_val_str, NULL, 0); if (val != *p_val) { - char buff[128]; - sprintf(buff, " Loading Cached Option:%s = %u\n", - p_key, val); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); + log_config_value(p_key, "%u", val); *p_val = val; } } @@ -541,11 +537,7 @@ opts_unpack_net16(IN char *p_req_key, val = strtoul(p_val_str, NULL, 0); CL_ASSERT(val < 0x10000); if (cl_hton32(val) != *p_val) { - char buff[128]; - sprintf(buff, " Loading Cached Option:%s = 0x%04x\n", - p_key, val); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); + log_config_value(p_key, "0x%04x", val); *p_val = cl_hton16((uint16_t) val); } } @@ -562,11 +554,7 @@ opts_unpack_uint8(IN char *p_req_key, val = strtoul(p_val_str, NULL, 0); CL_ASSERT(val < 0x100); if (val != *p_val) { - char buff[128]; - sprintf(buff, " Loading Cached Option:%s = %u\n", - p_key, val); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); + log_config_value(p_key, "%u", val); *p_val = (uint8_t) val; } } @@ -586,11 +574,7 @@ opts_unpack_boolean(IN char *p_req_key, val = TRUE; if (val != *p_val) { - char buff[128]; - sprintf(buff, " Loading Cached Option:%s = %s\n", - p_key, p_val_str); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); + log_config_value(p_key, "%s", p_val_str); *p_val = val; } } @@ -604,12 +588,7 @@ opts_unpack_charp(IN char *p_req_key, { if (!strcmp(p_req_key, p_key) && p_val_str) { if ((*p_val == NULL) || strcmp(p_val_str, *p_val)) { - char buff[128]; - sprintf(buff, " Loading Cached Option:%s = %s\n", - p_key, p_val_str); - printf(buff); - cl_log_event("OpenSM", CL_LOG_INFO, buff, NULL, 0); - + log_config_value(p_key, "%s", p_val_str); /* special case the "(null)" string */ if (strcmp(null_str, p_val_str) == 0) { *p_val = NULL; -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sat Jun 14 05:51:48 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 14 Jun 2008 15:51:48 +0300 Subject: [ofa-general] [PATCH] opensm/perfmgr: move event_db_dump_file to common OpenSM dump dir Message-ID: <20080614125148.GB24365@sashak.voltaire.com> This places default PerfMgr port counters dump file into common OpenSM dump files directory. Also file name definitions is only in subnet's options, so if config is reloaded PerfMgr will take a new value. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_perfmgr.h | 7 +------ opensm/opensm/osm_perfmgr.c | 16 ++++++++++++---- opensm/opensm/osm_subnet.c | 5 +++-- 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/opensm/include/opensm/osm_perfmgr.h b/opensm/include/opensm/osm_perfmgr.h index 9152f5f..50a2c32 100644 --- a/opensm/include/opensm/osm_perfmgr.h +++ b/opensm/include/opensm/osm_perfmgr.h @@ -72,7 +72,7 @@ extern "C" { *********/ #define OSM_PERFMGR_DEFAULT_SWEEP_TIME_S 180 -#define OSM_PERFMGR_DEFAULT_DUMP_FILE OSM_DEFAULT_TMP_DIR "/opensm_port_counters.log" +#define OSM_PERFMGR_DEFAULT_DUMP_FILE "opensm_port_counters.log" #define OSM_PERFMGR_DEFAULT_MAX_OUTSTANDING_QUERIES 500 /****s* OpenSM: PerfMgr/osm_perfmgr_state_t */ @@ -124,8 +124,6 @@ typedef struct _osm_perfmgr { osm_perfmgr_state_t state; osm_perfmgr_sweep_state_t sweep_state; uint16_t sweep_time_s; - char *db_file; - char *event_db_dump_file; perfmgr_db_t *db; atomic32_t outstanding_queries; /* this along with sig_query */ cl_event_t sig_query; /* will throttle our querys */ @@ -145,9 +143,6 @@ typedef struct _osm_perfmgr { * mad_pool * Pointer to the MAD pool. * -* event_db_dump_file -* File to be used to dump the Port Counters -* * mad_ctrl * Mad Controller *********/ diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index 5c53c24..1633c41 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -856,7 +856,6 @@ void osm_perfmgr_shutdown(osm_perfmgr_t * const pm) void osm_perfmgr_destroy(osm_perfmgr_t * const pm) { OSM_LOG_ENTER(pm->log); - free(pm->event_db_dump_file); perfmgr_db_destroy(pm->db); cl_timer_destroy(&pm->sweep_timer); OSM_LOG_EXIT(pm->log); @@ -1245,7 +1244,6 @@ osm_perfmgr_init(osm_perfmgr_t * const pm, pm->state = p_opt->perfmgr ? PERFMGR_STATE_ENABLED : PERFMGR_STATE_DISABLE; pm->sweep_time_s = p_opt->perfmgr_sweep_time_s; - pm->event_db_dump_file = strdup(p_opt->event_db_dump_file); pm->max_outstanding_queries = p_opt->perfmgr_max_outstanding_queries; pm->event_plugin = event_plugin; @@ -1290,9 +1288,19 @@ void osm_perfmgr_clear_counters(osm_perfmgr_t * pm) *******************************************************************/ void osm_perfmgr_dump_counters(osm_perfmgr_t * pm, perfmgr_db_dump_t dump_type) { - if (perfmgr_db_dump(pm->db, pm->event_db_dump_file, dump_type) != 0) + char path[256]; + char *file_name; + if (pm->subn->opt.event_db_dump_file) + file_name = pm->subn->opt.event_db_dump_file; + else { + snprintf(path, sizeof(path), "%s/%s", + pm->subn->opt.dump_files_dir, + OSM_PERFMGR_DEFAULT_DUMP_FILE); + file_name = path; + } + if (perfmgr_db_dump(pm->db, file_name, dump_type) != 0) OSM_LOG(pm->log, OSM_LOG_ERROR, "Failed to dump file %s : %s", - pm->event_db_dump_file, strerror(errno)); + file_name, strerror(errno)); } /******************************************************************* diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index b1486ca..1bf3daf 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -427,7 +427,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * const p_opt) p_opt->perfmgr_sweep_time_s = OSM_PERFMGR_DEFAULT_SWEEP_TIME_S; p_opt->perfmgr_max_outstanding_queries = OSM_PERFMGR_DEFAULT_MAX_OUTSTANDING_QUERIES; - p_opt->event_db_dump_file = OSM_PERFMGR_DEFAULT_DUMP_FILE; + p_opt->event_db_dump_file = NULL; /* use default */ #endif /* ENABLE_OSM_PERF_MGR */ p_opt->event_plugin_name = OSM_DEFAULT_EVENT_PLUGIN_NAME; @@ -1600,7 +1600,8 @@ int osm_subn_write_conf_file(char *file_name, IN osm_subn_opt_t *const p_opts) fprintf(opts_file, "#\n# Event DB Options\n#\n" "# Dump file to dump the events to\n" - "event_db_dump_file %s\n\n", p_opts->event_db_dump_file); + "event_db_dump_file %s\n\n", p_opts->event_db_dump_file ? + p_opts->event_db_dump_file : null_str); #endif /* ENABLE_OSM_PERF_MGR */ fprintf(opts_file, -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sat Jun 14 05:54:12 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 14 Jun 2008 15:54:12 +0300 Subject: [ofa-general] [PATCH] opensm/config: allow space separated strings as values Message-ID: <20080614125412.GC24365@sashak.voltaire.com> This slightly extends OpenSM config file format and allow using space separated strings (quoted or not) as config values. Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_subnet.c | 31 ++++++++++++++++++++++++++++--- 1 files changed, 28 insertions(+), 3 deletions(-) diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 1bf3daf..a9081c9 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include #include @@ -606,6 +607,30 @@ opts_unpack_charp(IN char *p_req_key, /********************************************************************** **********************************************************************/ +static char *clean_val(char *val) +{ + char *p = val; + /* clean leading spaces */ + while (isspace(*p)) + p++; + val = p; + if (!*val) + return val; + /* clean trailing spaces */ + p = val + strlen(val) - 1; + while (p > val && isspace(*p)) + p--; + p[1] = '\0'; + /* clean quotas */ + if ((*val == '\"' && *p == '\"') || (*val == '\'' && *p == '\'')) { + val++; + p--; + } + return val; +} + +/********************************************************************** + **********************************************************************/ static void subn_parse_qos_options(IN const char *prefix, IN char *p_key, @@ -1106,7 +1131,7 @@ int osm_subn_parse_conf_file(char *file_name, osm_subn_opt_t * const p_opts) { char line[1024]; FILE *opts_file; - char *p_key, *p_val, *p_last; + char *p_key, *p_val; opts_file = fopen(file_name, "r"); if (!opts_file) { @@ -1124,11 +1149,11 @@ int osm_subn_parse_conf_file(char *file_name, osm_subn_opt_t * const p_opts) while (fgets(line, 1023, opts_file) != NULL) { /* get the first token */ - p_key = strtok_r(line, " \t\n", &p_last); + p_key = strtok_r(line, " \t\n", &p_val); if (!p_key) continue; - p_val = strtok_r(NULL, " \t\n", &p_last); + p_val = clean_val(p_val); opts_unpack_net64("guid", p_key, p_val, &p_opts->guid); -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sat Jun 14 05:55:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 14 Jun 2008 15:55:29 +0300 Subject: [ofa-general] [PATCH] opensm: move event_plugin from perfmgr to opensm In-Reply-To: <20080614125412.GC24365@sashak.voltaire.com> References: <20080614125412.GC24365@sashak.voltaire.com> Message-ID: <20080614125529.GD24365@sashak.voltaire.com> OpenSM event_plugin can be used for various purposes and not only by PerfMgr. So make it more OpenSM centric rather than PerfMgr specific. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_event_plugin.h | 2 - opensm/include/opensm/osm_opensm.h | 3 ++ opensm/include/opensm/osm_perfmgr.h | 39 +++----------------- opensm/include/opensm/osm_perfmgr_db.h | 11 +++--- opensm/opensm/osm_console.c | 4 +- opensm/opensm/osm_event_plugin.c | 8 ---- opensm/opensm/osm_opensm.c | 21 ++++++----- opensm/opensm/osm_perfmgr.c | 48 ++++++++++-------------- opensm/opensm/osm_perfmgr_db.c | 57 ++++++++++++++++-------------- 9 files changed, 78 insertions(+), 115 deletions(-) diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h index 5f939b6..a7cad32 100644 --- a/opensm/include/opensm/osm_event_plugin.h +++ b/opensm/include/opensm/osm_event_plugin.h @@ -173,8 +173,6 @@ typedef struct osm_epi_plugin { */ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name); void osm_epi_destroy(osm_epi_plugin_t * plugin); -void osm_epi_report(osm_epi_plugin_t * plugin, - osm_epi_event_id_t event_id, void *event_data); /** ========================================================================= * Helper functions diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index 92e4238..63d2247 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -495,6 +495,9 @@ osm_routing_engine_type_t osm_routing_engine_type(IN const char *str); * SEE ALSO *********/ +void osm_opensm_report_event(osm_opensm_t *osm, osm_epi_event_id_t event_id, + void *event_data); + /* dump helpers */ void osm_dump_mcast_routes(osm_opensm_t * osm); void osm_dump_all(osm_opensm_t * osm); diff --git a/opensm/include/opensm/osm_perfmgr.h b/opensm/include/opensm/osm_perfmgr.h index 50a2c32..874f6cf 100644 --- a/opensm/include/opensm/osm_perfmgr.h +++ b/opensm/include/opensm/osm_perfmgr.h @@ -105,6 +105,7 @@ typedef struct _monitored_node { redir_t redir_port[1]; /* redirection on a per port basis */ } __monitored_node_t; +struct _osm_opensm_t; /****s* OpenSM: PerfMgr/osm_perfmgr_t * This object should be treated as opaque and should * be manipulated only through the provided functions. @@ -112,6 +113,7 @@ typedef struct _monitored_node { typedef struct _osm_perfmgr { cl_event_t sig_sweep; cl_timer_t sweep_timer; + struct _osm_opensm_t *osm; osm_subn_t *subn; osm_sm_t *sm; cl_plock_t *lock; @@ -130,7 +132,6 @@ typedef struct _osm_perfmgr { uint32_t max_outstanding_queries; cl_qmap_t monitored_map; /* map the nodes we are tracking */ __monitored_node_t *remove_list; - osm_epi_plugin_t *event_plugin; } osm_perfmgr_t; /* * FIELDS @@ -223,47 +224,19 @@ void osm_perfmgr_process(osm_perfmgr_t * pm); /****f* OpenSM: PerfMgr/osm_perfmgr_init */ ib_api_status_t osm_perfmgr_init(osm_perfmgr_t * const perfmgr, - osm_subn_t * const subn, - osm_sm_t * const sm, - osm_log_t * const log, - osm_mad_pool_t * const mad_pool, - osm_vendor_t * const vendor, - cl_dispatcher_t * const disp, - cl_plock_t * const lock, - const osm_subn_opt_t * const p_opt, - osm_epi_plugin_t * event_plugin); + struct _osm_opensm_t *osm, + const osm_subn_opt_t * const p_opt); /* * PARAMETERS * perfmgr * [in] Pointer to an osm_perfmgr_t object to initialize. * -* subn -* [in] Pointer to the Subnet object for this subnet. -* -* sm -* [in] Pointer to the Subnet object for this subnet. -* -* log -* [in] Pointer to the log object. -* -* mad_pool -* [in] Pointer to the MAD pool. -* -* vendor -* [in] Pointer to the vendor specific interfaces object. -* -* disp -* [in] Pointer to the OpenSM central Dispatcher. -* -* lock -* [in] Pointer to the OpenSM serializing lock. +* osm +* [in] Pointer to the OpenSM object. * * p_opt * [in] Starting options * -* event_plugin -* [in] Event plugin (Can be NULL if not available) -* * RETURN VALUES * IB_SUCCESS if the PerfMgr object was initialized successfully. *********/ diff --git a/opensm/include/opensm/osm_perfmgr_db.h b/opensm/include/opensm/osm_perfmgr_db.h index 2c4c520..79f369e 100644 --- a/opensm/include/opensm/osm_perfmgr_db.h +++ b/opensm/include/opensm/osm_perfmgr_db.h @@ -34,12 +34,11 @@ #ifndef _PERFMGR_EVENT_DB_H_ #define _PERFMGR_EVENT_DB_H_ +#include #include -#include #include #include #include -#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -50,6 +49,8 @@ #endif /* __cplusplus */ BEGIN_C_DECLS + +struct _osm_perfmgr; /****h* OpenSM/PerfMgr Event Database * DESCRIPTION * Database interface to record subnet events @@ -149,15 +150,13 @@ typedef struct _db_node { typedef struct _db { cl_qmap_t pc_data; /* stores type (_db_node_t *) */ cl_plock_t lock; - osm_log_t *osm_log; - osm_epi_plugin_t *event_plugin; + struct _osm_perfmgr *perfmgr; } perfmgr_db_t; /** * functions */ -perfmgr_db_t *perfmgr_db_construct(osm_log_t * p_log, - osm_epi_plugin_t * event_plugin); +perfmgr_db_t *perfmgr_db_construct(struct _osm_perfmgr *perfmgr); void perfmgr_db_destroy(perfmgr_db_t * db); perfmgr_db_err_t perfmgr_db_create_entry(perfmgr_db_t * db, uint64_t guid, diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index bda48a3..f74465e 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -1121,8 +1121,8 @@ static void perfmgr_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) osm_perfmgr_get_sweep_time_s(&(p_osm->perfmgr)), p_osm->perfmgr.outstanding_queries, p_osm->perfmgr.max_outstanding_queries, - p_osm->perfmgr.event_plugin ? - p_osm->perfmgr.event_plugin->plugin_name : "NONE"); + p_osm->event_plugin ? + p_osm->event_plugin->plugin_name : "NONE"); } } #endif /* ENABLE_OSM_PERF_MGR */ diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c index 42282c6..3575060 100644 --- a/opensm/opensm/osm_event_plugin.c +++ b/opensm/opensm/osm_event_plugin.c @@ -137,11 +137,3 @@ void osm_epi_destroy(osm_epi_plugin_t * plugin) free(plugin); } } - -void -osm_epi_report(osm_epi_plugin_t * plugin, osm_epi_event_id_t event_id, - void *event_data) -{ - if (plugin && plugin->impl->report) - plugin->impl->report(plugin->plugin_data, event_id, event_data); -} diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 167caab..a85bcb7 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -57,6 +57,7 @@ #include #include #include +#include struct routing_engine_module { const char *name; @@ -351,15 +352,7 @@ osm_opensm_init(IN osm_opensm_t * const p_osm, p_opt->event_plugin_name); #ifdef ENABLE_OSM_PERF_MGR - status = osm_perfmgr_init(&p_osm->perfmgr, - &p_osm->subn, - &p_osm->sm, - &p_osm->log, - &p_osm->mad_pool, - p_osm->p_vendor, - &p_osm->disp, - &p_osm->lock, p_opt, p_osm->event_plugin); - + status = osm_perfmgr_init(&p_osm->perfmgr, p_osm, p_opt); if (status != IB_SUCCESS) goto Exit; #endif /* ENABLE_OSM_PERF_MGR */ @@ -407,3 +400,13 @@ Exit: OSM_LOG_EXIT(&p_osm->log); return (status); } + +/********************************************************************** + **********************************************************************/ +void osm_opensm_report_event(osm_opensm_t *osm, osm_epi_event_id_t event_id, + void *event_data) +{ + if (osm->event_plugin && osm->event_plugin->impl->report) + osm->event_plugin->impl->report(osm->event_plugin->plugin_data, + event_id, event_data); +} diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index 1633c41..315a065 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -51,18 +51,18 @@ #include #include #include -#include -#include -#include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include -#include -#include -#include #define OSM_PERFMGR_INITIAL_TID_VALUE 0xcafe @@ -1213,51 +1213,43 @@ Exit: * Initialize the PerfMgr object **********************************************************************/ ib_api_status_t -osm_perfmgr_init(osm_perfmgr_t * const pm, - osm_subn_t * const subn, - osm_sm_t * const sm, - osm_log_t * const log, - osm_mad_pool_t * const mad_pool, - osm_vendor_t * const vendor, - cl_dispatcher_t * const disp, - cl_plock_t * const lock, - const osm_subn_opt_t * const p_opt, - osm_epi_plugin_t * event_plugin) +osm_perfmgr_init(osm_perfmgr_t * const pm, osm_opensm_t *osm, + const osm_subn_opt_t * const p_opt) { ib_api_status_t status = IB_SUCCESS; - OSM_LOG_ENTER(log); + OSM_LOG_ENTER(&osm->log); - OSM_LOG(log, OSM_LOG_VERBOSE, "Initializing PerfMgr\n"); + OSM_LOG(&osm->log, OSM_LOG_VERBOSE, "Initializing PerfMgr\n"); memset(pm, 0, sizeof(*pm)); cl_event_construct(&pm->sig_sweep); cl_event_init(&pm->sig_sweep, FALSE); - pm->subn = subn; - pm->sm = sm; - pm->log = log; - pm->mad_pool = mad_pool; - pm->vendor = vendor; + pm->subn = &osm->subn; + pm->sm = &osm->sm; + pm->log = &osm->log; + pm->mad_pool = &osm->mad_pool; + pm->vendor = osm->p_vendor; pm->trans_id = OSM_PERFMGR_INITIAL_TID_VALUE; - pm->lock = lock; + pm->lock = &osm->lock; pm->state = p_opt->perfmgr ? PERFMGR_STATE_ENABLED : PERFMGR_STATE_DISABLE; pm->sweep_time_s = p_opt->perfmgr_sweep_time_s; pm->max_outstanding_queries = p_opt->perfmgr_max_outstanding_queries; - pm->event_plugin = event_plugin; + pm->osm = osm; status = cl_timer_init(&pm->sweep_timer, perfmgr_sweep, pm); if (status != IB_SUCCESS) goto Exit; - pm->db = perfmgr_db_construct(pm->log, pm->event_plugin); + pm->db = perfmgr_db_construct(pm); if (!pm->db) { pm->state = PERFMGR_STATE_NO_DB; goto Exit; } - pm->pc_disp_h = cl_disp_register(disp, OSM_MSG_MAD_PORT_COUNTERS, + pm->pc_disp_h = cl_disp_register(&osm->disp, OSM_MSG_MAD_PORT_COUNTERS, osm_pc_rcv_process, pm); if (pm->pc_disp_h == CL_DISP_INVALID_HANDLE) goto Exit; @@ -1267,7 +1259,7 @@ osm_perfmgr_init(osm_perfmgr_t * const pm, cl_timer_start(&pm->sweep_timer, pm->sweep_time_s * 1000); Exit: - OSM_LOG_EXIT(log); + OSM_LOG_EXIT(pm->log); return (status); } diff --git a/opensm/opensm/osm_perfmgr_db.c b/opensm/opensm/osm_perfmgr_db.c index 21ba923..031c9ac 100644 --- a/opensm/opensm/osm_perfmgr_db.c +++ b/opensm/opensm/osm_perfmgr_db.c @@ -42,11 +42,12 @@ #include #include +#include +#include /** ========================================================================= */ -perfmgr_db_t *perfmgr_db_construct(osm_log_t * p_log, - osm_epi_plugin_t * event_plugin) +perfmgr_db_t *perfmgr_db_construct(osm_perfmgr_t *perfmgr) { perfmgr_db_t *db = malloc(sizeof(*db)); if (!db) @@ -55,8 +56,7 @@ perfmgr_db_t *perfmgr_db_construct(osm_log_t * p_log, cl_qmap_init(&(db->pc_data)); cl_plock_construct(&(db->lock)); cl_plock_init(&(db->lock)); - db->osm_log = p_log; - db->event_plugin = event_plugin; + db->perfmgr = perfmgr; return ((void *)db); } @@ -178,57 +178,59 @@ static inline void debug_dump_err_reading(perfmgr_db_t * db, uint64_t guid, uint8_t port_num, _db_port_t * port, perfmgr_db_err_reading_t * cur) { - if (!osm_log_is_active(db->osm_log, OSM_LOG_DEBUG)) + osm_log_t *log = db->perfmgr->log; + + if (!osm_log_is_active(log, OSM_LOG_DEBUG)) return; /* optimize this a bit */ - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "GUID 0x%" PRIx64 " Port %u:\n", guid, port_num); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "sym %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->symbol_err_cnt, port->err_previous.symbol_err_cnt, port->err_total.symbol_err_cnt); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "ler %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->link_err_recover, port->err_previous.link_err_recover, port->err_total.link_err_recover); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "ld %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->link_downed, port->err_previous.link_downed, port->err_total.link_downed); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "re %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->rcv_err, port->err_previous.rcv_err, port->err_total.rcv_err); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "rrp %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->rcv_rem_phys_err, port->err_previous.rcv_rem_phys_err, port->err_total.rcv_rem_phys_err); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "rsr %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->rcv_switch_relay_err, port->err_previous.rcv_switch_relay_err, port->err_total.rcv_switch_relay_err); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "xd %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->xmit_discards, port->err_previous.xmit_discards, port->err_total.xmit_discards); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "xce %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->xmit_constraint_err, port->err_previous.xmit_constraint_err, port->err_total.xmit_constraint_err); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "rce %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->rcv_constraint_err, port->err_previous.rcv_constraint_err, port->err_total.rcv_constraint_err); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "li %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->link_integrity, port->err_previous.link_integrity, port->err_total.link_integrity); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "bo %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->buffer_overrun, port->err_previous.buffer_overrun, port->err_total.buffer_overrun); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "vld %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->vl15_dropped, port->err_previous.vl15_dropped, port->err_total.vl15_dropped); @@ -302,8 +304,8 @@ perfmgr_db_add_err_reading(perfmgr_db_t * db, uint64_t guid, p_port->err_previous = *reading; - osm_epi_report(db->event_plugin, OSM_EVENT_ID_PORT_ERRORS, - (void *)&epi_pe_data); + osm_opensm_report_event(db->perfmgr->osm, OSM_EVENT_ID_PORT_ERRORS, + &epi_pe_data); Exit: cl_plock_release(&(db->lock)); @@ -356,21 +358,22 @@ static inline void debug_dump_dc_reading(perfmgr_db_t * db, uint64_t guid, uint8_t port_num, _db_port_t * port, perfmgr_db_data_cnt_reading_t * cur) { - if (!osm_log_is_active(db->osm_log, OSM_LOG_DEBUG)) + osm_log_t *log = db->perfmgr->log; + if (!osm_log_is_active(log, OSM_LOG_DEBUG)) return; /* optimize this a big */ - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "xd %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->xmit_data, port->dc_previous.xmit_data, port->dc_total.xmit_data); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "rd %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->rcv_data, port->dc_previous.rcv_data, port->dc_total.rcv_data); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "xp %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->xmit_pkts, port->dc_previous.xmit_pkts, port->dc_total.xmit_pkts); - osm_log(db->osm_log, OSM_LOG_DEBUG, + osm_log(log, OSM_LOG_DEBUG, "rp %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", cur->rcv_pkts, port->dc_previous.rcv_pkts, port->dc_total.rcv_pkts); } @@ -426,8 +429,8 @@ perfmgr_db_add_dc_reading(perfmgr_db_t * db, uint64_t guid, p_port->dc_previous = *reading; - osm_epi_report(db->event_plugin, OSM_EVENT_ID_PORT_DATA_COUNTERS, - (void *)&epi_dc_data); + osm_opensm_report_event(db->perfmgr->osm, + OSM_EVENT_ID_PORT_DATA_COUNTERS, &epi_dc_data); Exit: cl_plock_release(&(db->lock)); -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sat Jun 14 05:57:37 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 14 Jun 2008 15:57:37 +0300 Subject: [ofa-general] [PATCH] opensm: support for multiple event plugins In-Reply-To: <20080614125529.GD24365@sashak.voltaire.com> References: <20080614125412.GC24365@sashak.voltaire.com> <20080614125529.GD24365@sashak.voltaire.com> Message-ID: <20080614125737.GE24365@sashak.voltaire.com> More than one event plugin can be passed to OpenSM via config file as space separated list: event_plugin name1 name2 name3 Each name will be resolved as usual - lib.so. Signed-off-by: Sasha Khapyorsky --- Ira, I didn't check this with opensmskumme plugin, but guess it should not be affected. opensm/include/opensm/osm_event_plugin.h | 4 ++- opensm/include/opensm/osm_opensm.h | 4 +- opensm/opensm/osm_console.c | 13 ++++++++--- opensm/opensm/osm_opensm.c | 34 +++++++++++++++++++++++++---- 4 files changed, 43 insertions(+), 12 deletions(-) diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h index a7cad32..7467694 100644 --- a/opensm/include/opensm/osm_event_plugin.h +++ b/opensm/include/opensm/osm_event_plugin.h @@ -35,8 +35,9 @@ #define _OSM_EVENT_PLUGIN_H_ #include -#include #include +#include +#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -161,6 +162,7 @@ typedef struct osm_event_plugin { * The plugin structure should be considered opaque */ typedef struct osm_epi_plugin { + cl_list_item_t list; void *handle; osm_event_plugin_t *impl; void *plugin_data; diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index 63d2247..1b758bd 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -44,6 +44,7 @@ #define _OSM_OPENSM_H_ #include +#include #include #include #include @@ -54,7 +55,6 @@ #include #include #include -#include #include #include #include @@ -168,7 +168,7 @@ typedef struct _osm_opensm_t { #ifdef ENABLE_OSM_PERF_MGR osm_perfmgr_t perfmgr; #endif /* ENABLE_OSM_PERF_MGR */ - osm_epi_plugin_t *event_plugin; + cl_qlist_t plugin_list; osm_db_t db; osm_mad_pool_t mad_pool; osm_vendor_t *p_vendor; diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index f74465e..ca4982f 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -1110,19 +1110,24 @@ static void perfmgr_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) fprintf(out, "\"%s\" option not found\n", p_cmd); } } else { + cl_list_item_t *item; fprintf(out, "Performance Manager status:\n" "state : %s\n" "sweep state : %s\n" "sweep time : %us\n" "outstanding queries/max : %d/%u\n" - "loaded event plugin : %s\n", + "loaded event plugin :", osm_perfmgr_get_state_str(&(p_osm->perfmgr)), osm_perfmgr_get_sweep_state_str(&(p_osm->perfmgr)), osm_perfmgr_get_sweep_time_s(&(p_osm->perfmgr)), p_osm->perfmgr.outstanding_queries, - p_osm->perfmgr.max_outstanding_queries, - p_osm->event_plugin ? - p_osm->event_plugin->plugin_name : "NONE"); + p_osm->perfmgr.max_outstanding_queries); + for (item = cl_qlist_head(&p_osm->plugin_list); + item != cl_qlist_end(&p_osm->plugin_list); + item = cl_qlist_next(item)) + fprintf(out, " %s", + ((osm_epi_plugin_t *)item)->plugin_name); + fprintf(out, "\n"); } } #endif /* ENABLE_OSM_PERF_MGR */ diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index a85bcb7..a1d4aa9 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -243,6 +243,23 @@ void osm_opensm_destroy(IN osm_opensm_t * const p_osm) osm_log_destroy(&p_osm->log); } +static void load_plugins(osm_opensm_t *osm, char *plugin_names) +{ + osm_epi_plugin_t *epi; + char *name, *p; + + name = strtok_r(plugin_names, " \t\n", &p); + while (name && *name) { + epi = osm_epi_construct(&osm->log, name); + if (!epi) + osm_log(&osm->log, OSM_LOG_ERROR, + "cannot load plugin \'%s\'\n", name); + else + cl_qlist_insert_tail(&osm->plugin_list, &epi->list); + name = strtok_r(NULL, " \t\n", &p); + } +} + /********************************************************************** **********************************************************************/ ib_api_status_t @@ -348,8 +365,9 @@ osm_opensm_init(IN osm_opensm_t * const p_osm, if (status != IB_SUCCESS) goto Exit; - p_osm->event_plugin = osm_epi_construct(&p_osm->log, - p_opt->event_plugin_name); + cl_qlist_init(&p_osm->plugin_list); + + load_plugins(p_osm, p_opt->event_plugin_name); #ifdef ENABLE_OSM_PERF_MGR status = osm_perfmgr_init(&p_osm->perfmgr, p_osm, p_opt); @@ -406,7 +424,13 @@ Exit: void osm_opensm_report_event(osm_opensm_t *osm, osm_epi_event_id_t event_id, void *event_data) { - if (osm->event_plugin && osm->event_plugin->impl->report) - osm->event_plugin->impl->report(osm->event_plugin->plugin_data, - event_id, event_data); + cl_list_item_t *item; + + for (item = cl_qlist_head(&osm->plugin_list); + item != cl_qlist_end(&osm->plugin_list); + item = cl_qlist_next(item)) { + osm_epi_plugin_t *p = (osm_epi_plugin_t *)item; + if (p->impl->report) + p->impl->report(p->plugin_data, event_id, event_data); + } } -- 1.5.5.1.178.g1f811 From hrosenstock at xsigo.com Sat Jun 14 06:01:26 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sat, 14 Jun 2008 06:01:26 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_subnet.c: Fix typo Message-ID: <1213448486.14047.585.camel@hrosenstock-ws.xsigo.com> pensm/osm_subnet.c: Fix typo Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index dd71fa3..02dfad7 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -1629,7 +1629,7 @@ int osm_subn_write_conf_file(char *file_name, IN osm_subn_opt_t *const p_opts) "event_plugin_name %s\n\n", p_opts->event_plugin_name); fprintf(opts_file, - "#\n# Node name map for mapping node's to more descirptive node descriptors\n" + "#\n# Node name map for mapping node's to more descriptive node descriptions\n" "# (man ibnetdiscover for more information)\n#\n" "node_name_map_name %s\n\n", p_opts->node_name_map_name ? p_opts->node_name_map_name : "(null)"); From roland.list at gmail.com Sat Jun 14 11:03:08 2008 From: roland.list at gmail.com (Roland Dreier) Date: Sat, 14 Jun 2008 11:03:08 -0700 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <48513E4E.4040601@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> Message-ID: > We have linkstate/operstate for this. How is a user supposed > to configure the network device when it appears at a more or > less random time from his perspective? Well, keep in mind that this driver is for an ethernet virtual NIC that is actually remote across another network (an InfiniBand fabric) -- so these devices can actually appear or disappear at random times by their nature. It's a similar issue to, say, a USB ethernet adapter presents -- there's no sane way to have operstate set for a device that hasn't been attached yet. - R. From roland.list at gmail.com Sat Jun 14 11:05:10 2008 From: roland.list at gmail.com (Roland Dreier) Date: Sat, 14 Jun 2008 11:05:10 -0700 Subject: [ofa-general] mlx4_core CQ overrun In-Reply-To: References: Message-ID: > Thanks for your prompt response. So it is not possible a driver or FW bug? > We will recheck our test. It could definitely still be a driver or FW bug. But a real CQ overrun seems the most likely cause to me right now... without the test code it's really hard to say. From xma at us.ibm.com Sat Jun 14 11:18:00 2008 From: xma at us.ibm.com (Shirley Ma) Date: Sat, 14 Jun 2008 11:18:00 -0700 Subject: [ofa-general] mlx4_core CQ overrun In-Reply-To: Message-ID: "Roland Dreier" wrote on 06/14/2008 11:05:10 AM: > > Thanks for your prompt response. So it is not possible a driver or FW bug? > > We will recheck our test. > > It could definitely still be a driver or FW bug. But a real CQ > overrun seems the > most likely cause to me right now... without the test code it's really > hard to say. Thanks Roland. Test code is under debugging, I will let you know the results. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jgunthorpe at obsidianresearch.com Sat Jun 14 12:03:58 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Sat, 14 Jun 2008 13:03:58 -0600 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> Message-ID: <20080614190358.GU22807@obsidianresearch.com> On Sat, Jun 14, 2008 at 11:03:08AM -0700, Roland Dreier wrote: > > We have linkstate/operstate for this. How is a user supposed > > to configure the network device when it appears at a more or > > less random time from his perspective? > Well, keep in mind that this driver is for an ethernet virtual NIC > that is actually > remote across another network (an InfiniBand fabric) -- so these devices > can actually appear or disappear at random times by their nature. It's a > similar issue to, say, a USB ethernet adapter presents -- there's no sane way > to have operstate set for a device that hasn't been attached yet. I think that is only a fair analogy if the VNIC used some kind of hot-plug auto discovery procedure like USB does. As soon as you require the administrator to run a command to bind a device to a VNIC GID you are much more like a tunnel device and the ethernet device should exist from the moment the administator creates it up until the administator destroys it. The state of the connection to the VNIC should be reflected in some way other than device presence. Jason From ogerlitz at voltaire.com Sat Jun 14 22:57:24 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 15 Jun 2008 08:57:24 +0300 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: <4852832D.4000404@opengridcomputing.com> References: <20080612211302.25120.41155.stgit@dell3.ogc.int> <4852832D.4000404@opengridcomputing.com> Message-ID: <4854AF44.1090704@voltaire.com> Steve Wise wrote: > I don't see any counter stuff in /sys on my mthca system. Where > should it be and do I need to load a specific module to get these stats? see /sys/class/infiniband/$device/ports/$port/counters Or. From sashak at voltaire.com Sat Jun 14 23:10:32 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:10:32 +0300 Subject: [ofa-general] [PATCH v2] opensm: support for multiple event plugins In-Reply-To: <20080614125737.GE24365@sashak.voltaire.com> References: <20080614125412.GC24365@sashak.voltaire.com> <20080614125529.GD24365@sashak.voltaire.com> <20080614125737.GE24365@sashak.voltaire.com> Message-ID: <20080615061032.GH24365@sashak.voltaire.com> More than one event plugin can be passed to OpenSM via config file as space separated list: event_plugin name1 name2 name3 Each name will be resolved as usual - lib.so. Signed-off-by: Sasha Khapyorsky --- The difference against original version is "NONE" default event plugin name removing - it is not needed anymore, but when was in use caused segmentation fault due to constant string parsing by strtok_r(). opensm/include/opensm/osm_event_plugin.h | 11 +++------ opensm/include/opensm/osm_opensm.h | 4 +- opensm/opensm/osm_console.c | 13 +++++++--- opensm/opensm/osm_event_plugin.c | 3 +- opensm/opensm/osm_opensm.c | 35 +++++++++++++++++++++++++---- opensm/opensm/osm_subnet.c | 5 ++- 6 files changed, 49 insertions(+), 22 deletions(-) diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h index a7cad32..d9e546c 100644 --- a/opensm/include/opensm/osm_event_plugin.h +++ b/opensm/include/opensm/osm_event_plugin.h @@ -35,8 +35,9 @@ #define _OSM_EVENT_PLUGIN_H_ #include -#include #include +#include +#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -57,12 +58,7 @@ BEGIN_C_DECLS * Ira Weiny, LLNL * *********/ -#define OSM_EVENT_PLUGIN_NAME_NONE "NONE" -#ifdef ENABLE_OSM_DEFAULT_EVENT_PLUGIN -#define OSM_DEFAULT_EVENT_PLUGIN_NAME "osmeventplugin" -#else /* !ENABLE_OSM_DEFAULT_EVENT_PLUGIN */ -#define OSM_DEFAULT_EVENT_PLUGIN_NAME OSM_EVENT_PLUGIN_NAME_NONE -#endif /* ENABLE_OSM_DEFAULT_EVENT_PLUGIN */ + #define OSM_EPI_NODE_NAME_LEN (128) /** ========================================================================= * Event types @@ -161,6 +157,7 @@ typedef struct osm_event_plugin { * The plugin structure should be considered opaque */ typedef struct osm_epi_plugin { + cl_list_item_t list; void *handle; osm_event_plugin_t *impl; void *plugin_data; diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index 63d2247..1b758bd 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -44,6 +44,7 @@ #define _OSM_OPENSM_H_ #include +#include #include #include #include @@ -54,7 +55,6 @@ #include #include #include -#include #include #include #include @@ -168,7 +168,7 @@ typedef struct _osm_opensm_t { #ifdef ENABLE_OSM_PERF_MGR osm_perfmgr_t perfmgr; #endif /* ENABLE_OSM_PERF_MGR */ - osm_epi_plugin_t *event_plugin; + cl_qlist_t plugin_list; osm_db_t db; osm_mad_pool_t mad_pool; osm_vendor_t *p_vendor; diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index f74465e..ca4982f 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -1110,19 +1110,24 @@ static void perfmgr_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) fprintf(out, "\"%s\" option not found\n", p_cmd); } } else { + cl_list_item_t *item; fprintf(out, "Performance Manager status:\n" "state : %s\n" "sweep state : %s\n" "sweep time : %us\n" "outstanding queries/max : %d/%u\n" - "loaded event plugin : %s\n", + "loaded event plugin :", osm_perfmgr_get_state_str(&(p_osm->perfmgr)), osm_perfmgr_get_sweep_state_str(&(p_osm->perfmgr)), osm_perfmgr_get_sweep_time_s(&(p_osm->perfmgr)), p_osm->perfmgr.outstanding_queries, - p_osm->perfmgr.max_outstanding_queries, - p_osm->event_plugin ? - p_osm->event_plugin->plugin_name : "NONE"); + p_osm->perfmgr.max_outstanding_queries); + for (item = cl_qlist_head(&p_osm->plugin_list); + item != cl_qlist_end(&p_osm->plugin_list); + item = cl_qlist_next(item)) + fprintf(out, " %s", + ((osm_epi_plugin_t *)item)->plugin_name); + fprintf(out, "\n"); } } #endif /* ENABLE_OSM_PERF_MGR */ diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c index 3575060..e579fad 100644 --- a/opensm/opensm/osm_event_plugin.c +++ b/opensm/opensm/osm_event_plugin.c @@ -63,8 +63,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) char lib_name[OSM_PATH_MAX]; osm_epi_plugin_t *rc = NULL; - if (!plugin_name || - strcmp(plugin_name, OSM_EVENT_PLUGIN_NAME_NONE) == 0) + if (!plugin_name || !*plugin_name) return (NULL); /* find the plugin */ diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index a85bcb7..adacd93 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -243,6 +243,23 @@ void osm_opensm_destroy(IN osm_opensm_t * const p_osm) osm_log_destroy(&p_osm->log); } +static void load_plugins(osm_opensm_t *osm, char *plugin_names) +{ + osm_epi_plugin_t *epi; + char *name, *p; + + name = strtok_r(plugin_names, " \t\n", &p); + while (name && *name) { + epi = osm_epi_construct(&osm->log, name); + if (!epi) + osm_log(&osm->log, OSM_LOG_ERROR, + "cannot load plugin \'%s\'\n", name); + else + cl_qlist_insert_tail(&osm->plugin_list, &epi->list); + name = strtok_r(NULL, " \t\n", &p); + } +} + /********************************************************************** **********************************************************************/ ib_api_status_t @@ -348,8 +365,10 @@ osm_opensm_init(IN osm_opensm_t * const p_osm, if (status != IB_SUCCESS) goto Exit; - p_osm->event_plugin = osm_epi_construct(&p_osm->log, - p_opt->event_plugin_name); + cl_qlist_init(&p_osm->plugin_list); + + if (p_opt->event_plugin_name) + load_plugins(p_osm, p_opt->event_plugin_name); #ifdef ENABLE_OSM_PERF_MGR status = osm_perfmgr_init(&p_osm->perfmgr, p_osm, p_opt); @@ -406,7 +425,13 @@ Exit: void osm_opensm_report_event(osm_opensm_t *osm, osm_epi_event_id_t event_id, void *event_data) { - if (osm->event_plugin && osm->event_plugin->impl->report) - osm->event_plugin->impl->report(osm->event_plugin->plugin_data, - event_id, event_data); + cl_list_item_t *item; + + for (item = cl_qlist_head(&osm->plugin_list); + item != cl_qlist_end(&osm->plugin_list); + item = cl_qlist_next(item)) { + osm_epi_plugin_t *p = (osm_epi_plugin_t *)item; + if (p->impl->report) + p->impl->report(p->plugin_data, event_id, event_data); + } } diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index a9081c9..0341805 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -431,7 +431,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * const p_opt) p_opt->event_db_dump_file = NULL; /* use default */ #endif /* ENABLE_OSM_PERF_MGR */ - p_opt->event_plugin_name = OSM_DEFAULT_EVENT_PLUGIN_NAME; + p_opt->event_plugin_name = NULL; p_opt->node_name_map_name = NULL; p_opt->dump_files_dir = getenv("OSM_TMP_DIR"); @@ -1631,7 +1631,8 @@ int osm_subn_write_conf_file(char *file_name, IN osm_subn_opt_t *const p_opts) fprintf(opts_file, "#\n# Event Plugin Options\n#\n" - "event_plugin_name %s\n\n", p_opts->event_plugin_name); + "event_plugin_name %s\n\n", p_opts->event_plugin_name ? + p_opts->event_plugin_name : null_str); fprintf(opts_file, "#\n# Node name map for mapping node's to more descirptive node descriptors\n" -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sat Jun 14 23:19:35 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:19:35 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_sa_mcmember_record.c: Better message on failure due to MC group not found In-Reply-To: <1213368790.14047.551.camel@hrosenstock-ws.xsigo.com> References: <1213368790.14047.551.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080615061935.GI24365@sashak.voltaire.com> On 07:53 Fri 13 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: Better message on failure due to MC > group not found > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 14 23:34:13 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:34:13 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_sa_mcmember_record.c: Better message on failure due to MC group not found In-Reply-To: <20080615061935.GI24365@sashak.voltaire.com> References: <1213368790.14047.551.camel@hrosenstock-ws.xsigo.com> <20080615061935.GI24365@sashak.voltaire.com> Message-ID: <20080615063413.GJ24365@sashak.voltaire.com> On 09:19 Sun 15 Jun , Sasha Khapyorsky wrote: > On 07:53 Fri 13 Jun , Hal Rosenstock wrote: > > opensm/osm_sa_mcmember_record.c: Better message on failure due to MC > > group not found > > > > Signed-off-by: Hal Rosenstock > > Applied. Thanks. Actually not. I noticed v2 and v3 of this and have some comments. Sasha From sashak at voltaire.com Sat Jun 14 23:37:27 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:37:27 +0300 Subject: [ofa-general] Re: [PATCHv3][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements In-Reply-To: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> References: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080615063727.GK24365@sashak.voltaire.com> On 08:30 Fri 13 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: Some error message improvements > > Signed-off-by: Hal Rosenstock > > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > index de174c5..df4eca3 100644 > --- a/opensm/opensm/osm_sa_mcmember_record.c > +++ b/opensm/opensm/osm_sa_mcmember_record.c > @@ -1121,7 +1121,10 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > > if (p_ctxt->p_mgrp) { > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B30: " > - "Multiple MC groups for same MGID\n"); > + "Multiple MC groups for MGID " > + "0x%016" PRIx64 " : 0x%016" PRIx64 "\n", > + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix), > + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id)); > return; > } > p_ctxt->p_mgrp = p_mgrp; > @@ -1278,8 +1281,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, > } > } else { > CL_PLOCK_RELEASE(sa->p_lock); > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > - "Failed since multicast group not present\n"); > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " > + "Failed since multicast group 0x%16" Why to change log level? It is not OpenSM error, but invalid request. Please next time if you are changing log level state it in commit comment message. Sasha > + PRIx64 " : 0x%016" PRIx64 " not present\n", > + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix), > + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id)); > osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); > goto Exit; > } > > From sashak at voltaire.com Sat Jun 14 23:44:08 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:44:08 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_sa_mcmember_record.c: When consolidating SNM, need separate group per PKey In-Reply-To: <1213378109.14047.563.camel@hrosenstock-ws.xsigo.com> References: <1213378109.14047.563.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080615064408.GL24365@sashak.voltaire.com> On 10:28 Fri 13 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: When consolidating SNM, need separate > group per PKey > > If don't do this, leaves on one group can cause the entire group to be > prematurely removed > > Also, fix consolidation issue introduced at commit > a20dad09d2dbaecff6c05d4566fe8fac358b614e > > Also, cosmetic change to log message > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 14 23:45:16 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:45:16 +0300 Subject: [ofa-general] Re: [PATCH] opensm/man/opensm.8.in: Update consolidate_ipv6_snm_req description In-Reply-To: <1213378555.14047.568.camel@hrosenstock-ws.xsigo.com> References: <1213378555.14047.568.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080615064516.GM24365@sashak.voltaire.com> On 10:35 Fri 13 Jun , Hal Rosenstock wrote: > opensm/man/opensm.8.in: Update consolidate_ipv6_snm_req description > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sat Jun 14 23:48:08 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 09:48:08 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_subnet.c: Fix typo In-Reply-To: <1213448486.14047.585.camel@hrosenstock-ws.xsigo.com> References: <1213448486.14047.585.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080615064808.GN24365@sashak.voltaire.com> On 06:01 Sat 14 Jun , Hal Rosenstock wrote: > pensm/osm_subnet.c: Fix typo > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From kliteyn at dev.mellanox.co.il Sun Jun 15 01:17:00 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sun, 15 Jun 2008 11:17:00 +0300 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <1213397313.5369.356.camel@cardanus.llnl.gov> References: <1213397313.5369.356.camel@cardanus.llnl.gov> Message-ID: <4854CFFC.7040508@dev.mellanox.co.il> Hi Al, Al Chu wrote: > Hey Sasha, > > This is a conceptually simple option I've developed for updn routing. > > Currently in updn routing, nodes/guids are routed on switches in a > seemingly-random order, which I believe is due to internal data > structure organization (i.e. cl_qmap_apply_func is called on > port_guid_tbl) as well as how the fabric is scanned (it is logically > scanned from a port perspective, but it may not be logical from a node > perspective). I had a hypothesis that this was leading to increased > contention in the network for MPI. > > For example, suppose we have 12 uplinks from a leaf switch to a spine > switch. If we want to send data from this leaf switch to node[13-24], > the up links we will send on are pretty random. It's because: > > A) node[13-24] are individually routed at seemingly-random points based > on when they are called by cl_qmap_apply_func(). > > B) the ports chosen for routing are based on least used port usage. > > C) least used port usage is based on whatever was routed earlier on. > > So I developed this patch series, which supports an option called > "guid_routing_order_file" which allows the user to input a file with a > list of port_guids which will indicate the order in which guids are > routed instead (naturally, those guids not listed are routed last). Great idea! I understand that this guid_routing_order_file is synchronized with an MPI rank file, right? If not, then synchronizing them might give even better results. Another idea: OpenSM can create such file (list, doesn't have to be actual file) automatically, just by checking topologically-adjacent leaf switches and their HCAs. > I list the port guids of the nodes of the cluster from node0 to nodeN, one > per line in the file. By listing the nodes in this order, I believe we > could get less contention in the network. In the example above, sending > to node[13-24] should use all of the 12 uplinks, b/c the ports will be > equally used b/c nodes[1-12] were routed beforehand in order. > > The results from some tests are pretty impressive when I do this. LMC=0 > average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s > when I use guid_routing_order. Can you compare this to the fat-tree routing? Conceptually, fat-tree is doing the same - it routes LIDs on nodes in a topological order, so it would be interesting to see the comparison. Also, fat-tree produces the guid order file automatically, but nobody used it yet as an input to produce MPI rank file. -- Yevgeny > A variety of other positive performance > increases were found when doing other tests, other MPIs, and other LMCs > if anyone is interested. > > BTW, I developed this patch series before your preserve-base-lid patch > series. It will 100% conflict with the preserve-base-lid patch series. > I will fix this patch series once the preserve-base-lids patch series is > committed to git. I'm just looking for comments right now. > > Al > From vlad at lists.openfabrics.org Sun Jun 15 02:17:00 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 15 Jun 2008 02:17:00 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080615-0200 daily build status Message-ID: <20080615091700.DF8F7E284A0@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.24 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 1 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: warning: passing argument 2 of 'ip_route_output_key' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.c:1034: error: too many arguments to function 'ip_route_output_key' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes/nes_cm.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:930: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from include/linux/inetdevice.h:8, from /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: for each function it appears in.) /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.c:907: error: too many arguments to function 'dev_get_by_name' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3/cxio_hal.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/cxgb3] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080615-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From sashak at voltaire.com Sun Jun 15 04:50:34 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 14:50:34 +0300 Subject: [ofa-general] [PATCH] opensm/osm_console.c: fix guid parsing Message-ID: <20080615115034.GW24365@sashak.voltaire.com> Fix switch guid parsing used in lidbalance and switchbalance commands, use strtoull() instead of strtoul() which truncates 64-bit GUID value on 32-bit machines. Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_console.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 6db7aa6..59b2ed9 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -828,7 +828,7 @@ static void switchbalance_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) } if (p_cmd) { - guid = strtoul(p_cmd, &p_end, 0); + guid = strtoull(p_cmd, &p_end, 0); if (!guid || *p_end != '\0') { fprintf(out, "Invalid guid specified\n"); help_switchbalance(out, 1); @@ -1031,7 +1031,7 @@ static void lidbalance_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) if (p_cmd) { char *p_end; - switchguid = strtoul(p_cmd, &p_end, 0); + switchguid = strtoull(p_cmd, &p_end, 0); if (!switchguid || *p_end != '\0') { fprintf(out, "Invalid switchguid specified\n"); help_lidbalance(out, 1); -- 1.5.5.1.178.g1f811 From 40agmares at cox.net Sun Jun 15 03:15:03 2008 From: 40agmares at cox.net (freemon tex) Date: Sun, 15 Jun 2008 10:15:03 +0000 Subject: [ofa-general] to general Message-ID: <000601c8cedf$06f2b60f$81aaee86@udkeak> Get the great discounts on popular software today All sofware is instantly available to download - No Need Wait! ALL OUR SOFTWARES ON ALL EUROPEAN LANGUAGES - USA, English, France, Italy, Spanish, German and more!!! Windows XP Pro With SP2 - $59.95 Adobe Acrobat Pro 8 - $69.95 Office 2003 Pro - $59.95 Adobe Photoshop CS2 - $79.95 AutoCAD 2007 - $149.95 Also we have so much soft for MACINTOSH!!! Microsoft Office 2004 for MAC – $79.95 Adobe Acrobat 7 Professional for MAC – $59.95 Adobe Creative Suite 2 Premium for MAC – $229.95 Macromedia Dreamweaver 8 for MAC – $69.95 - Visit our site: www.xorgesoft[DOT]com (copy this link and replace "[DOT]" to ".") From moshek at voltaire.com Sun Jun 15 05:45:44 2008 From: moshek at voltaire.com (Moshe Kazir) Date: Sun, 15 Jun 2008 15:45:44 +0300 Subject: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp In-Reply-To: <39C75744D164D948A170E9792AF8E7CAC5AFD6@exil.voltaire.com> References: <39C75744D164D948A170E9792AF8E7CAC5AFD1@exil.voltaire.com> <4850C11C.80106@mellanox.co.il> <39C75744D164D948A170E9792AF8E7CAC5AFD6@exil.voltaire.com> Message-ID: <39C75744D164D948A170E9792AF8E7CAC5AFF0@exil.voltaire.com> Vlad, did you use the patch for OFED-1.4 ? Moshe ____________________________________________________________ Moshe Katzir | +972-9971-8639 (o) | +972-52-860-6042 (m) Voltaire - The Grid Backbone www.voltaire.com -----Original Message----- From: Moshe Kazir Sent: Thursday, June 12, 2008 9:32 AM To: 'tziporet at dev.mellanox.co.il' Cc: Vladimir Sokolovsky (Mellanox); general at lists.openfabrics.org; Olga Shern Subject: RE: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp OFED-1.3.1 & OFED-1.3 (OFA_kernel*src.rpm) have comppile errors on SLES10 SP1 Lustre without this patch .... Add it to OFED-1.4 ? Moshe ____________________________________________________________ Moshe Katzir | +972-9971-8639 (o) | +972-52-860-6042 (m) Voltaire - The Grid Backbone www.voltaire.com -----Original Message----- From: Tziporet Koren [mailto:tziporet at dev.mellanox.co.il] Sent: Thursday, June 12, 2008 9:24 AM To: Moshe Kazir Cc: Vladimir Sokolovsky (Mellanox); general at lists.openfabrics.org; Yair Goldel; Olga Shern; Avi Dahan Subject: Re: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp Moshe, What do you expect us to do with these backport patches? Tziporet From vlad at dev.mellanox.co.il Sun Jun 15 05:54:45 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Sun, 15 Jun 2008 15:54:45 +0300 Subject: [ofa-general] ofa_kernel-1.3.1 backport for kernel 2.6.16-54-0.2.5_lustre.1.6.4.3-smp In-Reply-To: <39C75744D164D948A170E9792AF8E7CAC5AFF0@exil.voltaire.com> References: <39C75744D164D948A170E9792AF8E7CAC5AFD1@exil.voltaire.com> <4850C11C.80106@mellanox.co.il> <39C75744D164D948A170E9792AF8E7CAC5AFD6@exil.voltaire.com> <39C75744D164D948A170E9792AF8E7CAC5AFF0@exil.voltaire.com> Message-ID: <48551115.1080306@dev.mellanox.co.il> Moshe Kazir wrote: > Vlad, > > did you use the patch for OFED-1.4 ? > > Moshe > > > ____________________________________________________________ > Moshe Katzir | +972-9971-8639 (o) | +972-52-860-6042 (m) > > Voltaire - The Grid Backbone > > www.voltaire.com > Hi Moshe, I did not applied this patch yet. I will take it when I'll prepare the daily build for OFED-1.4. Regards, Vladimir From sashak at voltaire.com Sun Jun 15 11:40:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 21:40:21 +0300 Subject: [ofa-general] [PATCH] opensm: make osm_perfmgr_db.* content dependent on PerfMgr availability Message-ID: <20080615184021.GX24365@sashak.voltaire.com> This wraps content of osm_perfmgr_db.[ch] by #ifdef ENABLE_OSM_PERF_MGR, and don't actually compile it when PerfMgr is disabled. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_perfmgr_db.h | 5 +++++ opensm/opensm/osm_perfmgr_db.c | 3 +++ 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/opensm/include/opensm/osm_perfmgr_db.h b/opensm/include/opensm/osm_perfmgr_db.h index 2c4c520..dca38a7 100644 --- a/opensm/include/opensm/osm_perfmgr_db.h +++ b/opensm/include/opensm/osm_perfmgr_db.h @@ -34,6 +34,8 @@ #ifndef _PERFMGR_EVENT_DB_H_ #define _PERFMGR_EVENT_DB_H_ +#ifdef ENABLE_OSM_PERF_MGR + #include #include #include @@ -201,4 +203,7 @@ void perfmgr_db_fill_data_cnt_read_epc(ib_port_counters_ext_t * wire_read, perfmgr_db_data_cnt_reading_t * reading); END_C_DECLS + +#endif /* ENABLE_OSM_PERF_MGR */ + #endif /* _PERFMGR_PM_DB_H_ */ diff --git a/opensm/opensm/osm_perfmgr_db.c b/opensm/opensm/osm_perfmgr_db.c index 21ba923..133e1d7 100644 --- a/opensm/opensm/osm_perfmgr_db.c +++ b/opensm/opensm/osm_perfmgr_db.c @@ -35,6 +35,8 @@ # include #endif /* HAVE_CONFIG_H */ +#ifdef ENABLE_OSM_PERF_MGR + #include #include #include @@ -798,3 +800,4 @@ perfmgr_db_fill_data_cnt_read_epc(ib_port_counters_ext_t * wire_read, reading->multicast_rcv_pkts = cl_ntoh64(wire_read->multicast_rcv_pkts); reading->time = time(NULL); } +#endif /* ENABLE_OSM_PERF_MGR */ -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sun Jun 15 11:42:10 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 21:42:10 +0300 Subject: [ofa-general] [PATCH] osm_vendor_select.h: make OSM_VENDOR_INTF_OPENIB to be default Message-ID: <20080615184210.GY24365@sashak.voltaire.com> This should be helpful for third party software (specifically OpenSM event plugin(s)) to not bother with vendor selection and compilation flags setup. Signed-off-by: Sasha Khapyorsky --- opensm/include/vendor/osm_vendor_select.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/include/vendor/osm_vendor_select.h b/opensm/include/vendor/osm_vendor_select.h index 8dcc05b..a7d12dc 100644 --- a/opensm/include/vendor/osm_vendor_select.h +++ b/opensm/include/vendor/osm_vendor_select.h @@ -58,7 +58,7 @@ #ifndef OSM_VENDOR_INTF_TS #ifndef OSM_VENDOR_INTF_SIM #ifndef OSM_VENDOR_INTF_AL -#define OSM_VENDOR_INTF_AL +#define OSM_VENDOR_INTF_OPENIB #endif /* AL */ #endif /* TS */ #endif /* SIM */ -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sun Jun 15 12:09:05 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 15 Jun 2008 22:09:05 +0300 Subject: [ofa-general] [PATCH] opensm: rename 'struct _osm_*_t' to 'struct osm_*' Message-ID: <20080615190905.GZ24365@sashak.voltaire.com> This renames OpenSM structures from 'struct _osm_*_t' to 'struct osm_*' form removing meaningless leading underscore character and '_t' prefix. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_attrib_req.h | 2 +- opensm/include/opensm/osm_console_io.h | 2 +- opensm/include/opensm/osm_db.h | 6 ++-- opensm/include/opensm/osm_db_pack.h | 2 +- opensm/include/opensm/osm_fwd_tbl.h | 2 +- opensm/include/opensm/osm_inform.h | 2 +- opensm/include/opensm/osm_lid_mgr.h | 2 +- opensm/include/opensm/osm_lin_fwd_tbl.h | 2 +- opensm/include/opensm/osm_log.h | 2 +- opensm/include/opensm/osm_mad_pool.h | 2 +- opensm/include/opensm/osm_madw.h | 30 ++++++++-------- opensm/include/opensm/osm_mcast_tbl.h | 2 +- opensm/include/opensm/osm_mcm_info.h | 2 +- opensm/include/opensm/osm_mcm_port.h | 2 +- opensm/include/opensm/osm_mtree.h | 6 ++-- opensm/include/opensm/osm_multicast.h | 2 +- opensm/include/opensm/osm_node.h | 6 ++-- opensm/include/opensm/osm_opensm.h | 2 +- opensm/include/opensm/osm_partition.h | 2 +- opensm/include/opensm/osm_path.h | 2 +- opensm/include/opensm/osm_perfmgr.h | 2 +- opensm/include/opensm/osm_pkey.h | 34 +++++++++--------- opensm/include/opensm/osm_port.h | 22 ++++++------ opensm/include/opensm/osm_port_profile.h | 2 +- opensm/include/opensm/osm_qos_policy.h | 14 ++++---- opensm/include/opensm/osm_rand_fwd_tbl.h | 2 +- opensm/include/opensm/osm_remote_sm.h | 2 +- opensm/include/opensm/osm_router.h | 2 +- opensm/include/opensm/osm_sa.h | 8 ++-- opensm/include/opensm/osm_sa_mad_ctrl.h | 8 ++-- opensm/include/opensm/osm_service.h | 2 +- opensm/include/opensm/osm_sm_mad_ctrl.h | 2 +- opensm/include/opensm/osm_stats.h | 2 +- opensm/include/opensm/osm_subnet.h | 48 +++++++++++++------------- opensm/include/opensm/osm_sweep_fail_ctrl.h | 2 +- opensm/include/opensm/osm_switch.h | 2 +- opensm/include/opensm/osm_ucast_mgr.h | 2 +- opensm/include/opensm/osm_vl15intf.h | 4 +- opensm/opensm/osm_db_files.c | 4 +- opensm/opensm/osm_inform.c | 2 +- opensm/opensm/osm_lid_mgr.c | 2 +- opensm/opensm/osm_mcast_mgr.c | 2 +- opensm/opensm/osm_port.c | 6 ++-- opensm/opensm/osm_sa_guidinfo_record.c | 4 +- opensm/opensm/osm_sa_informinfo.c | 4 +- opensm/opensm/osm_sa_lft_record.c | 4 +- opensm/opensm/osm_sa_link_record.c | 2 +- opensm/opensm/osm_sa_mcmember_record.c | 2 +- opensm/opensm/osm_sa_mft_record.c | 4 +- opensm/opensm/osm_sa_multipath_record.c | 4 +- opensm/opensm/osm_sa_node_record.c | 4 +- opensm/opensm/osm_sa_path_record.c | 4 +- opensm/opensm/osm_sa_pkey_record.c | 4 +- opensm/opensm/osm_sa_portinfo_record.c | 4 +- opensm/opensm/osm_sa_service_record.c | 4 +- opensm/opensm/osm_sa_slvl_record.c | 4 +- opensm/opensm/osm_sa_sminfo_record.c | 4 +- opensm/opensm/osm_sa_sw_info_record.c | 4 +- opensm/opensm/osm_sa_vlarb_record.c | 4 +- opensm/opensm/osm_trap_rcv.c | 2 +- opensm/opensm/osm_vl15intf.c | 2 +- 61 files changed, 160 insertions(+), 160 deletions(-) diff --git a/opensm/include/opensm/osm_attrib_req.h b/opensm/include/opensm/osm_attrib_req.h index 8a702bc..e79073b 100644 --- a/opensm/include/opensm/osm_attrib_req.h +++ b/opensm/include/opensm/osm_attrib_req.h @@ -80,7 +80,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_attrib_req { +typedef struct osm_attrib_req { uint16_t attrib_id; uint32_t attrib_mod; osm_madw_context_t context; diff --git a/opensm/include/opensm/osm_console_io.h b/opensm/include/opensm/osm_console_io.h index f8f10a5..cf9019f 100644 --- a/opensm/include/opensm/osm_console_io.h +++ b/opensm/include/opensm/osm_console_io.h @@ -64,7 +64,7 @@ #endif /* __cplusplus */ BEGIN_C_DECLS -typedef struct _osm_console_t { +typedef struct osm_console { int socket; int in_fd; int out_fd; diff --git a/opensm/include/opensm/osm_db.h b/opensm/include/opensm/osm_db.h index a35bae8..e937df9 100644 --- a/opensm/include/opensm/osm_db.h +++ b/opensm/include/opensm/osm_db.h @@ -88,8 +88,8 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_db_domain { - struct _osm_db *p_db; +typedef struct osm_db_domain { + struct osm_db *p_db; void *p_domain_imp; } osm_db_domain_t; /* @@ -116,7 +116,7 @@ typedef struct _osm_db_domain { * * SYNOPSIS */ -typedef struct _osm_db { +typedef struct osm_db { void *p_db_imp; osm_log_t *p_log; cl_list_t domains; diff --git a/opensm/include/opensm/osm_db_pack.h b/opensm/include/opensm/osm_db_pack.h index 45838da..83861a6 100644 --- a/opensm/include/opensm/osm_db_pack.h +++ b/opensm/include/opensm/osm_db_pack.h @@ -100,7 +100,7 @@ static inline osm_db_domain_t *osm_db_guid2lid_init(IN osm_db_t * const p_db) * * SYNOPSIS */ -typedef struct _osm_db_guid_elem { +typedef struct osm_db_guid_elem { cl_list_item_t item; uint64_t guid; } osm_db_guid_elem_t; diff --git a/opensm/include/opensm/osm_fwd_tbl.h b/opensm/include/opensm/osm_fwd_tbl.h index 35e469a..55e853f 100644 --- a/opensm/include/opensm/osm_fwd_tbl.h +++ b/opensm/include/opensm/osm_fwd_tbl.h @@ -88,7 +88,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_fwd_tbl_t { +typedef struct osm_fwd_tbl { osm_rand_fwd_tbl_t *p_rnd_tbl; osm_lin_fwd_tbl_t *p_lin_tbl; } osm_fwd_tbl_t; diff --git a/opensm/include/opensm/osm_inform.h b/opensm/include/opensm/osm_inform.h index 67d1c61..e2325da 100644 --- a/opensm/include/opensm/osm_inform.h +++ b/opensm/include/opensm/osm_inform.h @@ -94,7 +94,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_infr_t { +typedef struct osm_infr { cl_list_item_t list_item; osm_bind_handle_t h_bind; osm_sa_t *sa; diff --git a/opensm/include/opensm/osm_lid_mgr.h b/opensm/include/opensm/osm_lid_mgr.h index 7d6b84c..556eb48 100644 --- a/opensm/include/opensm/osm_lid_mgr.h +++ b/opensm/include/opensm/osm_lid_mgr.h @@ -90,7 +90,7 @@ struct osm_sm; * * SYNOPSIS */ -typedef struct _osm_lid_mgr { +typedef struct osm_lid_mgr { struct osm_sm *sm; osm_subn_t *p_subn; osm_db_t *p_db; diff --git a/opensm/include/opensm/osm_lin_fwd_tbl.h b/opensm/include/opensm/osm_lin_fwd_tbl.h index 6c64c09..98c6b4f 100644 --- a/opensm/include/opensm/osm_lin_fwd_tbl.h +++ b/opensm/include/opensm/osm_lin_fwd_tbl.h @@ -84,7 +84,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_lin_fwd_tbl { +typedef struct osm_lin_fwdbl { uint16_t size; uint8_t port_tbl[1]; } osm_lin_fwd_tbl_t; diff --git a/opensm/include/opensm/osm_log.h b/opensm/include/opensm/osm_log.h index f90aa9b..c6409be 100644 --- a/opensm/include/opensm/osm_log.h +++ b/opensm/include/opensm/osm_log.h @@ -110,7 +110,7 @@ typedef uint8_t osm_log_level_t; * * SYNOPSIS */ -typedef struct _osm_log { +typedef struct osm_log { osm_log_level_t level; cl_spinlock_t lock; unsigned long count; diff --git a/opensm/include/opensm/osm_mad_pool.h b/opensm/include/opensm/osm_mad_pool.h index 3f5df01..9810a8f 100644 --- a/opensm/include/opensm/osm_mad_pool.h +++ b/opensm/include/opensm/osm_mad_pool.h @@ -88,7 +88,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_mad_pool { +typedef struct osm_mad_pool { atomic32_t mads_out; } osm_mad_pool_t; /* diff --git a/opensm/include/opensm/osm_madw.h b/opensm/include/opensm/osm_madw.h index 955b500..70649a3 100644 --- a/opensm/include/opensm/osm_madw.h +++ b/opensm/include/opensm/osm_madw.h @@ -67,7 +67,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_bind_info { +typedef struct osm_bind_info { ib_net64_t port_guid; uint8_t mad_class; uint8_t class_version; @@ -135,7 +135,7 @@ typedef struct _osm_bind_info { * * SYNOPSIS */ -typedef struct _osm_ni_context { +typedef struct osm_ni_context { ib_net64_t node_guid; uint8_t port_num; ib_net64_t dup_node_guid; @@ -167,7 +167,7 @@ typedef struct _osm_ni_context { * * SYNOPSIS */ -typedef struct _osm_pi_context { +typedef struct osm_pi_context { ib_net64_t node_guid; ib_net64_t port_guid; boolean_t set_method; @@ -186,7 +186,7 @@ typedef struct _osm_pi_context { * * SYNOPSIS */ -typedef struct _osm_nd_context { +typedef struct osm_nd_context { ib_net64_t node_guid; } osm_nd_context_t; /*********/ @@ -200,7 +200,7 @@ typedef struct _osm_nd_context { * * SYNOPSIS */ -typedef struct _osm_si_context { +typedef struct osm_si_context { ib_net64_t node_guid; boolean_t set_method; boolean_t light_sweep; @@ -216,7 +216,7 @@ typedef struct _osm_si_context { * * SYNOPSIS */ -typedef struct _osm_lft_context { +typedef struct osm_lft_context { ib_net64_t node_guid; boolean_t set_method; } osm_lft_context_t; @@ -231,7 +231,7 @@ typedef struct _osm_lft_context { * * SYNOPSIS */ -typedef struct _osm_mft_context { +typedef struct osm_mft_context { ib_net64_t node_guid; boolean_t set_method; } osm_mft_context_t; @@ -246,7 +246,7 @@ typedef struct _osm_mft_context { * * SYNOPSIS */ -typedef struct _osm_smi_context { +typedef struct osm_smi_context { ib_net64_t port_guid; boolean_t set_method; } osm_smi_context_t; @@ -261,7 +261,7 @@ typedef struct _osm_smi_context { * * SYNOPSIS */ -typedef struct _osm_pkey_context { +typedef struct osm_pkey_context { ib_net64_t node_guid; ib_net64_t port_guid; boolean_t set_method; @@ -277,7 +277,7 @@ typedef struct _osm_pkey_context { * * SYNOPSIS */ -typedef struct _osm_slvl_context { +typedef struct osm_slvl_context { ib_net64_t node_guid; ib_net64_t port_guid; boolean_t set_method; @@ -293,7 +293,7 @@ typedef struct _osm_slvl_context { * * SYNOPSIS */ -typedef struct _osm_vla_context { +typedef struct osm_vla_context { ib_net64_t node_guid; ib_net64_t port_guid; boolean_t set_method; @@ -304,7 +304,7 @@ typedef struct _osm_vla_context { * DESCRIPTION * Context for Performance manager queries */ -typedef struct _osm_perfmgr_context { +typedef struct osm_perfmgr_context { uint64_t node_guid; uint16_t port; uint8_t mad_method; /* was this a get or a set */ @@ -324,7 +324,7 @@ typedef struct _osm_perfmgr_context { * * SYNOPSIS */ -typedef struct _osm_arbitrary_context { +typedef struct osm_arbitrary_context { void *context1; void *context2; } osm_arbitrary_context_t; @@ -366,7 +366,7 @@ typedef union _osm_madw_context { * * SYNOPSIS */ -typedef struct _osm_mad_addr { +typedef struct osm_mad_addr { ib_net16_t dest_lid; uint8_t path_bits; uint8_t static_rate; @@ -401,7 +401,7 @@ typedef struct _osm_mad_addr { * * SYNOPSIS */ -typedef struct _osm_madw { +typedef struct osm_madw { cl_list_item_t list_item; osm_bind_handle_t h_bind; osm_vend_wrap_t vend_wrap; diff --git a/opensm/include/opensm/osm_mcast_tbl.h b/opensm/include/opensm/osm_mcast_tbl.h index 3aa5aa9..15b95cf 100644 --- a/opensm/include/opensm/osm_mcast_tbl.h +++ b/opensm/include/opensm/osm_mcast_tbl.h @@ -67,7 +67,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_mcast_fwd_tbl { +typedef struct osm_mcast_fwdbl { uint8_t num_ports; uint8_t max_position; uint16_t max_block; diff --git a/opensm/include/opensm/osm_mcm_info.h b/opensm/include/opensm/osm_mcm_info.h index b40771c..dec607f 100644 --- a/opensm/include/opensm/osm_mcm_info.h +++ b/opensm/include/opensm/osm_mcm_info.h @@ -71,7 +71,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_mcm_info { +typedef struct osm_mcm_info { cl_list_item_t list_item; ib_net16_t mlid; } osm_mcm_info_t; diff --git a/opensm/include/opensm/osm_mcm_port.h b/opensm/include/opensm/osm_mcm_port.h index e8c560e..c2b18de 100644 --- a/opensm/include/opensm/osm_mcm_port.h +++ b/opensm/include/opensm/osm_mcm_port.h @@ -69,7 +69,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_mcm_port { +typedef struct osm_mcm_port { cl_map_item_t map_item; ib_gid_t port_gid; uint8_t scope_state; diff --git a/opensm/include/opensm/osm_mtree.h b/opensm/include/opensm/osm_mtree.h index e9d663c..7297442 100644 --- a/opensm/include/opensm/osm_mtree.h +++ b/opensm/include/opensm/osm_mtree.h @@ -97,12 +97,12 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_mtree_node { +typedef struct osm_mtree_node { cl_map_item_t map_item; osm_switch_t *p_sw; uint8_t max_children; - struct _osm_mtree_node *p_up; - struct _osm_mtree_node *child_array[1]; + struct osm_mtree_node *p_up; + struct osm_mtree_node *child_array[1]; } osm_mtree_node_t; /* * FIELDS diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index 57e5ef5..c109c91 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -151,7 +151,7 @@ typedef struct osm_mcast_mgr_ctxt { * * SYNOPSIS */ -typedef struct _osm_mgrp { +typedef struct osm_mgrp { cl_map_item_t map_item; ib_net16_t mlid; osm_mtree_node_t *p_root; diff --git a/opensm/include/opensm/osm_node.h b/opensm/include/opensm/osm_node.h index a679f65..0c06ee8 100644 --- a/opensm/include/opensm/osm_node.h +++ b/opensm/include/opensm/osm_node.h @@ -60,7 +60,7 @@ BEGIN_C_DECLS -struct _osm_switch; +struct osm_switch; /****h* OpenSM/Node * NAME @@ -94,9 +94,9 @@ struct _osm_switch; * * SYNOPSIS */ -typedef struct _osm_node { +typedef struct osm_node { cl_map_item_t map_item; - struct _osm_switch *sw; + struct osm_switch *sw; ib_node_info_t node_info; ib_node_desc_t node_desc; uint32_t discovery_count; diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index 92e4238..fd94b03 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -161,7 +161,7 @@ struct osm_routing_engine { * * SYNOPSIS */ -typedef struct _osm_opensm_t { +typedef struct osm_opensm { osm_subn_t subn; osm_sm_t sm; osm_sa_t sa; diff --git a/opensm/include/opensm/osm_partition.h b/opensm/include/opensm/osm_partition.h index 6ddb8ae..38acdc9 100644 --- a/opensm/include/opensm/osm_partition.h +++ b/opensm/include/opensm/osm_partition.h @@ -89,7 +89,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_prtn { +typedef struct osm_prtn { cl_map_item_t map_item; ib_net16_t pkey; ib_net16_t mlid; diff --git a/opensm/include/opensm/osm_path.h b/opensm/include/opensm/osm_path.h index 7b36d68..0b765cb 100644 --- a/opensm/include/opensm/osm_path.h +++ b/opensm/include/opensm/osm_path.h @@ -78,7 +78,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_dr_path { +typedef struct osm_dr_path { osm_bind_handle_t h_bind; uint8_t hop_count; uint8_t path[IB_SUBNET_PATH_HOPS_MAX]; diff --git a/opensm/include/opensm/osm_perfmgr.h b/opensm/include/opensm/osm_perfmgr.h index 9152f5f..e2eafd7 100644 --- a/opensm/include/opensm/osm_perfmgr.h +++ b/opensm/include/opensm/osm_perfmgr.h @@ -109,7 +109,7 @@ typedef struct _monitored_node { * This object should be treated as opaque and should * be manipulated only through the provided functions. */ -typedef struct _osm_perfmgr { +typedef struct osm_perfmgr { cl_event_t sig_sweep; cl_timer_t sweep_timer; osm_subn_t *subn; diff --git a/opensm/include/opensm/osm_pkey.h b/opensm/include/opensm/osm_pkey.h index 54bfcd9..c10a55e 100644 --- a/opensm/include/opensm/osm_pkey.h +++ b/opensm/include/opensm/osm_pkey.h @@ -55,11 +55,11 @@ BEGIN_C_DECLS /* Forward references. */ -struct _osm_physp; -struct _osm_port; -struct _osm_subn; -struct _osm_node; -struct _osm_physp; +struct osm_physp; +struct osm_port; +struct osm_subn; +struct osm_node; +struct osm_physp; /* * Abstract: @@ -80,7 +80,7 @@ struct _osm_physp; * * SYNOPSIS */ -typedef struct _osm_pkey_tbl { +typedef struct osm_pkeybl { cl_ptr_vector_t blocks; cl_ptr_vector_t new_blocks; cl_map_t keys; @@ -132,7 +132,7 @@ typedef struct _osm_pkey_tbl { * * SYNOPSIS */ -typedef struct _osm_pending_pkey { +typedef struct osm_pending_pkey { cl_list_item_t list_item; uint16_t pkey; uint16_t block; @@ -439,8 +439,8 @@ osm_pkey_tbl_set(IN osm_pkey_tbl_t * p_pkey_tbl, * * SYNOPSIS */ -boolean_t osm_physp_share_this_pkey(IN const struct _osm_physp *const p_physp1, - IN const struct _osm_physp *const p_physp2, +boolean_t osm_physp_share_this_pkey(IN const struct osm_physp *const p_physp1, + IN const struct osm_physp *const p_physp2, IN const ib_net16_t pkey); /* * PARAMETERS @@ -471,9 +471,9 @@ boolean_t osm_physp_share_this_pkey(IN const struct _osm_physp *const p_physp1, * * SYNOPSIS */ -ib_net16_t osm_physp_find_common_pkey(IN const struct _osm_physp *const +ib_net16_t osm_physp_find_common_pkey(IN const struct osm_physp *const p_physp1, - IN const struct _osm_physp *const + IN const struct osm_physp *const p_physp2); /* * PARAMETERS @@ -517,8 +517,8 @@ ib_net16_t osm_physp_find_common_pkey(IN const struct _osm_physp *const * SYNOPSIS */ boolean_t osm_physp_share_pkey(IN osm_log_t * p_log, - IN const struct _osm_physp *const p_physp_1, - IN const struct _osm_physp *const p_physp_2); + IN const struct osm_physp *const p_physp_1, + IN const struct osm_physp *const p_physp_2); /* * PARAMETERS @@ -564,8 +564,8 @@ boolean_t osm_physp_share_pkey(IN osm_log_t * p_log, * SYNOPSIS */ boolean_t osm_port_share_pkey(IN osm_log_t * p_log, - IN const struct _osm_port *const p_port_1, - IN const struct _osm_port *const p_port_2); + IN const struct osm_port *const p_port_1, + IN const struct osm_port *const p_port_2); /* * PARAMETERS @@ -611,7 +611,7 @@ boolean_t osm_port_share_pkey(IN osm_log_t * p_log, * SYNOPSIS */ boolean_t osm_lid_share_pkey(IN osm_log_t * p_log, - IN const struct _osm_subn *const p_subn, + IN const struct osm_subn *const p_subn, IN const ib_net16_t lid1, IN const uint8_t port_num1, IN const ib_net16_t lid2, @@ -671,7 +671,7 @@ boolean_t osm_lid_share_pkey(IN osm_log_t * p_log, */ boolean_t osm_physp_has_pkey(IN osm_log_t * p_log, IN const ib_net16_t pkey, - IN const struct _osm_physp *const p_physp); + IN const struct osm_physp *const p_physp); /* * PARAMETERS diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index c1cbd39..514f6e7 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -65,8 +65,8 @@ BEGIN_C_DECLS /* Forward references. */ -struct _osm_port; -struct _osm_node; +struct osm_port; +struct osm_node; /****h* OpenSM/Physical Port * NAME @@ -103,12 +103,12 @@ struct _osm_node; * * SYNOPSIS */ -typedef struct _osm_physp { +typedef struct osm_physp { ib_port_info_t port_info; ib_net64_t port_guid; uint8_t port_num; - struct _osm_node *p_node; - struct _osm_physp *p_remote_physp; + struct osm_node *p_node; + struct osm_physp *p_remote_physp; boolean_t healthy; uint8_t vl_high_limit; unsigned need_update; @@ -209,7 +209,7 @@ void osm_physp_init(IN osm_physp_t * const p_physp, IN const ib_net64_t port_guid, IN const uint8_t port_num, - IN const struct _osm_node *const p_node, + IN const struct osm_node *const p_node, IN const osm_bind_handle_t h_bind, IN const uint8_t hop_count, IN const uint8_t * const p_initial_path); @@ -963,12 +963,12 @@ osm_physp_get_port_num(IN const osm_physp_t * const p_physp) * * SYNOPSIS */ -static inline struct _osm_node *osm_physp_get_node_ptr(IN const osm_physp_t * +static inline struct osm_node *osm_physp_get_node_ptr(IN const osm_physp_t * const p_physp) { CL_ASSERT(p_physp); CL_ASSERT(osm_physp_is_valid(p_physp)); - return ((struct _osm_node *)p_physp->p_node); + return ((struct osm_node *)p_physp->p_node); } /* @@ -1141,9 +1141,9 @@ static inline osm_dr_path_t *osm_physp_get_dr_path_ptr(IN const osm_physp_t * * * SYNOPSIS */ -typedef struct _osm_port { +typedef struct osm_port { cl_map_item_t map_item; - struct _osm_node *p_node; + struct osm_node *p_node; ib_net64_t guid; uint32_t discovery_count; unsigned is_new; @@ -1214,7 +1214,7 @@ void osm_port_delete(IN OUT osm_port_t ** const pp_port); * SYNOPSIS */ osm_port_t *osm_port_new(IN const ib_node_info_t * p_ni, - IN struct _osm_node *const p_parent_node); + IN struct osm_node *const p_parent_node); /* * PARAMETERS * p_ni diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index 9997d0b..9c80c03 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -89,7 +89,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_port_profile { +typedef struct osm_port_profile { uint32_t num_paths; } osm_port_profile_t; /* diff --git a/opensm/include/opensm/osm_qos_policy.h b/opensm/include/opensm/osm_qos_policy.h index d57a40a..1924d9d 100644 --- a/opensm/include/opensm/osm_qos_policy.h +++ b/opensm/include/opensm/osm_qos_policy.h @@ -66,12 +66,12 @@ /***************************************************/ -typedef struct _osm_qos_port_t { +typedef struct osm_qos_port { cl_map_item_t map_item; osm_physp_t * p_physp; } osm_qos_port_t; -typedef struct _osm_qos_port_group_t { +typedef struct osm_qos_port_group { char *name; /* single string (this port group name) */ char *use; /* single string (description) */ uint8_t node_types; /* node types bitmask */ @@ -80,7 +80,7 @@ typedef struct _osm_qos_port_group_t { /***************************************************/ -typedef struct _osm_qos_vlarb_scope_t { +typedef struct osm_qos_vlarb_scope { cl_list_t group_list; /* list of group names (strings) */ cl_list_t across_list; /* list of 'across' group names (strings) */ cl_list_t vlarb_high_list; /* list of num pairs (n:m,...), 32-bit values */ @@ -91,7 +91,7 @@ typedef struct _osm_qos_vlarb_scope_t { /***************************************************/ -typedef struct _osm_qos_sl2vl_scope_t { +typedef struct osm_qos_sl2vl_scope { cl_list_t group_list; /* list of strings (port group names) */ boolean_t from[OSM_QOS_POLICY_MAX_PORTS_ON_SWITCH]; boolean_t to[OSM_QOS_POLICY_MAX_PORTS_ON_SWITCH]; @@ -103,7 +103,7 @@ typedef struct _osm_qos_sl2vl_scope_t { /***************************************************/ -typedef struct _osm_qos_level_t { +typedef struct osm_qos_level { char *use; char *name; uint8_t sl; @@ -123,7 +123,7 @@ typedef struct _osm_qos_level_t { /***************************************************/ -typedef struct _osm_qos_match_rule_t { +typedef struct osm_qos_match_rule { char *use; cl_list_t source_list; /* list of strings */ cl_list_t source_group_list; /* list of pointers to relevant port-group */ @@ -141,7 +141,7 @@ typedef struct _osm_qos_match_rule_t { /***************************************************/ -typedef struct _osm_qos_policy_t { +typedef struct osm_qos_policy { cl_list_t port_groups; /* list of osm_qos_port_group_t */ cl_list_t sl2vl_tables; /* list of osm_qos_sl2vl_scope_t */ cl_list_t vlarb_tables; /* list of osm_qos_vlarb_scope_t */ diff --git a/opensm/include/opensm/osm_rand_fwd_tbl.h b/opensm/include/opensm/osm_rand_fwd_tbl.h index c7c4586..8e9ac90 100644 --- a/opensm/include/opensm/osm_rand_fwd_tbl.h +++ b/opensm/include/opensm/osm_rand_fwd_tbl.h @@ -87,7 +87,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_rand_fwd_tbl { +typedef struct osm_rand_fwdbl { /* PLACE HOLDER STRUCTURE ONLY!! */ uint32_t size; } osm_rand_fwd_tbl_t; diff --git a/opensm/include/opensm/osm_remote_sm.h b/opensm/include/opensm/osm_remote_sm.h index 1bdc6cb..4cc980b 100644 --- a/opensm/include/opensm/osm_remote_sm.h +++ b/opensm/include/opensm/osm_remote_sm.h @@ -85,7 +85,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_remote_sm { +typedef struct osm_remote_sm { cl_map_item_t map_item; const osm_port_t *p_port; ib_sm_info_t smi; diff --git a/opensm/include/opensm/osm_router.h b/opensm/include/opensm/osm_router.h index 7ba6b3d..8cabdf8 100644 --- a/opensm/include/opensm/osm_router.h +++ b/opensm/include/opensm/osm_router.h @@ -92,7 +92,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_router { +typedef struct osm_router { cl_map_item_t map_item; osm_port_t *p_port; } osm_router_t; diff --git a/opensm/include/opensm/osm_sa.h b/opensm/include/opensm/osm_sa.h index a797a9d..72d4d86 100644 --- a/opensm/include/opensm/osm_sa.h +++ b/opensm/include/opensm/osm_sa.h @@ -113,7 +113,7 @@ typedef enum _osm_sa_state { * * SYNOPSIS */ -typedef struct _osm_sa { +typedef struct osm_sa { osm_sa_state_t state; osm_sm_t *sm; osm_subn_t *p_subn; @@ -418,7 +418,7 @@ void osm_sa_respond(osm_sa_t *sa, osm_madw_t *madw, size_t attr_size, * SA object *********/ -struct _osm_opensm_t; +struct osm_opensm; /****f* OpenSM: SA/osm_sa_db_file_dump * NAME * osm_sa_db_file_dump @@ -428,7 +428,7 @@ struct _osm_opensm_t; * * SYNOPSIS */ -int osm_sa_db_file_dump(struct _osm_opensm_t *p_osm); +int osm_sa_db_file_dump(struct osm_opensm *p_osm); /* * PARAMETERS * p_osm @@ -448,7 +448,7 @@ int osm_sa_db_file_dump(struct _osm_opensm_t *p_osm); * * SYNOPSIS */ -int osm_sa_db_file_load(struct _osm_opensm_t *p_osm); +int osm_sa_db_file_load(struct osm_opensm *p_osm); /* * PARAMETERS * p_osm diff --git a/opensm/include/opensm/osm_sa_mad_ctrl.h b/opensm/include/opensm/osm_sa_mad_ctrl.h index 1fde803..581ea12 100644 --- a/opensm/include/opensm/osm_sa_mad_ctrl.h +++ b/opensm/include/opensm/osm_sa_mad_ctrl.h @@ -78,7 +78,7 @@ BEGIN_C_DECLS * *********/ -struct _osm_sa; +struct osm_sa; /****s* OpenSM: SA MAD Controller/osm_sa_mad_ctrl_t * NAME * osm_sa_mad_ctrl_t @@ -91,8 +91,8 @@ struct _osm_sa; * * SYNOPSIS */ -typedef struct _osm_sa_mad_ctrl { - struct _osm_sa *sa; +typedef struct osm_sa_mad_ctrl { + struct osm_sa *sa; osm_log_t *p_log; osm_mad_pool_t *p_mad_pool; osm_vendor_t *p_vendor; @@ -205,7 +205,7 @@ void osm_sa_mad_ctrl_destroy(IN osm_sa_mad_ctrl_t * const p_ctrl); * SYNOPSIS */ ib_api_status_t osm_sa_mad_ctrl_init(IN osm_sa_mad_ctrl_t * const p_ctrl, - IN struct _osm_sa * sa, + IN struct osm_sa * sa, IN osm_mad_pool_t * const p_mad_pool, IN osm_vendor_t * const p_vendor, IN osm_subn_t * const p_subn, diff --git a/opensm/include/opensm/osm_service.h b/opensm/include/opensm/osm_service.h index d1dc506..1d97a68 100644 --- a/opensm/include/opensm/osm_service.h +++ b/opensm/include/opensm/osm_service.h @@ -88,7 +88,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_svcr_t { +typedef struct osm_svcr { cl_list_item_t list_item; ib_service_record_t service_record; uint32_t modified_time; diff --git a/opensm/include/opensm/osm_sm_mad_ctrl.h b/opensm/include/opensm/osm_sm_mad_ctrl.h index f003ed8..1c41168 100644 --- a/opensm/include/opensm/osm_sm_mad_ctrl.h +++ b/opensm/include/opensm/osm_sm_mad_ctrl.h @@ -92,7 +92,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_sm_mad_ctrl { +typedef struct osm_sm_mad_ctrl { osm_log_t *p_log; osm_subn_t *p_subn; osm_mad_pool_t *p_mad_pool; diff --git a/opensm/include/opensm/osm_stats.h b/opensm/include/opensm/osm_stats.h index ad50d5a..575e9e4 100644 --- a/opensm/include/opensm/osm_stats.h +++ b/opensm/include/opensm/osm_stats.h @@ -81,7 +81,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_stats { +typedef struct osm_stats { atomic32_t qp0_mads_outstanding; atomic32_t qp0_mads_outstanding_on_wire; atomic32_t qp0_mads_rcvd; diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 338b274..1b97508 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -63,8 +63,8 @@ BEGIN_C_DECLS #define OSM_SUBNET_VECTOR_MIN_SIZE 0 #define OSM_SUBNET_VECTOR_GROW_SIZE 1 #define OSM_SUBNET_VECTOR_CAPACITY 256 -struct _osm_opensm_t; -struct _osm_qos_policy_t; +struct osm_opensm; +struct osm_qos_policy; /****h* OpenSM/Subnet * NAME @@ -161,7 +161,7 @@ typedef void * * SYNOPSIS */ -typedef struct _osm_qos_options_t { +typedef struct osm_qos_options { unsigned max_vls; unsigned high_limit; char *vlarb_high; @@ -199,7 +199,7 @@ typedef struct _osm_qos_options_t { * * SYNOPSIS */ -typedef struct _osm_subn_opt { +typedef struct osm_subn_opt { char *config_file; ib_net64_t guid; ib_net64_t m_key; @@ -521,8 +521,8 @@ typedef struct _osm_subn_opt { * * SYNOPSIS */ -typedef struct _osm_subn { - struct _osm_opensm_t *p_osm; +typedef struct osm_subn { + struct osm_opensm *p_osm; cl_qmap_t sw_guid_tbl; cl_qmap_t node_guid_tbl; cl_qmap_t port_guid_tbl; @@ -540,7 +540,7 @@ typedef struct _osm_subn { ib_net64_t sm_port_guid; uint8_t sm_state; osm_subn_opt_t opt; - struct _osm_qos_policy_t *p_qos_policy; + struct osm_qos_policy *p_qos_policy; uint16_t max_unicast_lid_ho; uint16_t max_multicast_lid_ho; uint8_t min_ca_mtu; @@ -749,7 +749,7 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn); */ ib_api_status_t osm_subn_init(IN osm_subn_t * const p_subn, - IN struct _osm_opensm_t *const p_osm, + IN struct osm_opensm *const p_osm, IN const osm_subn_opt_t * const p_opt); /* * PARAMETERS @@ -772,11 +772,11 @@ osm_subn_init(IN osm_subn_t * const p_subn, /* Forward references. */ -struct _osm_mad_addr; -struct _osm_log; -struct _osm_switch; -struct _osm_physp; -struct _osm_port; +struct osm_mad_addr; +struct osm_log; +struct osm_switch; +struct osm_physp; +struct osm_port; /****f* OpenSM: Helper/osm_get_gid_by_mad_addr * NAME @@ -791,9 +791,9 @@ struct _osm_port; * SYNOPSIS */ ib_api_status_t -osm_get_gid_by_mad_addr(IN struct _osm_log *p_log, +osm_get_gid_by_mad_addr(IN struct osm_log *p_log, IN const osm_subn_t * p_subn, - IN const struct _osm_mad_addr *p_mad_addr, + IN const struct osm_mad_addr *p_mad_addr, OUT ib_gid_t * p_gid); /* * PARAMETERS @@ -829,9 +829,9 @@ osm_get_gid_by_mad_addr(IN struct _osm_log *p_log, * * SYNOPSIS */ -struct _osm_physp *osm_get_physp_by_mad_addr(IN struct _osm_log *p_log, +struct osm_physp *osm_get_physp_by_mad_addr(IN struct osm_log *p_log, IN const osm_subn_t * p_subn, - IN struct _osm_mad_addr + IN struct osm_mad_addr *p_mad_addr); /* * PARAMETERS @@ -864,9 +864,9 @@ struct _osm_physp *osm_get_physp_by_mad_addr(IN struct _osm_log *p_log, * * SYNOPSIS */ -struct _osm_port *osm_get_port_by_mad_addr(IN struct _osm_log *p_log, +struct osm_port *osm_get_port_by_mad_addr(IN struct osm_log *p_log, IN const osm_subn_t * p_subn, - IN struct _osm_mad_addr *p_mad_addr); + IN struct osm_mad_addr *p_mad_addr); /* * PARAMETERS * p_log @@ -897,7 +897,7 @@ struct _osm_port *osm_get_port_by_mad_addr(IN struct _osm_log *p_log, * * SYNOPSIS */ -struct _osm_switch *osm_get_switch_by_guid(IN const osm_subn_t * p_subn, +struct osm_switch *osm_get_switch_by_guid(IN const osm_subn_t * p_subn, IN uint64_t guid); /* * PARAMETERS @@ -926,7 +926,7 @@ struct _osm_switch *osm_get_switch_by_guid(IN const osm_subn_t * p_subn, * * SYNOPSIS */ -struct _osm_node *osm_get_node_by_guid(IN osm_subn_t const *p_subn, +struct osm_node *osm_get_node_by_guid(IN osm_subn_t const *p_subn, IN uint64_t guid); /* * PARAMETERS @@ -955,7 +955,7 @@ struct _osm_node *osm_get_node_by_guid(IN osm_subn_t const *p_subn, * * SYNOPSIS */ -struct _osm_port *osm_get_port_by_guid(IN osm_subn_t const *p_subn, +struct osm_port *osm_get_port_by_guid(IN osm_subn_t const *p_subn, IN ib_net64_t guid); /* * PARAMETERS @@ -985,9 +985,9 @@ struct _osm_port *osm_get_port_by_guid(IN osm_subn_t const *p_subn, * * SYNOPSIS */ -struct _osm_physp *osm_get_physp_by_mad_addr(IN struct _osm_log *p_log, +struct osm_physp *osm_get_physp_by_mad_addr(IN struct osm_log *p_log, IN const osm_subn_t * p_subn, - IN struct _osm_mad_addr + IN struct osm_mad_addr *p_mad_addr); /* * PARAMETERS diff --git a/opensm/include/opensm/osm_sweep_fail_ctrl.h b/opensm/include/opensm/osm_sweep_fail_ctrl.h index fccc5fa..12832c0 100644 --- a/opensm/include/opensm/osm_sweep_fail_ctrl.h +++ b/opensm/include/opensm/osm_sweep_fail_ctrl.h @@ -89,7 +89,7 @@ struct osm_sm; * * SYNOPSIS */ -typedef struct _osm_sweep_fail_ctrl { +typedef struct osm_sweep_fail_ctrl { struct osm_sm *sm; cl_disp_reg_handle_t h_disp; } osm_sweep_fail_ctrl_t; diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index c1521a6..90119b9 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -92,7 +92,7 @@ BEGIN_C_DECLS * * SYNOPSIS */ -typedef struct _osm_switch { +typedef struct osm_switch { cl_map_item_t map_item; osm_node_t *p_node; ib_switch_info_t switch_info; diff --git a/opensm/include/opensm/osm_ucast_mgr.h b/opensm/include/opensm/osm_ucast_mgr.h index 478078d..9be059c 100644 --- a/opensm/include/opensm/osm_ucast_mgr.h +++ b/opensm/include/opensm/osm_ucast_mgr.h @@ -90,7 +90,7 @@ struct osm_sm; * * SYNOPSIS */ -typedef struct _osm_ucast_mgr { +typedef struct osm_ucast_mgr { struct osm_sm *sm; osm_subn_t *p_subn; osm_log_t *p_log; diff --git a/opensm/include/opensm/osm_vl15intf.h b/opensm/include/opensm/osm_vl15intf.h index 7453de8..b2a225f 100644 --- a/opensm/include/opensm/osm_vl15intf.h +++ b/opensm/include/opensm/osm_vl15intf.h @@ -113,7 +113,7 @@ typedef enum _osm_vl15_state { * * SYNOPSIS */ -typedef struct _osm_vl15 { +typedef struct osm_vl15 { osm_thread_state_t thread_state; osm_vl15_state_t state; uint32_t max_wire_smps; @@ -206,7 +206,7 @@ void osm_vl15_construct(IN osm_vl15_t * const p_vl15); * SYNOPSIS */ void -osm_vl15_destroy(IN osm_vl15_t * const p_vl15, IN struct _osm_mad_pool *p_pool); +osm_vl15_destroy(IN osm_vl15_t * const p_vl15, IN struct osm_mad_pool *p_pool); /* * PARAMETERS * p_vl15 diff --git a/opensm/opensm/osm_db_files.c b/opensm/opensm/osm_db_files.c index 1d1aaab..3e25dab 100644 --- a/opensm/opensm/osm_db_files.c +++ b/opensm/opensm/osm_db_files.c @@ -83,7 +83,7 @@ * * SYNOPSIS */ -typedef struct _osm_db_domain_imp { +typedef struct osm_db_domain_imp { char *file_name; st_table *p_hash; cl_spinlock_t lock; @@ -104,7 +104,7 @@ typedef struct _osm_db_domain_imp { * * SYNOPSIS */ -typedef struct _osm_db_imp { +typedef struct osm_db_imp { char *db_dir_name; } osm_db_imp_t; /* diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index 7782f02..1a4f18f 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -51,7 +51,7 @@ #include #include -typedef struct _osm_infr_match_ctxt { +typedef struct osm_infr_match_ctxt { cl_list_t *p_remove_infr_list; ib_mad_notice_attr_t *p_ntc; } osm_infr_match_ctxt_t; diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index 1ad9ef0..e3ebf11 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -98,7 +98,7 @@ /********************************************************************** lid range item of qlist **********************************************************************/ -typedef struct _osm_lid_mgr_range { +typedef struct osm_lid_mgr_range { cl_list_item_t item; uint16_t min_lid; uint16_t max_lid; diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index 75afbbb..5d554b3 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -57,7 +57,7 @@ /********************************************************************** **********************************************************************/ -typedef struct _osm_mcast_work_obj { +typedef struct osm_mcast_work_obj { cl_list_item_t list_item; osm_port_t *p_port; } osm_mcast_work_obj_t; diff --git a/opensm/opensm/osm_port.c b/opensm/opensm/osm_port.c index 18f1ef2..b213e89 100644 --- a/opensm/opensm/osm_port.c +++ b/opensm/opensm/osm_port.c @@ -92,7 +92,7 @@ void osm_physp_init(IN osm_physp_t * const p_physp, IN const ib_net64_t port_guid, IN const uint8_t port_num, - IN const struct _osm_node *const p_node, + IN const struct osm_node *const p_node, IN const osm_bind_handle_t h_bind, IN const uint8_t hop_count, IN const uint8_t * const p_initial_path) @@ -107,7 +107,7 @@ osm_physp_init(IN osm_physp_t * const p_physp, p_physp->port_num = port_num; p_physp->healthy = TRUE; p_physp->need_update = 2; - p_physp->p_node = (struct _osm_node *)p_node; + p_physp->p_node = (struct osm_node *)p_node; osm_dr_path_init(&p_physp->dr_path, h_bind, hop_count, p_initial_path); @@ -159,7 +159,7 @@ osm_port_init(IN osm_port_t * const p_port, memset(p_port, 0, sizeof(*p_port)); cl_qlist_init(&p_port->mcm_list); - p_port->p_node = (struct _osm_node *)p_parent_node; + p_port->p_node = (struct osm_node *)p_parent_node; port_guid = p_ni->port_guid; p_port->guid = port_guid; port_num = p_ni->node_type == IB_NODE_TYPE_SWITCH ? diff --git a/opensm/opensm/osm_sa_guidinfo_record.c b/opensm/opensm/osm_sa_guidinfo_record.c index f38c575..7760f42 100644 --- a/opensm/opensm/osm_sa_guidinfo_record.c +++ b/opensm/opensm/osm_sa_guidinfo_record.c @@ -57,12 +57,12 @@ #include #include -typedef struct _osm_gir_item { +typedef struct osm_gir_item { cl_list_item_t list_item; ib_guidinfo_record_t rec; } osm_gir_item_t; -typedef struct _osm_gir_search_ctxt { +typedef struct osm_gir_search_ctxt { const ib_guidinfo_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_informinfo.c b/opensm/opensm/osm_sa_informinfo.c index 2e0c504..798f5db 100644 --- a/opensm/opensm/osm_sa_informinfo.c +++ b/opensm/opensm/osm_sa_informinfo.c @@ -59,12 +59,12 @@ #include #include -typedef struct _osm_iir_item { +typedef struct osm_iir_item { cl_list_item_t list_item; ib_inform_info_record_t rec; } osm_iir_item_t; -typedef struct _osm_iir_search_ctxt { +typedef struct osm_iir_search_ctxt { const ib_inform_info_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_lft_record.c b/opensm/opensm/osm_sa_lft_record.c index 1699824..f4b9b6e 100644 --- a/opensm/opensm/osm_sa_lft_record.c +++ b/opensm/opensm/osm_sa_lft_record.c @@ -54,12 +54,12 @@ #include #include -typedef struct _osm_lftr_item { +typedef struct osm_lftr_item { cl_list_item_t list_item; ib_lft_record_t rec; } osm_lftr_item_t; -typedef struct _osm_lftr_search_ctxt { +typedef struct osm_lftr_search_ctxt { const ib_lft_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_link_record.c b/opensm/opensm/osm_sa_link_record.c index 6cc1836..0b761ba 100644 --- a/opensm/opensm/osm_sa_link_record.c +++ b/opensm/opensm/osm_sa_link_record.c @@ -55,7 +55,7 @@ #include #include -typedef struct _osm_lr_item { +typedef struct osm_lr_item { cl_list_item_t list_item; ib_link_record_t link_rec; } osm_lr_item_t; diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 1a9bc54..5db2a4d 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -75,7 +75,7 @@ IB_MCR_COMPMASK_FLOW | \ IB_MCR_COMPMASK_SL) -typedef struct _osm_mcmr_item { +typedef struct osm_mcmr_item { cl_list_item_t list_item; ib_member_rec_t rec; } osm_mcmr_item_t; diff --git a/opensm/opensm/osm_sa_mft_record.c b/opensm/opensm/osm_sa_mft_record.c index 24b4c3b..57350a6 100644 --- a/opensm/opensm/osm_sa_mft_record.c +++ b/opensm/opensm/osm_sa_mft_record.c @@ -54,12 +54,12 @@ #include #include -typedef struct _osm_mftr_item { +typedef struct osm_mftr_item { cl_list_item_t list_item; ib_mft_record_t rec; } osm_mftr_item_t; -typedef struct _osm_mftr_search_ctxt { +typedef struct osm_mftr_search_ctxt { const ib_mft_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_multipath_record.c b/opensm/opensm/osm_sa_multipath_record.c index f8da64d..bad1962 100644 --- a/opensm/opensm/osm_sa_multipath_record.c +++ b/opensm/opensm/osm_sa_multipath_record.c @@ -63,7 +63,7 @@ #define OSM_SA_MPR_MAX_NUM_PATH 127 -typedef struct _osm_mpr_item { +typedef struct osm_mpr_item { cl_list_item_t list_item; ib_path_rec_t path_rec; const osm_port_t *p_src_port; @@ -71,7 +71,7 @@ typedef struct _osm_mpr_item { int hops; } osm_mpr_item_t; -typedef struct _osm_path_parms { +typedef struct osm_path_parms { ib_net16_t pkey; uint8_t mtu; uint8_t rate; diff --git a/opensm/opensm/osm_sa_node_record.c b/opensm/opensm/osm_sa_node_record.c index a43bba7..91b748b 100644 --- a/opensm/opensm/osm_sa_node_record.c +++ b/opensm/opensm/osm_sa_node_record.c @@ -54,12 +54,12 @@ #include #include -typedef struct _osm_nr_item { +typedef struct osm_nr_item { cl_list_item_t list_item; ib_node_record_t rec; } osm_nr_item_t; -typedef struct _osm_nr_search_ctxt { +typedef struct osm_nr_search_ctxt { const ib_node_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index c468ba2..2a9155c 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -69,12 +69,12 @@ extern uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, const osm_port_t * p_src_port, const osm_port_t * p_dst_port); -typedef struct _osm_pr_item { +typedef struct osm_pr_item { cl_list_item_t list_item; ib_path_rec_t path_rec; } osm_pr_item_t; -typedef struct _osm_path_parms { +typedef struct osm_path_parms { ib_net16_t pkey; uint8_t mtu; uint8_t rate; diff --git a/opensm/opensm/osm_sa_pkey_record.c b/opensm/opensm/osm_sa_pkey_record.c index 4d19ed4..d953369 100644 --- a/opensm/opensm/osm_sa_pkey_record.c +++ b/opensm/opensm/osm_sa_pkey_record.c @@ -50,12 +50,12 @@ #include #include -typedef struct _osm_pkey_item { +typedef struct osm_pkey_item { cl_list_item_t list_item; ib_pkey_table_record_t rec; } osm_pkey_item_t; -typedef struct _osm_pkey_search_ctxt { +typedef struct osm_pkey_search_ctxt { const ib_pkey_table_record_t *p_rcvd_rec; ib_net64_t comp_mask; uint16_t block_num; diff --git a/opensm/opensm/osm_sa_portinfo_record.c b/opensm/opensm/osm_sa_portinfo_record.c index 88d8c23..3f3163d 100644 --- a/opensm/opensm/osm_sa_portinfo_record.c +++ b/opensm/opensm/osm_sa_portinfo_record.c @@ -58,12 +58,12 @@ #include #include -typedef struct _osm_pir_item { +typedef struct osm_pir_item { cl_list_item_t list_item; ib_portinfo_record_t rec; } osm_pir_item_t; -typedef struct _osm_pir_search_ctxt { +typedef struct osm_pir_search_ctxt { const ib_portinfo_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_service_record.c b/opensm/opensm/osm_sa_service_record.c index d68ad22..c9f4ee0 100644 --- a/opensm/opensm/osm_sa_service_record.c +++ b/opensm/opensm/osm_sa_service_record.c @@ -59,7 +59,7 @@ #include #include -typedef struct _osm_sr_item { +typedef struct osm_sr_item { cl_list_item_t list_item; ib_service_record_t service_rec; } osm_sr_item_t; @@ -71,7 +71,7 @@ typedef struct osm_sr_match_item { osm_sa_t *sa; } osm_sr_match_item_t; -typedef struct _osm_sr_search_ctxt { +typedef struct osm_sr_search_ctxt { osm_sr_match_item_t *p_sr_item; const osm_physp_t *p_req_physp; } osm_sr_search_ctxt_t; diff --git a/opensm/opensm/osm_sa_slvl_record.c b/opensm/opensm/osm_sa_slvl_record.c index 0248594..f3b53dc 100644 --- a/opensm/opensm/osm_sa_slvl_record.c +++ b/opensm/opensm/osm_sa_slvl_record.c @@ -57,12 +57,12 @@ #include #include -typedef struct _osm_slvl_item { +typedef struct osm_slvl_item { cl_list_item_t list_item; ib_slvl_table_record_t rec; } osm_slvl_item_t; -typedef struct _osm_slvl_search_ctxt { +typedef struct osm_slvl_search_ctxt { const ib_slvl_table_record_t *p_rcvd_rec; ib_net64_t comp_mask; uint8_t in_port_num; diff --git a/opensm/opensm/osm_sa_sminfo_record.c b/opensm/opensm/osm_sa_sminfo_record.c index 1860e55..6eddc59 100644 --- a/opensm/opensm/osm_sa_sminfo_record.c +++ b/opensm/opensm/osm_sa_sminfo_record.c @@ -63,12 +63,12 @@ #include #include -typedef struct _osm_smir_item { +typedef struct osm_smir_item { cl_list_item_t list_item; ib_sminfo_record_t rec; } osm_smir_item_t; -typedef struct _osm_smir_search_ctxt { +typedef struct osm_smir_search_ctxt { const ib_sminfo_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_sw_info_record.c b/opensm/opensm/osm_sa_sw_info_record.c index 9003a6c..a05c814 100644 --- a/opensm/opensm/osm_sa_sw_info_record.c +++ b/opensm/opensm/osm_sa_sw_info_record.c @@ -54,12 +54,12 @@ #include #include -typedef struct _osm_sir_item { +typedef struct osm_sir_item { cl_list_item_t list_item; ib_switch_info_record_t rec; } osm_sir_item_t; -typedef struct _osm_sir_search_ctxt { +typedef struct osm_sir_search_ctxt { const ib_switch_info_record_t *p_rcvd_rec; ib_net64_t comp_mask; cl_qlist_t *p_list; diff --git a/opensm/opensm/osm_sa_vlarb_record.c b/opensm/opensm/osm_sa_vlarb_record.c index 6308a70..6b97a2e 100644 --- a/opensm/opensm/osm_sa_vlarb_record.c +++ b/opensm/opensm/osm_sa_vlarb_record.c @@ -57,12 +57,12 @@ #include #include -typedef struct _osm_vl_arb_item { +typedef struct osm_vl_arb_item { cl_list_item_t list_item; ib_vl_arb_table_record_t rec; } osm_vl_arb_item_t; -typedef struct _osm_vl_arb_search_ctxt { +typedef struct osm_vl_arb_search_ctxt { const ib_vl_arb_table_record_t *p_rcvd_rec; ib_net64_t comp_mask; uint8_t block_num; diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index 8210e7f..1ff8e6e 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -80,7 +80,7 @@ extern void osm_req_get_node_desc(IN osm_sm_t * sm, osm_physp_t *p_physp); * **********************************************************************/ -typedef struct _osm_trap_aging_tracker_context { +typedef struct osm_trap_agingracker_context { osm_log_t *p_log; osm_physp_t *p_physp; } osm_trap_aging_tracker_context_t; diff --git a/opensm/opensm/osm_vl15intf.c b/opensm/opensm/osm_vl15intf.c index 246e6cc..a47f9ba 100644 --- a/opensm/opensm/osm_vl15intf.c +++ b/opensm/opensm/osm_vl15intf.c @@ -200,7 +200,7 @@ void osm_vl15_construct(IN osm_vl15_t * const p_vl) /********************************************************************** **********************************************************************/ void -osm_vl15_destroy(IN osm_vl15_t * const p_vl, IN struct _osm_mad_pool *p_pool) +osm_vl15_destroy(IN osm_vl15_t * const p_vl, IN struct osm_mad_pool *p_pool) { osm_madw_t *p_madw; -- 1.5.5.1.178.g1f811 From oratorize at consuelobellini.com Sun Jun 15 13:38:12 2008 From: oratorize at consuelobellini.com (King Graham) Date: Sun, 15 Jun 2008 22:38:12 +0200 Subject: [ofa-general] :EncoreDVD CS 3: Message-ID: <000201c8cf26$7e9d3d00$0100007f@vibbfcy> : Adobe CS3 Master Collection for PC or MAC includes: : InDesign CS3 : Photoshop CS3 : Illustrator CS3 : Acrobat 8 Professional : Flash CS3 Professional : Dreamweaver CS3 : Fireworks CS3 : Contribute CS3 : After Effects CS3 Professional : Premiere Pro CS3 : Encore DVD CS3 : Soundbooth CS3 : getsoftasap . com in Internet Exp!orer : System Requirements : For PC: : Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core : Duo (or compatible) processor; SSE2-enabled processor required for AMD systems : Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) : 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components : 38GB of available hard-disk space (additional free space required during installation) : Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred : Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) : 1,280x1,024 monitor resolution with 32-bit color adapter : DVD-ROM drive : For MAC: : PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) : Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server : 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components : 36GB of available hard-disk space (additional free space required during installation) : Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred : Core Audio compatible sound card : 1,280x1,024 monitor resolution with 32-bit color adapter : DVD-ROM drive: DVD+-R burner required for DVD creation A fungus that only attacks cacao trees swept through eastern Brazil in the late 1980s and early '90s, ravaging the area's cocoa industry. The blight set in motion a series of events that changed the ecology of the entire region, a devastating lesson in the laws of nature. In France, Bush Says Rift Is Not Permanent From 21ted at ijot.com Sun Jun 15 20:09:34 2008 From: 21ted at ijot.com (hinze morton) Date: Mon, 16 Jun 2008 03:09:34 +0000 Subject: [ofa-general] Best of Adidas, Burberry, Versace Message-ID: <000601c8cf6d$0479f13c$70e44285@jowtbbg> The most popular and largest luxury store for shoes and bags is just one click away. Estimated by thousands of satisfied customers all around the world, we carry dozens of populars brands including: Chanel Bally Boss Paul Smith Versace Here you willc see millions of stunning designs for shoes, and leather products, at rock bottom pri'c'ing. Sale close this week, so visit our store today and start pampering yourself and your loved ones. - Visit our site: www.jumpastio[DOT]com (copy this link and replace "[DOT]" to ".") From unhalter at thejerktowel.com Sun Jun 15 23:37:13 2008 From: unhalter at thejerktowel.com (Malloy Cox) Date: Mon, 16 Jun 2008 08:37:13 +0200 Subject: [ofa-general] *Premiere Pro CS 3* Message-ID: <000301c8cf79$a79aed80$0100007f@qhnwcmp> * Adobe CS3 Master Collection for PC or MAC includes: * InDesign CS3 * Photoshop CS3 * Illustrator CS3 * Acrobat 8 Professional * Flash CS3 Professional * Dreamweaver CS3 * Fireworks CS3 * Contribute CS3 * After Effects CS3 Professional * Premiere Pro CS3 * Encore DVD CS3 * Soundbooth CS3 * getsoftxp . com in your Internet Exp1orer * System Requirements * For PC: * Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core * Duo (or compatible) processor; SSE2-enabled processor required for AMD systems * Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) * 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components * 38GB of available hard-disk space (additional free space required during installation) * Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred * Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) * 1,280x1,024 monitor resolution with 32-bit color adapter * DVD-ROM drive * For MAC: * PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) * Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server * 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components * 36GB of available hard-disk space (additional free space required during installation) * Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred * Core Audio compatible sound card * 1,280x1,024 monitor resolution with 32-bit color adapter * DVD-ROM drive* DVD+-R burner required for DVD creation In Pakistan, lawyers and reformers have revived their campaign to reinstate the independent-minded judges that President Pervez Musharraf fired last year. The two main parties in the new legislative coalition do not agree on reinstating the judges. The two main parties in that coalition do not agree on reinstating the judges. In Paris today, President Bush delivered what amounted to a valedictory speech, looking back at U.S. and European relations over his time in the White House. He laid out the challenges ahead, including the wars in Afghanistan and Iraq. Bush is on what he says will be his final trip to Europe as president; earlier Friday, he met with Pope Benedict XVI. From sashak at voltaire.com Sun Jun 15 23:39:06 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 09:39:06 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616063905.GD24365@sashak.voltaire.com> Hi Hal, On 06:03 Sat 07 Jun , Hal Rosenstock wrote: > > 1. Change the current underlying multicast tree from being MLID based > to MGID based. This involves using fleximap rather than qmap. The > downside of this is that MLID lookups will be slower as now they > are not as "direct" as the MLID will no longer be the key in the map. > Rather than searching by MLID key, the tree will need to be scanned > entry by entry for MLID matches. Actually it is not necessary to slow down there. Instead of mgrp_mlid_tbl and in addition to MGID map we can just keep array of pointers to existing multicast groups indexed by MLID offset from 0xc000. It will be always limited in size and should be even faster than original "quick map". > It's unclear how much this will slow down > MLID searches but it is thought that none of these searches are > time critical (and shouldn't cause any existing timeouts to "pop"). Some searches are on a fast paths. I think it is important to not slow down there. > 2. Add in support for overloading MLIDs. On the configuration side, a > number of additional options would be added to consolidate_ipv6_snm_req. > These include the number of MLIDs to compress down to (default 16), > a multicast group (MGID) base address and (full MGID) mask. this would > default to 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY where Z is the scope, > XXXX is the P_Key, and YYYYYY is the last 24 bits of the port guid ( > the YYYYYY bits would be masked out by default). This is what the > current workaround uses for collapsing the multicast groups. > > The criteria for overloading MLIDs includes any group parameters that > need to be in common (e.g. rate. MTU, perhaps PKey (see below), etc.). > > Aside from changing the underlying implementation of MLID searches, > multicast group deletion wll need another check when there are no ports > left in a group. If that group is on a compressed MLID (this part of the > check is an optimization), then the multicast group tree needs to be checked > to ensure there are no other groups sharing that MLID. If we will have MLID indexed array with pointers to multicast groups kept in a list it should be easy to do. Something like: i: (MLID = 0xc000 + i) -> MGRP MGID1 -> MGRP MGID2 -> ... > IBA 1.2.1 v1 p.151 4.1.3 Local Identifiers item 10) states: > "When a multicast LID is overloaded, the multicast groups > sharing the same MLID must have the same P_Key. This simplification > is required to allow switches and routers that implement optional > P_Key enforcement for multicast operations." > This is part of the C4-5 compliance. > > OPEN ISSUE > As PKey is part of the MGID, does this need to be addressed (and if > so) how ? Since it is required by spec and we want to care about another common multicast parameters (mtu, rate, etc.) pkey can be handled in similar manner. In worst case we can disable/enable it with command line option or so. > If the approach above seems reasonable, I will work on such a set of patches. Great! Sasha From sashak at voltaire.com Sun Jun 15 23:50:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 09:50:29 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080608054426.GK22807@obsidianresearch.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080608054426.GK22807@obsidianresearch.com> Message-ID: <20080616065029.GF24365@sashak.voltaire.com> Hi Jason, On 23:44 Sat 07 Jun , Jason Gunthorpe wrote: > > Please, please use IPv6 notation for this (and all other GIDs). Since > the addressing scheme is taken from IPv6 the interesting bits align > sensibly.. The double 64 bit number notation used in osm is horrible, > and should be killed off :) AFAIR it was discussed and agreed in a past. Somebody just need to care about patches :) Sasha From sashak at voltaire.com Mon Jun 16 01:24:15 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 11:24:15 +0300 Subject: [ofa-general] Re: [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <1213397313.5369.356.camel@cardanus.llnl.gov> References: <1213397313.5369.356.camel@cardanus.llnl.gov> Message-ID: <20080616082415.GK24365@sashak.voltaire.com> Hi Al, On 15:48 Fri 13 Jun , Al Chu wrote: > > This is a conceptually simple option I've developed for updn routing. > > Currently in updn routing, nodes/guids are routed on switches in a > seemingly-random order, which I believe is due to internal data > structure organization (i.e. cl_qmap_apply_func is called on > port_guid_tbl) as well as how the fabric is scanned (it is logically > scanned from a port perspective, but it may not be logical from a node > perspective). I had a hypothesis that this was leading to increased > contention in the network for MPI. > > For example, suppose we have 12 uplinks from a leaf switch to a spine > switch. If we want to send data from this leaf switch to node[13-24], > the up links we will send on are pretty random. Yeah, the issue is known. And idea is good and useful. Actually we discussed this issue with Yiftah some time ago and his idea was to have an option to order routing generation by ports connected to leaf switches with higher number of active links. Something like: foreach switch reverse (higher is first) sorted by number of active links foreach port connected to the switch do_rounting() Which is good for most cases, but also I thought that in addition we will need something more configurable - just like your patch series :). Now comment: why do you think it is useful only for Up/Down? IMO min hops algo could benefit from this feature just well. If so this simplifies implementation - patches 2,3,4 are not needed, code from patch 5 is going to osm_ucast_mgr.c. What do you think? Sasha From sashak at voltaire.com Mon Jun 16 01:26:28 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 11:26:28 +0300 Subject: [ofa-general] Re: [OPENSM PATCH 1/5]: add guid_routing_order_file option In-Reply-To: <1213397327.5369.357.camel@cardanus.llnl.gov> References: <1213397327.5369.357.camel@cardanus.llnl.gov> Message-ID: <20080616082628.GL24365@sashak.voltaire.com> Hi Al, On 15:48 Fri 13 Jun , Al Chu wrote: > @@ -1547,6 +1551,10 @@ int osm_subn_write_conf_file(char *file_name, IN osm_subn_opt_t *const p_opts) > " Up/Down algorithm instead\n# of GUIDs (one guid and" > " id in each line)\nids_guid_file %s\n\n", > p_opts->ids_guid_file ? p_opts->ids_guid_file : null_str); > + > + fprintf(opts_file, > + "# The file holding guid routing order guids (for Up/Down)\n" > + "guid_routing_order_file %s\n\n", p_opts->guid_routing_order_file); I guess here should be: p_opts->guid_routing_order_file ? p_opts->guid_routing_order_file : null_str Sasha From sashak at voltaire.com Mon Jun 16 01:52:36 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 11:52:36 +0300 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616085236.GM24365@sashak.voltaire.com> On 14:40 Thu 12 Jun , Hal Rosenstock wrote: > > > @@ -1144,9 +1145,10 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > > > > > > *dcl = cl; > > > > > > - DEBUG("client %d, size %d", cl->id, size); > > > - if (size != sizeof(*r)) { > > > - IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); > > > + DEBUG("client %d size %d", cl->id, size); > > > + if (size > sizeof(*r) + MAD_BLOCK_SIZE) { > > > > What about partial write? > > This change was needed for short RMPP. Yes, but it is risky. Of course 'ibsim' is for testing, and not really production code, but anyway.. Another solution would be to encode packet (or payload) length into header. For this we even don't need to break ABI - there is unused 64-bit context field. > > The socket is non-blocked datagram. > > Is a framing layer needed ? I don't know. In my old implementation I switched to STREAM socket in order to not reinvent a wheel, but it is mostly due to long packets. > > > diff --git a/include/ibsim.h b/include/ibsim.h > > > index 84568e6..14a3f90 100644 > > > --- a/include/ibsim.h > > > +++ b/include/ibsim.h > > > @@ -1,5 +1,6 @@ > > > /* > > > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > > * > > > * This file is part of ibsim. > > > * > > > @@ -61,6 +62,8 @@ struct sim_port { > > > #define SIM_MAGIC 0xdeadbeef > > > #define SIM_CTL_MAX_DATA 64 > > > > > > +#define MAD_BLOCK_SIZE 256 > > > + > > > struct sim_request { > > > uint32_t dlid; > > > uint32_t slid; > > > @@ -68,7 +71,17 @@ struct sim_request { > > > uint32_t sqp; > > > uint32_t status; > > > uint64_t context; > > > - char mad[256]; > > > + char mad[0]; > > > +}; > > > + > > > +struct sim_req256 { > > > + uint32_t dlid; > > > + uint32_t slid; > > > + uint32_t dqp; > > > + uint32_t sqp; > > > + uint32_t status; > > > + uint64_t context; > > > + char mad[MAD_BLOCK_SIZE]; > > > }; > > > > Why do we need two separate structures? > > The second one was just to eliminate the memory allocation (and use the > stack). Only forwarded packets have the variability in size. Then I think it would be cleaner to have something like 'struct sim_req_header' and leave a lot of code unmodified. > > > > enum SIM_CTL_TYPES { > > > diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c > > > index 4cbf8da..9c69fb5 100644 > > > --- a/umad2sim/umad2sim.c > > > +++ b/umad2sim/umad2sim.c > > > @@ -1,5 +1,6 @@ > > > /* > > > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > > * > > > * This file is part of ibsim. > > > * > > > @@ -376,7 +377,7 @@ static int dev_sysfs_create(struct umad2sim_dev *dev) > > > > > > static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > { > > > - struct sim_request req; > > > + struct sim_req256 req; > > > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > > > unsigned mgmt_class; > > > int cnt; > > > @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > > > cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); > > > DEBUG("umad2sim_read: got %d...\n", cnt); > > > - if (cnt < sizeof(req)) { > > > - ERROR("umad2sim_read: partial request - skip.\n"); > > > - umad->status = EAGAIN; > > > - return umad_size(); > > > - } > > > > > > mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); > > > > This checked that we got at least req header before decode this. > > No; I think it checked that the entire packet existed as req was the > entire packet not just the header. > > > Why is it removed? > > Because of short RMPP. It can add back at least the header check part > but I think the partial write (long writes) is the fundamental issue. Yes, some check is needed. And above length encoded in sim_request header can be useful. > > > > @@ -411,7 +407,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > umad->status = ntohl(req.status); > > > umad->timeout_ms = 0; > > > umad->retries = 0; > > > - umad->length = umad_size() + sizeof(req.mad); > > > + umad->length = umad_size() + cnt; > > > > 'cnt' is normally actual MAD length + sim_request header size. Is it > > correct to put it as is in umad->length (shouldn't be umad_size() + cnt > > - sizeof(struct sim_request))? > > Not sure about this. I'll need to get back on this. > > > > > > > umad->addr.qpn = req.sqp; > > > umad->addr.qkey = 0; // agent->qkey; > > > @@ -431,9 +427,9 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > static ssize_t umad2sim_write(struct umad2sim_dev *dev, > > > const void *buf, size_t count) > > > { > > > - struct sim_request req; > > > + struct sim_request *req; > > > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > > > - int cnt; > > > + int cnt, ocnt; > > > > > > #ifdef SIMULATE_SEND_ERRORS > > > { static int err_count; > > > @@ -464,25 +460,32 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, > > > mad_get_field(umad_get_mad(umad), 0, IB_MAD_ATTRMOD_F) > > > ); > > > > > > - req.dlid = umad->addr.lid; > > > - req.slid = req.dlid == 0xffff ? 0xffff : 0; /* 0 - means auto > > > + cnt = count - umad_size(); > > > + if (cnt > MAD_BLOCK_SIZE) > > > + cnt = MAD_BLOCK_SIZE; > > > + req = malloc(sizeof(*req) + cnt); > > > > So why malloc() if size is limited by struct sim_request anyway? > > Because long RMPP isn't handled yet. Yes, and it is why I asked :). I don't think we need to touch this function yet for "for future" reason - long RMPP will require much more massive changes. Sasha From kaber at trash.net Mon Jun 16 01:54:25 2008 From: kaber at trash.net (Patrick McHardy) Date: Mon, 16 Jun 2008 10:54:25 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <20080614190358.GU22807@obsidianresearch.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <20080614190358.GU22807@obsidianresearch.com> Message-ID: <48562A41.7000304@trash.net> Jason Gunthorpe wrote: > On Sat, Jun 14, 2008 at 11:03:08AM -0700, Roland Dreier wrote: >>> We have linkstate/operstate for this. How is a user supposed >>> to configure the network device when it appears at a more or >>> less random time from his perspective? > >> Well, keep in mind that this driver is for an ethernet virtual NIC >> that is actually >> remote across another network (an InfiniBand fabric) -- so these devices >> can actually appear or disappear at random times by their nature. It's a >> similar issue to, say, a USB ethernet adapter presents -- there's no sane way >> to have operstate set for a device that hasn't been attached yet. > > I think that is only a fair analogy if the VNIC used some kind of > hot-plug auto discovery procedure like USB does. As soon as you > require the administrator to run a command to bind a device to a VNIC > GID you are much more like a tunnel device and the ethernet device > should exist from the moment the administator creates it up until the > administator destroys it. The state of the connection to the VNIC > should be reflected in some way other than device presence. Fully agreed. From vlad at lists.openfabrics.org Mon Jun 16 02:20:44 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 16 Jun 2008 02:20:44 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080616-0200 daily build status Message-ID: <20080616092044.AB56EE611F0@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on x86_64 with linux-2.6.24 Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -fomit-frame-pointer -g -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/.tmp_ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/.tmp_ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/.tmp_ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/.tmp_ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:361: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:361: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:361: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080616-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From 784marybeth.maruski at oag.state.ny.us Mon Jun 16 02:11:40 2008 From: 784marybeth.maruski at oag.state.ny.us (christopher boyd) Date: Mon, 16 Jun 2008 09:11:40 +0000 Subject: [ofa-general] Best of Bally, Burberry, UGG Message-ID: <000401c8cf9f$07537788$dabd52b3@pbuhjec> The most popular and largest luxury store for shoes and bags is just one click away. Recommended by thousands of grateful customers all around the world, we have dozens of famous brands including: Chanel Armani Chanel Shoes Dior Prada Shoes Here you willc find hundred thousands of best designs for shoes, and leather products, at rock bottom pri'c'ing. Sale close this weekend, so visit our store immediately and start pampering yourself and your loved ones. - Visit our site: www.micaelex[DOT]com (copy this link then replace "[DOT]" to ".") From ogerlitz at voltaire.com Mon Jun 16 04:56:16 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 16 Jun 2008 14:56:16 +0300 (IDT) Subject: [ofa-general] [PATCH RFC] librdmacm: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: RDMA_CM_EVENT_ADDR_CHANGE event can be used by librdmacm consumers that wish to have their RDMA sessions always use the same links (eg ) as the IP stack does. In the current code, this does not happen when bonding is used and fail-over happened, but the IB link used by an already existing session is operating fine. The kernel rdma-cm code was enhanced to use netevent notification for sensing that a change has happened in the IP stack, and deliver this event for ID that is misaligned that respect with the IP stack. The user can act on the event or just ignore it Signed-off-by: Or Gerlitz Sean, This is the patch to librdmacm that brings RDMA_CM_EVENT_ADDR_CHANGE to user space, please let me know your thoughts. The net-core/bonding patches on which this patch set is depedent upon where pushed by the bonding maintainer (Jay Vosburgh) to the netdev maintainer (Jeff Garzik). Assuming Jeff accepts them into his tree, does it makes sense to you to pull his tree into yours and then push the patch sequence to Roland, or you want me to push it to Roland and you will sign/ack the patches? We are now at -rc6 and I want to bring this to the for-2.6.27 branch of Roland's this week. Roland - is there an easier way to do this push? Or. --- examples/rping.c | 4 ++++ include/rdma/rdma_cma.h | 3 ++- man/rdma_get_cm_event.3 | 5 +++++ src/cma.c | 2 ++ 4 files changed, 13 insertions(+), 1 deletion(-) Index: librdmacm/examples/rping.c =================================================================== --- librdmacm.orig/examples/rping.c +++ librdmacm/examples/rping.c @@ -226,6 +226,10 @@ static int rping_cma_event_handler(struc ret = -1; break; + case RDMA_CM_EVENT_ADDR_CHANGE: + fprintf(stderr, "ADDR CHANGE event, ignoring\n"); + break; + default: fprintf(stderr, "oof bad type!\n"); sem_post(&cb->sem); Index: librdmacm/include/rdma/rdma_cma.h =================================================================== --- librdmacm.orig/include/rdma/rdma_cma.h +++ librdmacm/include/rdma/rdma_cma.h @@ -57,7 +57,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_ADDR_CHANGE }; enum rdma_port_space { Index: librdmacm/src/cma.c =================================================================== --- librdmacm.orig/src/cma.c +++ librdmacm/src/cma.c @@ -1438,6 +1438,8 @@ const char *rdma_event_str(enum rdma_cm_ return "RDMA_CM_EVENT_MULTICAST_JOIN"; case RDMA_CM_EVENT_MULTICAST_ERROR: return "RDMA_CM_EVENT_MULTICAST_ERROR"; + case RDMA_CM_EVENT_ADDR_CHANGE: + return "RDMA_CM_EVENT_ADDR_CHANGE"; default: return "UNKNOWN EVENT"; } Index: librdmacm/man/rdma_get_cm_event.3 =================================================================== --- librdmacm.orig/man/rdma_get_cm_event.3 +++ librdmacm/man/rdma_get_cm_event.3 @@ -140,6 +140,11 @@ The multicast join operation (rdma_join_ An error either occurred joining a multicast group, or, if the group had already been joined, on an existing group. The specified multicast group is no longer accessible and should be rejoined, if desired. +.IP RDMA_CM_EVENT_ADDR_CHANGE_ +The network device associated with this ID through address resolution changed its +HW address, eg following of bonding failover. This event can serve as a hint for +applications who want the links used for their RDMA sessions to align with the +network stack. .SH "SEE ALSO" rdma_ack_cm_event(3), rdma_create_event_channel(3), rdma_resolve_addr(3), rdma_resolve_route(3), rdma_connect(3), rdma_listen(3), rdma_join_multicast(3), From ogerlitz at voltaire.com Mon Jun 16 05:04:34 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 16 Jun 2008 15:04:34 +0300 (IDT) Subject: [ofa-general] Re: [RFC V4 PATCH 4/5] rdma/cma: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: On Wed, 28 May 2008, Or Gerlitz wrote: > RDMA_CM_EVENT_ADDR_CHANGE event can be used by rdma-cm consuamers that wish > to have their RDMA sessions always use the same links (eg ) as the > IP stack does. In the current code, this does not happen when bonding is used > and fail-over happened, but the IB link used by an already existing session is > operating fine. > > Use netevent notification for sensing that a change has happened in the IP stack, > then scan the rdma-cm IDs list to see if there is an ID that is "misaligned" in > that respect with the IP stack, and deliver RDMA_CM_EVENT_ADDR_CHANGE for > this ID. The user can act on the event or just ignore it > > Signed-off-by: Or Gerlitz Tom, I'd like to patch the nfs/rdma kernel code at the way you'd like for this event. Do you prefer the code to just ignore it or you want to reconnect when this happens? Or. From hrosenstock at xsigo.com Mon Jun 16 06:07:51 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 06:07:51 -0700 Subject: [ofa-general] [PATCHv4][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements Message-ID: <1213621671.14047.606.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Some error message improvements Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 1a9bc54..caa50d7 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -1122,7 +1122,10 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) if (p_ctxt->p_mgrp) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B30: " - "Multiple MC groups for same MGID\n"); + "Multiple MC groups for MGID " + "0x%016" PRIx64 " : 0x%016" PRIx64 "\n", + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix), + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id)); return; } p_ctxt->p_mgrp = p_mgrp; @@ -1280,7 +1283,10 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, } else { CL_PLOCK_RELEASE(sa->p_lock); OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Failed since multicast group not present\n"); + "Failed since multicast group 0x%16" + PRIx64 " : 0x%016" PRIx64 " not present\n", + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix), + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id)); osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); goto Exit; } -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-osmsamcm4 Type: text/x-patch Size: 1196 bytes Desc: not available URL: From hrosenstock at xsigo.com Mon Jun 16 06:11:22 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 06:11:22 -0700 Subject: [ofa-general] Re: [PATCHv3][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements In-Reply-To: <20080615063727.GK24365@sashak.voltaire.com> References: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> <20080615063727.GK24365@sashak.voltaire.com> Message-ID: <1213621882.14047.611.camel@hrosenstock-ws.xsigo.com> On Sun, 2008-06-15 at 09:37 +0300, Sasha Khapyorsky wrote: > On 08:30 Fri 13 Jun , Hal Rosenstock wrote: > > opensm/osm_sa_mcmember_record.c: Some error message improvements > > > > Signed-off-by: Hal Rosenstock > > > > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > > index de174c5..df4eca3 100644 > > --- a/opensm/opensm/osm_sa_mcmember_record.c > > +++ b/opensm/opensm/osm_sa_mcmember_record.c > > @@ -1121,7 +1121,10 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > > > > if (p_ctxt->p_mgrp) { > > OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B30: " > > - "Multiple MC groups for same MGID\n"); > > + "Multiple MC groups for MGID " > > + "0x%016" PRIx64 " : 0x%016" PRIx64 "\n", > > + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix), > > + cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id)); > > return; > > } > > p_ctxt->p_mgrp = p_mgrp; > > @@ -1278,8 +1281,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, > > } > > } else { > > CL_PLOCK_RELEASE(sa->p_lock); > > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > - "Failed since multicast group not present\n"); > > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " > > + "Failed since multicast group 0x%16" > > Why to change log level? It is not OpenSM error, but invalid request. Are errors really restricted to OpenSM (internal) errors ? I thought they were more inclusive than that including network events that were "interesting events". In this specific case, I agree it is a bad idea as I think restarting IPoIB does causes this code path to be used. Modified patch already issued. -- Hal > Please next time if you are changing log level state it in commit > comment message. > > Sasha > > > + PRIx64 " : 0x%016" PRIx64 " not present\n", > > + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.prefix), > > + cl_ntoh64(p_recvd_mcmember_rec->mgid.unicast.interface_id)); > > osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); > > goto Exit; > > } > > > > From kliteyn at dev.mellanox.co.il Mon Jun 16 06:11:00 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 16 Jun 2008 16:11:00 +0300 Subject: [ofa-general] [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria Message-ID: <48566664.1010004@dev.mellanox.co.il> Hi Sasha, Adding log message for matched criteria of the QoS policy rule. This patch addresses all the issues that were brought up during the previous versions, mainly the fact that the prev. version was printing one log message per matched criterion (this one prints single message at the end of matching rule search). Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_qos_policy.c | 52 ++++++++++++++++++++++++++++++++++++++- 1 files changed, 50 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index f1d4e54..106a2cb 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -595,10 +595,15 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( { osm_qos_match_rule_t *p_qos_match_rule = NULL; cl_list_iterator_t list_iterator; + osm_log_t * p_log = &p_qos_policy->p_subn->p_osm->log; + char buff[999]; + int n; if (!cl_list_count(&p_qos_policy->qos_match_rules)) return NULL; + OSM_LOG_ENTER(p_log); + /* Go over all QoS match rules and find the one that matches the request */ list_iterator = cl_list_head(&p_qos_policy->qos_match_rules); @@ -609,6 +614,7 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( list_iterator = cl_list_next(list_iterator); continue; } + n = 0; /* If a match rule has Source groups, PR request source has to be in this list */ @@ -621,6 +627,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( list_iterator = cl_list_next(list_iterator); continue; } + + if (n) + n += snprintf(buff + n, sizeof(buff) - n, + ", "); + n += snprintf(buff + n, sizeof(buff) - n, + "S_Port (0x%016" PRIx64 ")", + cl_ntoh64(osm_physp_get_port_guid( + p_src_physp))); } /* If a match rule has Destination groups, PR request dest. has to be in this list */ @@ -634,6 +648,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( list_iterator = cl_list_next(list_iterator); continue; } + + if (n) + n += snprintf(buff + n, sizeof(buff) - n, + ", "); + n += snprintf(buff + n, sizeof(buff) - n, + "D_Port (0x%016" PRIx64 ")", + cl_ntoh64(osm_physp_get_port_guid( + p_dest_physp))); } /* If a match rule has QoS classes, PR request HAS @@ -653,6 +675,11 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( continue; } + if (n) + n += snprintf(buff + n, sizeof(buff) - n, + ", "); + n += snprintf(buff + n, sizeof(buff) - n, + "QoS_Class (0x%04x)", qos_class); } /* If a match rule has Service IDs, PR request HAS @@ -673,6 +700,12 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( continue; } + if (n) + n += snprintf(buff + n, sizeof(buff) - n, + ", "); + n += snprintf(buff + n, sizeof(buff) - n, + "ServiceID (0x%016" PRIx64 ")", + service_id); } /* If a match rule has PKeys, PR request HAS @@ -692,6 +725,11 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( continue; } + if (n) + n += snprintf(buff + n, sizeof(buff) - n, + ", "); + n += snprintf(buff + n, sizeof(buff) - n, + "PKey (0x%04x)", pkey); } /* if we got here, then this match-rule matched this PR request */ @@ -699,10 +737,20 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( } if (list_iterator == cl_list_end(&p_qos_policy->qos_match_rules)) - return NULL; + p_qos_match_rule = NULL; + if (p_qos_match_rule) + OSM_LOG(p_log, OSM_LOG_DEBUG, + "request matched by: %s\n", + (n == 0) ? "(empty)" : buff); + else + OSM_LOG(p_log, OSM_LOG_DEBUG, + "request not matched to any rule\n"); + + + OSM_LOG_EXIT(p_log); return p_qos_match_rule; -} /* __qos_policy_get_match_rule_by_pr() */ +} /* __qos_policy_get_match_rule_by_params() */ /*************************************************** ***************************************************/ -- 1.5.1.4 From kliteyn at dev.mellanox.co.il Mon Jun 16 06:14:00 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 16 Jun 2008 16:14:00 +0300 Subject: [ofa-general] [PATCH] opensm/osm_qos_parser.y: TRIVIAL addition in log message Message-ID: <48566718.80909@dev.mellanox.co.il> Hi Sasha, Adding new line in one log (error) message. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_qos_parser.y | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/opensm/osm_qos_parser.y b/opensm/opensm/osm_qos_parser.y index 4d9f572..a8fa16d 100644 --- a/opensm/opensm/osm_qos_parser.y +++ b/opensm/opensm/osm_qos_parser.y @@ -2381,7 +2381,7 @@ static void __qos_parser_error(const char *format, ...) va_end(pvar); OSM_LOG(p_qos_parser_osm_log, OSM_LOG_ERROR, "ERR AC05: " - "Syntax error (line %d:%d): %s", + "Syntax error (line %d:%d): %s\n", line_num, column_num, s); fprintf(stderr, "Error in QoS Policy File (line %d:%d): %s.\n", line_num, column_num, s); -- 1.5.1.4 From hrosenstock at xsigo.com Mon Jun 16 06:45:42 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 06:45:42 -0700 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <20080616085236.GM24365@sashak.voltaire.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> Message-ID: <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 11:52 +0300, Sasha Khapyorsky wrote: > On 14:40 Thu 12 Jun , Hal Rosenstock wrote: > > > > @@ -1144,9 +1145,10 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) > > > > > > > > *dcl = cl; > > > > > > > > - DEBUG("client %d, size %d", cl->id, size); > > > > - if (size != sizeof(*r)) { > > > > - IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); > > > > + DEBUG("client %d size %d", cl->id, size); > > > > + if (size > sizeof(*r) + MAD_BLOCK_SIZE) { > > > > > > What about partial write? > > > > This change was needed for short RMPP. > > Yes, but it is risky. Of course 'ibsim' is for testing, and not really > production code, but anyway.. > > Another solution would be to encode packet (or payload) length into > header. Yes, that was an approach I entertained. > For this we even don't need to break ABI - there is unused > 64-bit context field. > > > The socket is non-blocked datagram. > > > > Is a framing layer needed ? > > I don't know. In my old implementation I switched to STREAM socket in > order to not reinvent a wheel, but overall performance suffered, right ? > but it is mostly due to long packets. Right; so maybe an additional (reliable) socket for only these request/response types ? > > > > diff --git a/include/ibsim.h b/include/ibsim.h > > > > index 84568e6..14a3f90 100644 > > > > --- a/include/ibsim.h > > > > +++ b/include/ibsim.h > > > > @@ -1,5 +1,6 @@ > > > > /* > > > > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > > > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > > > * > > > > * This file is part of ibsim. > > > > * > > > > @@ -61,6 +62,8 @@ struct sim_port { > > > > #define SIM_MAGIC 0xdeadbeef > > > > #define SIM_CTL_MAX_DATA 64 > > > > > > > > +#define MAD_BLOCK_SIZE 256 > > > > + > > > > struct sim_request { > > > > uint32_t dlid; > > > > uint32_t slid; > > > > @@ -68,7 +71,17 @@ struct sim_request { > > > > uint32_t sqp; > > > > uint32_t status; > > > > uint64_t context; > > > > - char mad[256]; > > > > + char mad[0]; > > > > +}; > > > > + > > > > +struct sim_req256 { > > > > + uint32_t dlid; > > > > + uint32_t slid; > > > > + uint32_t dqp; > > > > + uint32_t sqp; > > > > + uint32_t status; > > > > + uint64_t context; > > > > + char mad[MAD_BLOCK_SIZE]; > > > > }; > > > > > > Why do we need two separate structures? > > > > The second one was just to eliminate the memory allocation (and use the > > stack). Only forwarded packets have the variability in size. > > Then I think it would be cleaner to have something like 'struct > sim_req_header' Sure. > and leave a lot of code unmodified. It's a minor amount of code currently. > > > > enum SIM_CTL_TYPES { > > > > diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c > > > > index 4cbf8da..9c69fb5 100644 > > > > --- a/umad2sim/umad2sim.c > > > > +++ b/umad2sim/umad2sim.c > > > > @@ -1,5 +1,6 @@ > > > > /* > > > > * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. > > > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > > > * > > > > * This file is part of ibsim. > > > > * > > > > @@ -376,7 +377,7 @@ static int dev_sysfs_create(struct umad2sim_dev *dev) > > > > > > > > static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > { > > > > - struct sim_request req; > > > > + struct sim_req256 req; > > > > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > > > > unsigned mgmt_class; > > > > int cnt; > > > > @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > > > > > cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); > > > > DEBUG("umad2sim_read: got %d...\n", cnt); > > > > - if (cnt < sizeof(req)) { > > > > - ERROR("umad2sim_read: partial request - skip.\n"); > > > > - umad->status = EAGAIN; > > > > - return umad_size(); > > > > - } > > > > > > > > mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); > > > > > > This checked that we got at least req header before decode this. > > > > No; I think it checked that the entire packet existed as req was the > > entire packet not just the header. > > > > > Why is it removed? > > > > Because of short RMPP. It can add back at least the header check part > > but I think the partial write (long writes) is the fundamental issue. > > Yes, some check is needed. And above length encoded in sim_request header > can be useful. A header check can be left there. It's insufficient IMO though. > > > > > > @@ -411,7 +407,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > umad->status = ntohl(req.status); > > > > umad->timeout_ms = 0; > > > > umad->retries = 0; > > > > - umad->length = umad_size() + sizeof(req.mad); > > > > + umad->length = umad_size() + cnt; > > > > > > 'cnt' is normally actual MAD length + sim_request header size. Is it > > > correct to put it as is in umad->length (shouldn't be umad_size() + cnt > > > - sizeof(struct sim_request))? > > > > Not sure about this. I'll need to get back on this. > > > > > > > > > > umad->addr.qpn = req.sqp; > > > > umad->addr.qkey = 0; // agent->qkey; > > > > @@ -431,9 +427,9 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > static ssize_t umad2sim_write(struct umad2sim_dev *dev, > > > > const void *buf, size_t count) > > > > { > > > > - struct sim_request req; > > > > + struct sim_request *req; > > > > ib_user_mad_t *umad = (ib_user_mad_t *) buf; > > > > - int cnt; > > > > + int cnt, ocnt; > > > > > > > > #ifdef SIMULATE_SEND_ERRORS > > > > { static int err_count; > > > > @@ -464,25 +460,32 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, > > > > mad_get_field(umad_get_mad(umad), 0, IB_MAD_ATTRMOD_F) > > > > ); > > > > > > > > - req.dlid = umad->addr.lid; > > > > - req.slid = req.dlid == 0xffff ? 0xffff : 0; /* 0 - means auto > > > > + cnt = count - umad_size(); > > > > + if (cnt > MAD_BLOCK_SIZE) > > > > + cnt = MAD_BLOCK_SIZE; > > > > + req = malloc(sizeof(*req) + cnt); > > > > > > So why malloc() if size is limited by struct sim_request anyway? > > > > Because long RMPP isn't handled yet. > > Yes, and it is why I asked :). I don't think we need to touch this > function yet for "for future" reason - long RMPP will require much more > massive changes. I guess at this point I'm unclear about moving forward. The only clear thing to me is that with long RMPP it might be accepted. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From tziporet at mellanox.co.il Mon Jun 16 06:59:34 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 16 Jun 2008 16:59:34 +0300 Subject: [ofa-general] OFED meeting agenda for today (June 16, 2008) Message-ID: <6C2C79E72C305246B504CBA17B5500C90440EF3B@mtlexch01.mtl.com> This is the agenda for OFED meeting today (June 16, 2008) 1. OFED 1.4 status: - Kernel: - Merged with 2.6.26-rc6 - SRP still failing - under work - qlogic_vnic, iSER and nes - we have not tried - owners need to take care - New components: iSER target and NFSoRDMA - we still wait for the maintainers to submit the new code. - Userspace status: - libiverbs - took latest changes from Roland's tree - other components - moving to master git trees today (unless someone has an objection) 2. Close feature freeze date for OFED 1.4: Since we delayed the release to October 6, and we wish to base our code on kernel 2.6.27 I think doing a feature freeze by end of June is too early, since features that will come will 2.6.27 will actually open the feature freeze. My proposal is to do the feature freeze closer to 2.6 27 - at middle of July Discussion ... 3. Electing a chair and co-chair to the EWG: Proposal - Chair - Tziporet - Co-chairs - Yiftach from Voltaire and Gopal from Cisco 4. Open discussion Tziporet -- Tziporet Koren Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 380) Mobile: +972 (54) 468 9426 E-mail: tziporet at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From hrosenstock at xsigo.com Mon Jun 16 07:01:11 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 07:01:11 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080616063905.GD24365@sashak.voltaire.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> Message-ID: <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> Hi Sasha, On Mon, 2008-06-16 at 09:39 +0300, Sasha Khapyorsky wrote: > Hi Hal, > > On 06:03 Sat 07 Jun , Hal Rosenstock wrote: > > > > 1. Change the current underlying multicast tree from being MLID based > > to MGID based. This involves using fleximap rather than qmap. The > > downside of this is that MLID lookups will be slower as now they > > are not as "direct" as the MLID will no longer be the key in the map. > > Rather than searching by MLID key, the tree will need to be scanned > > entry by entry for MLID matches. > > Actually it is not necessary to slow down there. Instead of mgrp_mlid_tbl > and in addition to MGID map we can just keep array of pointers to > existing multicast groups indexed by MLID offset from 0xc000. It will be > always limited in size and should be even faster than original "quick > map". Is a 16K array of pointers preferable to some map ? > > It's unclear how much this will slow down > > MLID searches but it is thought that none of these searches are > > time critical (and shouldn't cause any existing timeouts to "pop"). > > Some searches are on a fast paths. I think it is important to not slow > down there. The comment was at a fine not coarse level. There was some minor additions thought to be in the code path. > > 2. Add in support for overloading MLIDs. On the configuration side, a > > number of additional options would be added to consolidate_ipv6_snm_req. > > These include the number of MLIDs to compress down to (default 16), > > a multicast group (MGID) base address and (full MGID) mask. this would > > default to 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY where Z is the scope, > > XXXX is the P_Key, and YYYYYY is the last 24 bits of the port guid ( > > the YYYYYY bits would be masked out by default). This is what the > > current workaround uses for collapsing the multicast groups. > > > > The criteria for overloading MLIDs includes any group parameters that > > need to be in common (e.g. rate. MTU, perhaps PKey (see below), etc.). > > > > Aside from changing the underlying implementation of MLID searches, > > multicast group deletion wll need another check when there are no ports > > left in a group. If that group is on a compressed MLID (this part of the > > check is an optimization), then the multicast group tree needs to be checked > > to ensure there are no other groups sharing that MLID. > > If we will have MLID indexed array with pointers to multicast groups > kept in a list it should be easy to do. Something like: > > i: (MLID = 0xc000 + i) -> MGRP MGID1 -> MGRP MGID2 -> ... That's how I envisioned the mgrp_mlid_tbl changing. It would point at an MLID and a MGID "chain" rather than an individual MGID. > > IBA 1.2.1 v1 p.151 4.1.3 Local Identifiers item 10) states: > > "When a multicast LID is overloaded, the multicast groups > > sharing the same MLID must have the same P_Key. This simplification > > is required to allow switches and routers that implement optional > > P_Key enforcement for multicast operations." > > This is part of the C4-5 compliance. > > > > OPEN ISSUE > > As PKey is part of the MGID, does this need to be addressed (and if > > so) how ? > > Since it is required by spec It's unclear the spec is "right" in this. > and we want to care about another common > multicast parameters (mtu, rate, etc.) pkey can be handled in similar > manner. Yes, it's potentially just another parameter for the sharing eligibility. > In worst case we can disable/enable it with command line option > or so. I was hoping that isn't needed. -- Hal > > If the approach above seems reasonable, I will work on such a set of patches. > > Great! > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jackm at dev.mellanox.co.il Mon Jun 16 07:05:40 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 16 Jun 2008 17:05:40 +0300 Subject: [ofa-general] Re: [PATCH v2] Add enum strings and *_str functions for enums In-Reply-To: References: <20080415133548.414aeaea.weiny2@llnl.gov> Message-ID: <200806161705.41183.jackm@dev.mellanox.co.il> On Saturday 19 April 2008 00:54, Roland Dreier wrote: > Thanks, I added a man page and changed things a little and committed the > following: > > commit 1c0b7ac0a6bbbe4d246ef4cf50ae31bde4929ba3 > Author: Ira Weiny > Date: Tue Apr 15 13:35:48 2008 -0700 > > Add functions to convert enum values to strings > > Add ibv_xxx_str() functions to convert node type, port state, event > type and wc status enum values to strings. > > Signed-off-by: Ira K. Weiny > Signed-off-by: Roland Dreier > > The change below (in the output format of the port state string) is causing us problems (with all sorts of scripts based upon ibv_devinfo). This only surfaced now because we're only just now bringing up the OFED 1.4 tree based upon the current libibverbs and kernel 2.6.26. Unfortunately, I missed this change when you posted it to the list in April. I assume that the only reason for the change below, aside from adding the defer state, was cosmetic. Is it possible to change the output strings so that for previously defined port states, the output remains what it was previously? (I don't see this as any different from other userspace library backwards-compatibility issues, the change breaks existing scripts). To avoid breaking the scripts, we would need as a minimum: + static const char *const port_state_str[] = { + [IBV_PORT_NOP] = "no state change (NOP)", + [IBV_PORT_DOWN] = "PORT_DOWN", + [IBV_PORT_INIT] = "PORT_INIT", + [IBV_PORT_ARMED] = "PORT_ARMED", + [IBV_PORT_ACTIVE] = "PORT_ACTIVE", + [IBV_PORT_ACTIVE_DEFER] = "active defer" + }; + + if (port_state < IBV_PORT_NOP || port_state > IBV_PORT_ACTIVE_DEFER) + return "invalid state"; I have no preferences for "active defer", but you might consider "PORT_ACTIVE_DEFER" for consistency. Our scripts do not work with the ACTIVE_DEFER state. (100% backwards compatibility would dictate that the active_defer state and the NOP state also return "invalid state" -- but this is clearly impossible to do). Please note also that "unknown" was previously "invalid state". - Jack > -static const char *port_state_str(enum ibv_port_state pstate) > -{ > - switch (pstate) { > - case IBV_PORT_DOWN: return "PORT_DOWN"; > - case IBV_PORT_INIT: return "PORT_INIT"; > - case IBV_PORT_ARMED: return "PORT_ARMED"; > - case IBV_PORT_ACTIVE: return "PORT_ACTIVE"; > - default: return "invalid state"; > - } > -} > - > + > +const char *ibv_port_state_str(enum ibv_port_state port_state) > +{ > + static const char *const port_state_str[] = { > + [IBV_PORT_NOP] = "no state change (NOP)", > + [IBV_PORT_DOWN] = "down", > + [IBV_PORT_INIT] = "init", > + [IBV_PORT_ARMED] = "armed", > + [IBV_PORT_ACTIVE] = "active", > + [IBV_PORT_ACTIVE_DEFER] = "active defer" > + }; > + > + if (port_state < IBV_PORT_NOP || port_state > IBV_PORT_ACTIVE_DEFER) > + return "unknown"; > + > + return port_state_str[port_state]; > +} From hrosenstock at xsigo.com Mon Jun 16 07:14:01 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 07:14:01 -0700 Subject: [ofa-general] Re: [PATCH v2] Add enum strings and *_str functions for enums In-Reply-To: <200806161705.41183.jackm@dev.mellanox.co.il> References: <20080415133548.414aeaea.weiny2@llnl.gov> <200806161705.41183.jackm@dev.mellanox.co.il> Message-ID: <1213625641.14047.647.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 17:05 +0300, Jack Morgenstein wrote: > On Saturday 19 April 2008 00:54, Roland Dreier wrote: > > Thanks, I added a man page and changed things a little and committed the > > following: > > > > commit 1c0b7ac0a6bbbe4d246ef4cf50ae31bde4929ba3 > > Author: Ira Weiny > > Date: Tue Apr 15 13:35:48 2008 -0700 > > > > Add functions to convert enum values to strings > > > > Add ibv_xxx_str() functions to convert node type, port state, event > > type and wc status enum values to strings. > > > > Signed-off-by: Ira K. Weiny > > Signed-off-by: Roland Dreier > > > > > The change below (in the output format of the port state string) > is causing us problems (with all sorts of scripts based upon ibv_devinfo). > This only surfaced now because we're only just now bringing up the OFED 1.4 tree > based upon the current libibverbs and kernel 2.6.26. > > Unfortunately, I missed this change when you posted it to the list in April. > > I assume that the only reason for the change below, aside from adding the defer state, Is it needed to support active defer ? Isn't active defer a SNO condition in terms of PortState ? -- Hal > was cosmetic. > Is it possible to change the output strings so that for previously defined port states, > the output remains what it was previously? > > (I don't see this as any different from other userspace library backwards-compatibility issues, > the change breaks existing scripts). > > To avoid breaking the scripts, we would need as a minimum: > > + static const char *const port_state_str[] = { > + [IBV_PORT_NOP] = "no state change (NOP)", > + [IBV_PORT_DOWN] = "PORT_DOWN", > + [IBV_PORT_INIT] = "PORT_INIT", > + [IBV_PORT_ARMED] = "PORT_ARMED", > + [IBV_PORT_ACTIVE] = "PORT_ACTIVE", > + [IBV_PORT_ACTIVE_DEFER] = "active defer" > + }; > + > + if (port_state < IBV_PORT_NOP || port_state > IBV_PORT_ACTIVE_DEFER) > + return "invalid state"; > > I have no preferences for "active defer", but you might consider > "PORT_ACTIVE_DEFER" for consistency. Our scripts do not work with the ACTIVE_DEFER > state. (100% backwards compatibility would dictate that the active_defer state and > the NOP state also return "invalid state" -- but this is clearly impossible to do). > > Please note also that "unknown" was previously "invalid state". > - Jack > > > -static const char *port_state_str(enum ibv_port_state pstate) > > -{ > > - switch (pstate) { > > - case IBV_PORT_DOWN: return "PORT_DOWN"; > > - case IBV_PORT_INIT: return "PORT_INIT"; > > - case IBV_PORT_ARMED: return "PORT_ARMED"; > > - case IBV_PORT_ACTIVE: return "PORT_ACTIVE"; > > - default: return "invalid state"; > > - } > > -} > > - > > + > > +const char *ibv_port_state_str(enum ibv_port_state port_state) > > +{ > > + static const char *const port_state_str[] = { > > + [IBV_PORT_NOP] = "no state change (NOP)", > > + [IBV_PORT_DOWN] = "down", > > + [IBV_PORT_INIT] = "init", > > + [IBV_PORT_ARMED] = "armed", > > + [IBV_PORT_ACTIVE] = "active", > > + [IBV_PORT_ACTIVE_DEFER] = "active defer" > > + }; > > + > > + if (port_state < IBV_PORT_NOP || port_state > IBV_PORT_ACTIVE_DEFER) > > + return "unknown"; > > + > > + return port_state_str[port_state]; > > +} > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From kliteyn at dev.mellanox.co.il Mon Jun 16 07:32:42 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 16 Jun 2008 17:32:42 +0300 Subject: [ofa-general] opensm routing In-Reply-To: <48516793.3090708@nasa.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> Message-ID: <4856798A.8020006@dev.mellanox.co.il> Jeff, Jeff Becker wrote: > Hi Al > > Al Chu wrote: >> Hey Jeff, >> >> >>> That works. The compute nodes need to talk to other compute nodes for >>> MPI over one set of links, and they need to talk to the Lustre nodes >>> for I/O, but over a different (disjoint) set of links. Thanks. >>> >> >> Is there a strong belief that a different/disjoint set of links would be >> beneficial? Sometime ago, Sasha and I iterated on a patch in which I >> found out sometimes not all switch ports would be used. In this >> particular case, a chunk of leaf switches were sometimes using only 11 >> out of 12 uplinks. After the fix, mpigraph showed about 20% improvement >> in MPI bandwidth. >> > Basically, we want to avoid situations where I/O and MPI contend for the > same links, and get in each other's way. What about using different VLs for MPI and I/O? It won't buy more bandwidth, but it might prevent MPI and I/O from congesting each other - they will share the wire according to the priority that you will define. -- Yevgeny > -jeff From eli at dev.mellanox.co.il Mon Jun 16 07:38:02 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 16 Jun 2008 17:38:02 +0300 Subject: [ofa-general] Re: IB/ipoib: no need to call skb_orphan() in cm_send()? In-Reply-To: <200806022101.02400.hnguyen@linux.vnet.ibm.com> References: <200806022101.02400.hnguyen@linux.vnet.ibm.com> Message-ID: <1213627082.5447.80.camel@mtls03> On Mon, 2008-06-02 at 21:01 +0200, Hoang-Nam Nguyen wrote: > Hello Eli! > I'm referring to the thread below, where you explained the rational of > calling skb_orphan() after post_send() in UD path: > [ewg] Re: [PATCH] call skb_orphan() after sending an SKB > http://lists.openfabrics.org/pipermail/ewg/2008-February/005739.html > > Looking at cm_send path I don't see that call. Is there any reason not to > do that here? > Hi Nam, sorry I did not respond - your email somehow slid and I did not see it. There are two reason why I use skb_orphan for UD mode: 1. I poll the CQ once in 16 posts 2. I do not arm the CQ This may lead to a situation where it takes some time before the SKB is released and the socket might be sensitive to this (ICMP for example). In CM mode, the sender is CQ is interrupt driven so the SKB will be freed quite soon after it was posted so problem does not occur. I hope that answers your question. From sashak at voltaire.com Mon Jun 16 07:42:03 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 17:42:03 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616144203.GC13363@sashak.voltaire.com> On 07:01 Mon 16 Jun , Hal Rosenstock wrote: > > Is a 16K array of pointers preferable to some map ? I would think so, but didn't check deeply. And in fact actual size of such array can be limited by max supported number of MLIDs in a fabric (which is just 1K today). Sasha From hrosenstock at xsigo.com Mon Jun 16 07:48:03 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 07:48:03 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080616144203.GC13363@sashak.voltaire.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> Message-ID: <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 17:42 +0300, Sasha Khapyorsky wrote: > On 07:01 Mon 16 Jun , Hal Rosenstock wrote: > > > > Is a 16K array of pointers preferable to some map ? > > I would think so, but didn't check deeply. And in fact actual size of > such array can be limited by max supported number of MLIDs in a fabric Is that really a good idea to limit it in this way ? Guess it could be limited with some configurable option but even so if a device with a larger MFT comes along then the SM might not work. > (which is just 1K today). Depends on what you mean by today. -- Hal > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From eli at mellanox.co.il Mon Jun 16 07:51:42 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 16 Jun 2008 17:51:42 +0300 Subject: [ofa-general] [PATCH] libmlx4: modify a QP to reset before destroying it Message-ID: <1213627902.5447.88.camel@mtls03> Roland, this is just a resend since I did not CC the mailing list. >From 1705a59b2199e93bfcf73d03717f6431c471bfd0 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Sun, 15 Jun 2008 15:40:52 +0300 Subject: [PATCH] libmlx4: modify a QP to reset before destroying it This is required to be done before cleaning the CQ from CQEs belonging to this QP. If we fail to do this, we might get failures when subsequently polling the CQ. Signed-off-by: Eli Cohen --- src/verbs.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/src/verbs.c b/src/verbs.c index bb904e2..693ed68 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -557,6 +557,15 @@ int mlx4_destroy_qp(struct ibv_qp *ibqp) { struct mlx4_qp *qp = to_mqp(ibqp); int ret; + struct ibv_qp_attr attr = { + .qp_state = IBV_QPS_RESET, + }; + + if (ibqp->state != IBV_QPS_RESET) { + ret = mlx4_modify_qp(ibqp, &attr, IBV_QP_STATE); + if (ret) + return ret; + } mlx4_cq_clean(to_mcq(ibqp->recv_cq), ibqp->qp_num, ibqp->srq ? to_msrq(ibqp->srq) : NULL); -- 1.5.5.4 -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From sashak at voltaire.com Mon Jun 16 07:54:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 17:54:29 +0300 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616145429.GD13363@sashak.voltaire.com> On 06:45 Mon 16 Jun , Hal Rosenstock wrote: > > > For this we even don't need to break ABI - there is unused > > 64-bit context field. > > > > > The socket is non-blocked datagram. > > > > > > Is a framing layer needed ? > > > > I don't know. In my old implementation I switched to STREAM socket in > > order to not reinvent a wheel, > > but overall performance suffered, right ? Yes, but mainly it was with huge 100M-1G RMPP packets - just fing this not very useful. Didn't investigate howevert. > Right; so maybe an additional (reliable) socket for only these > request/response types ? I think having two different types will complicate a lot. > > > > > @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > > > > > > > cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); > > > > > DEBUG("umad2sim_read: got %d...\n", cnt); > > > > > - if (cnt < sizeof(req)) { > > > > > - ERROR("umad2sim_read: partial request - skip.\n"); > > > > > - umad->status = EAGAIN; > > > > > - return umad_size(); > > > > > - } > > > > > > > > > > mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); > > > > > > > > This checked that we got at least req header before decode this. > > > > > > No; I think it checked that the entire packet existed as req was the > > > entire packet not just the header. > > > > > > > Why is it removed? > > > > > > Because of short RMPP. It can add back at least the header check part > > > but I think the partial write (long writes) is the fundamental issue. > > > > Yes, some check is needed. And above length encoded in sim_request header > > can be useful. > > A header check can be left there. It's insufficient IMO though. If length is encoded in header you can check whole packet size. > I guess at this point I'm unclear about moving forward. The only clear > thing to me is that with long RMPP it might be accepted. With short RMPP too - it is better than nothing anyway :). I just think that we need to avoid partial read/write bugs (maybe by using packet or payload length in sim_request header). Sasha From eli at mellanox.co.il Mon Jun 16 07:58:30 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 16 Jun 2008 17:58:30 +0300 Subject: [ofa-general] [RFC] Receive Core Afinnity (aka RSS) support Message-ID: <1213628310.5447.91.camel@mtls03> I plan to implement RCA for IPoIB and would like your comments. Using RCA allows better utilization of available CPU cores power by evenly distributing received SKBs between a few receive queues. Since each such queue is connected to a different CQ, which in turn is connected to an EQ which triggers a distinct IRQ vector, we can have more power to process received buffers. The implementation involves extending the ib core API, make modifications in mlx4 and implementing in IPoIB. The planned work relies on VPI patches sent to the general list for review to provide means for allocating a group of QPs with consecutive QP numbers and alignment requirements. The patch bellow describes some of the required changes to the IB core. diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 4c3c76b..4ca946c 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -104,6 +104,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_MEM_MGT_EXTENSIONS = (1<<21), + IB_DEVICE_IPOIB_RCA = (1<<22), }; enum ib_atomic_cap { @@ -505,6 +506,7 @@ enum ib_qp_type { enum ib_qp_create_flags { IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + IB_QP_CREATE_IPOIB_RCA = 1 << 1, }; struct ib_qp_init_attr { @@ -989,6 +991,9 @@ struct ib_device { int (*post_srq_recv)(struct ib_srq *srq, struct ib_recv_wr *recv_wr, struct ib_recv_wr **bad_recv_wr); + int (*create_qp_range)(struct ib_pd *pd, + struct ib_qp_init_attr *qp_init_attr, + int nqps, int align, struct ib_qp *list[]); struct ib_qp * (*create_qp)(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, struct ib_udata *udata); @@ -1318,6 +1323,20 @@ static inline int ib_post_srq_recv(struct ib_srq *srq, } /** + * ib_create_qp_range - Creates a range of QPs associated with the + * specified protection domain. + * @pd: The protection domain associated with the QP. + * @qp_init_attr: A list of initial attributes required to create the + * QPs. If QP creation succeeds, then the attributes are updated to + * the actual capabilities of the created QP. + * @nqps: The number of required QPs + * @align: Required alignment for the allocated QP numbers. + * @list: pointer to an array to hold allocated QPs + */ +int ib_create_qp_range(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, + int nqps, int align, struct ib_qp *list[]); + +/** * ib_create_qp - Creates a QP associated with the specified protection * domain. * @pd: The protection domain associated with the QP. In IPoIB support will be added only for UD mode. For sending we will use the same QP used today. For receive we will create a an RCA QP (by using IB_QP_CREATE_IPOIB_RCA in the create flags) and a group of QPs. The base QPN, number of QPs and a default QPN have to be passed to the RCA QP -- I am not sure what is the best way to do that but I am thinking of passing these values with modify QP to init using new fields in struct ib_qp_attr. The default QPN will be the current QP used for UD. We will have multiple receive rings (as the number of QPs) so we don't need to add spinlocks. -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From eli at mellanox.co.il Mon Jun 16 07:59:34 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Mon, 16 Jun 2008 17:59:34 +0300 Subject: [ofa-general] [PATCH] libmlx4: Optimize QP stamping Message-ID: <1213628374.5447.93.camel@mtls03> >From d377b7289a192670dc670663253df8ac2da0bc96 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Mon, 2 Jun 2008 11:20:14 +0300 Subject: [PATCH] libmlx4: Optimize QP stamping Optimize samping by reading the value of the DS field just before we stamp, which would give the effective size of the descriptor as used in the previous post and. Then we stamp only that area, since the rest of the descriptor is already stamped. Signed-off-by: Eli Cohen --- src/qp.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/src/qp.c b/src/qp.c index 4322513..f3cf839 100644 --- a/src/qp.c +++ b/src/qp.c @@ -74,8 +74,9 @@ static void stamp_send_wqe(struct mlx4_qp *qp, int n) { uint32_t *wqe = get_send_wqe(qp, n); int i; + int ds = (((struct mlx4_wqe_ctrl_seg *)wqe)->fence_size & 0x3f) << 2; - for (i = 16; i < 1 << (qp->sq.wqe_shift - 2); i += 16) + for (i = 16; i < ds; i += 16) wqe[i] = 0xffffffff; } @@ -95,6 +96,7 @@ void mlx4_qp_init_sq_ownership(struct mlx4_qp *qp) for (i = 0; i < qp->sq.wqe_cnt; ++i) { ctrl = get_send_wqe(qp, i); ctrl->owner_opcode = htonl(1 << 31); + ctrl->fence_size = 1 << (qp->sq.wqe_shift - 4); stamp_send_wqe(qp, i); } -- 1.5.5.1 -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From hrosenstock at xsigo.com Mon Jun 16 08:07:04 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 08:07:04 -0700 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <20080616145429.GD13363@sashak.voltaire.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> <20080616145429.GD13363@sashak.voltaire.com> Message-ID: <1213628824.14047.674.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 17:54 +0300, Sasha Khapyorsky wrote: > On 06:45 Mon 16 Jun , Hal Rosenstock wrote: > > > > > For this we even don't need to break ABI - there is unused > > > 64-bit context field. > > > > > > > The socket is non-blocked datagram. > > > > > > > > Is a framing layer needed ? > > > > > > I don't know. In my old implementation I switched to STREAM socket in > > > order to not reinvent a wheel, > > > > but overall performance suffered, right ? > > Yes, but mainly it was with huge 100M-1G RMPP packets - just fing this > not very useful. Didn't investigate howevert. > > > Right; so maybe an additional (reliable) socket for only these > > request/response types ? > > I think having two different types will complicate a lot. I thought it might too and didn't see how to cleanly do this. > > > > > > @@ -385,11 +386,6 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > > > > > > > > > > > > cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); > > > > > > DEBUG("umad2sim_read: got %d...\n", cnt); > > > > > > - if (cnt < sizeof(req)) { > > > > > > - ERROR("umad2sim_read: partial request - skip.\n"); > > > > > > - umad->status = EAGAIN; > > > > > > - return umad_size(); > > > > > > - } > > > > > > > > > > > > mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); > > > > > > > > > > This checked that we got at least req header before decode this. > > > > > > > > No; I think it checked that the entire packet existed as req was the > > > > entire packet not just the header. > > > > > > > > > Why is it removed? > > > > > > > > Because of short RMPP. It can add back at least the header check part > > > > but I think the partial write (long writes) is the fundamental issue. > > > > > > Yes, some check is needed. And above length encoded in sim_request header > > > can be useful. > > > > A header check can be left there. It's insufficient IMO though. > > If length is encoded in header you can check whole packet size. It's only a partial measure. It doesn't account for dropped or duplicated packets. It's UDP, right ? > > I guess at this point I'm unclear about moving forward. The only clear > > thing to me is that with long RMPP it might be accepted. > > With short RMPP too - it is better than nothing anyway :). I just think > that we need to avoid partial read/write bugs (maybe by using packet or > payload length in sim_request header). OK; that's easy to add for short RMPP and things won't be any worse than they are now as the issues aren't likely to show there. Long RMPP is a whole 'nother matter... -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Mon Jun 16 08:09:42 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 08:09:42 -0700 Subject: [ofa-general] Re: [ewg] OFED meeting agenda for today (June 16, 2008) In-Reply-To: <6C2C79E72C305246B504CBA17B5500C90440EF3B@mtlexch01.mtl.com> References: <6C2C79E72C305246B504CBA17B5500C90440EF3B@mtlexch01.mtl.com> Message-ID: <1213628982.14047.677.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 16:59 +0300, Tziporet Koren wrote: > My proposal is to do the feature freeze closer to 2.6 27 - at middle > of July What does the schedule/projected end date then look like ? Does this impact any deliverables to SC or Plugfest/interop events ? -- Hal From sashak at voltaire.com Mon Jun 16 08:07:59 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 18:07:59 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616150759.GE13363@sashak.voltaire.com> On 07:48 Mon 16 Jun , Hal Rosenstock wrote: > > > > I would think so, but didn't check deeply. And in fact actual size of > > such array can be limited by max supported number of MLIDs in a fabric > > Is that really a good idea to limit it in this way ? Guess it could be > limited with some configurable option but even so if a device with a > larger MFT comes along then the SM might not work. I meant something else - any multicast stuff will be processed in SM after initial subnet discovery, so it is not a big problem to detect max needed array size in run-time. OTOH 16K for MLIDs is not a so big deal, I would not bother with such optimization unless would explicitly need this. Sasha From sashak at voltaire.com Mon Jun 16 08:11:00 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 18:11:00 +0300 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <1213628824.14047.674.camel@hrosenstock-ws.xsigo.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> <20080616145429.GD13363@sashak.voltaire.com> <1213628824.14047.674.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616151100.GF13363@sashak.voltaire.com> On 08:07 Mon 16 Jun , Hal Rosenstock wrote: > > > > If length is encoded in header you can check whole packet size. > > It's only a partial measure. It doesn't account for dropped or > duplicated packets. It's UDP, right ? Yes in inet mode and unix datagram when locally used. Sasha From sashak at voltaire.com Mon Jun 16 08:16:03 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 18:16:03 +0300 Subject: [ofa-general] Re: [PATCHv3][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements In-Reply-To: <1213621882.14047.611.camel@hrosenstock-ws.xsigo.com> References: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> <20080615063727.GK24365@sashak.voltaire.com> <1213621882.14047.611.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616151603.GG13363@sashak.voltaire.com> On 06:11 Mon 16 Jun , Hal Rosenstock wrote: > > > @@ -1278,8 +1281,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, > > > } > > > } else { > > > CL_PLOCK_RELEASE(sa->p_lock); > > > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > > - "Failed since multicast group not present\n"); > > > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " > > > + "Failed since multicast group 0x%16" > > > > Why to change log level? It is not OpenSM error, but invalid request. > > Are errors really restricted to OpenSM (internal) errors ? I guess it should be. > I thought > they were more inclusive than that including network events that were > "interesting events". We have VERBOSE level for "interesting" events and INFO for "very interesting" ones. > In this specific case, I agree it is a bad idea as I think restarting > IPoIB does causes this code path to be used. Modified patch already > issued. Cool. Thanks. Sasha From tziporet at dev.mellanox.co.il Mon Jun 16 08:20:23 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Mon, 16 Jun 2008 18:20:23 +0300 Subject: [ofa-general] Re: [ewg] OFED meeting agenda for today (June 16, 2008) In-Reply-To: <1213628982.14047.677.camel@hrosenstock-ws.xsigo.com> References: <6C2C79E72C305246B504CBA17B5500C90440EF3B@mtlexch01.mtl.com> <1213628982.14047.677.camel@hrosenstock-ws.xsigo.com> Message-ID: <485684B7.6080308@mellanox.co.il> Hal Rosenstock wrote: > On Mon, 2008-06-16 at 16:59 +0300, Tziporet Koren wrote: > >> My proposal is to do the feature freeze closer to 2.6 27 - at middle >> of July >> > > What does the schedule/projected end date then look like ? Does this > impact any deliverables to SC or Plugfest/interop events ? > > -- Hal > > > I think no The schedule should be: Ø * Alpha: Jul 15, 08 - feature freeze * Beta: Jul 30, 08 - kernel 2.6.27-rcX (assuming it will be available) * RC1: Aug 13, 08 * RC2: Aug 27, 08 * RC3-RC5/6 – every 5-10 days o Latest RC to be used in OFA interop event * GA: Oct 06 08 Tziporet From hrosenstock at xsigo.com Mon Jun 16 08:24:30 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 08:24:30 -0700 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <20080616151100.GF13363@sashak.voltaire.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> <20080616145429.GD13363@sashak.voltaire.com> <1213628824.14047.674.camel@hrosenstock-ws.xsigo.com> <20080616151100.GF13363@sashak.voltaire.com> Message-ID: <1213629870.14047.681.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 18:11 +0300, Sasha Khapyorsky wrote: > On 08:07 Mon 16 Jun , Hal Rosenstock wrote: > > > > > > If length is encoded in header you can check whole packet size. > > > > It's only a partial measure. It doesn't account for dropped or > > duplicated packets. It's UDP, right ? > > Yes in inet mode and unix datagram when locally used. So then even a length in the header is a coarse comparator for whether the receive is correct or not. If there's a reasonable probability that no errors occur (within the current retry strategy) then this might be OK but there is also the issue of resync'ing when something is dropped to make sure one is really starting on a header boundary. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Mon Jun 16 08:26:53 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 08:26:53 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080616150759.GE13363@sashak.voltaire.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> Message-ID: <1213630013.14047.685.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 18:07 +0300, Sasha Khapyorsky wrote: > On 07:48 Mon 16 Jun , Hal Rosenstock wrote: > > > > > > I would think so, but didn't check deeply. And in fact actual size of > > > such array can be limited by max supported number of MLIDs in a fabric > > > > Is that really a good idea to limit it in this way ? Guess it could be > > limited with some configurable option but even so if a device with a > > larger MFT comes along then the SM might not work. > > I meant something else - any multicast stuff will be processed in SM > after initial subnet discovery, so it is not a big problem to detect > max needed array size in run-time. Isn't a similar thing done in terms of unicast ? > OTOH 16K for MLIDs is not a so big deal, I would not bother with such optimization > unless would explicitly need this. OK with me; I would think such an optimization is a minor change anyhow. -- Hal > Sasha From hrosenstock at xsigo.com Mon Jun 16 08:33:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 08:33:29 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080616150759.GE13363@sashak.voltaire.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> Message-ID: <1213630409.14047.688.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 18:07 +0300, Sasha Khapyorsky wrote: > I meant something else - any multicast stuff will be processed in SM > after initial subnet discovery, so it is not a big problem to detect > max needed array size in run-time. This already exists: max_multicast_lid_ho -- Hal From chu11 at llnl.gov Mon Jun 16 08:52:32 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 16 Jun 2008 08:52:32 -0700 Subject: [ofa-general] Re: [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <20080616082415.GK24365@sashak.voltaire.com> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <20080616082415.GK24365@sashak.voltaire.com> Message-ID: <1213631552.5369.382.camel@cardanus.llnl.gov> Hey Sasha, On Mon, 2008-06-16 at 11:24 +0300, Sasha Khapyorsky wrote: > Hi Al, > > On 15:48 Fri 13 Jun , Al Chu wrote: > > > > This is a conceptually simple option I've developed for updn routing. > > > > Currently in updn routing, nodes/guids are routed on switches in a > > seemingly-random order, which I believe is due to internal data > > structure organization (i.e. cl_qmap_apply_func is called on > > port_guid_tbl) as well as how the fabric is scanned (it is logically > > scanned from a port perspective, but it may not be logical from a node > > perspective). I had a hypothesis that this was leading to increased > > contention in the network for MPI. > > > > For example, suppose we have 12 uplinks from a leaf switch to a spine > > switch. If we want to send data from this leaf switch to node[13-24], > > the up links we will send on are pretty random. > > Yeah, the issue is known. And idea is good and useful. > > Actually we discussed this issue with Yiftah some time ago and his idea > was to have an option to order routing generation by ports connected to > leaf switches with higher number of active links. > > Something like: > > foreach switch reverse (higher is first) sorted by number of active links > foreach port connected to the switch > do_rounting() What do you mean by "foreach switch reverse"? > Which is good for most cases, but also I thought that in addition we will > need something more configurable - just like your patch series :). Cool. > Now comment: why do you think it is useful only for Up/Down? At first, I imagined this patch only useful for networks with tree like configurations (thus people using updn). But now that you mention it, it is probably useful for many configurations and people that might use min-hop. > IMO min hops algo could benefit from this feature just well. If so this > simplifies implementation - patches 2,3,4 are not needed, code from patch > 5 is going to osm_ucast_mgr.c. What do you think? Sounds like a plan. I will adjust when I adjust for the preserve-base- lids patch too. Al > Sasha -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From sashak at voltaire.com Mon Jun 16 08:51:55 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 18:51:55 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1213630013.14047.685.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> <1213630013.14047.685.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616155155.GH13363@sashak.voltaire.com> On 08:26 Mon 16 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-16 at 18:07 +0300, Sasha Khapyorsky wrote: > > On 07:48 Mon 16 Jun , Hal Rosenstock wrote: > > > > > > > > I would think so, but didn't check deeply. And in fact actual size of > > > > such array can be limited by max supported number of MLIDs in a fabric > > > > > > Is that really a good idea to limit it in this way ? Guess it could be > > > limited with some configurable option but even so if a device with a > > > larger MFT comes along then the SM might not work. > > > > I meant something else - any multicast stuff will be processed in SM > > after initial subnet discovery, so it is not a big problem to detect > > max needed array size in run-time. > > Isn't a similar thing done in terms of unicast ? Yes, with cl_vector*() and it is *slow*. Sasha From DMowry at klondykeinc.com Mon Jun 16 08:54:52 2008 From: DMowry at klondykeinc.com (Don Mowry) Date: Mon, 16 Jun 2008 08:54:52 -0700 Subject: [ofa-general] Attention :$$: Do You Need An Urgent Loan :$$: { Apply Message-ID: <7AF7CD7AE08F9749AD664BD48D1567F7350FF9@SBSERVER.ki.local> Please contact me at 602-376-1623 Thank you Don Mowry -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Mon Jun 16 08:58:15 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 18:58:15 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1213630409.14047.688.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> <1213630409.14047.688.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616155815.GI13363@sashak.voltaire.com> On 08:33 Mon 16 Jun , Hal Rosenstock wrote: > > This already exists: max_multicast_lid_ho Right. Sasha From chu11 at llnl.gov Mon Jun 16 09:04:05 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 16 Jun 2008 09:04:05 -0700 Subject: [ofa-general] Re: [OPENSM PATCH 1/5]: add guid_routing_order_file option In-Reply-To: <20080616082628.GL24365@sashak.voltaire.com> References: <1213397327.5369.357.camel@cardanus.llnl.gov> <20080616082628.GL24365@sashak.voltaire.com> Message-ID: <1213632245.5369.394.camel@cardanus.llnl.gov> On Mon, 2008-06-16 at 11:26 +0300, Sasha Khapyorsky wrote: > Hi Al, > > On 15:48 Fri 13 Jun , Al Chu wrote: > > @@ -1547,6 +1551,10 @@ int osm_subn_write_conf_file(char *file_name, IN osm_subn_opt_t *const p_opts) > > " Up/Down algorithm instead\n# of GUIDs (one guid and" > > " id in each line)\nids_guid_file %s\n\n", > > p_opts->ids_guid_file ? p_opts->ids_guid_file : null_str); > > + > > + fprintf(opts_file, > > + "# The file holding guid routing order guids (for Up/Down)\n" > > + "guid_routing_order_file %s\n\n", p_opts->guid_routing_order_file); > > I guess here should be: > > p_opts->guid_routing_order_file ? p_opts->guid_routing_order_file : null_str Oops. Thanks. Al > Sasha -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From hrosenstock at xsigo.com Mon Jun 16 09:06:35 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 09:06:35 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080616155155.GH13363@sashak.voltaire.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> <1213630013.14047.685.camel@hrosenstock-ws.xsigo.com> <20080616155155.GH13363@sashak.voltaire.com> Message-ID: <1213632395.14047.693.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 18:51 +0300, Sasha Khapyorsky wrote: > On 08:26 Mon 16 Jun , Hal Rosenstock wrote: > > On Mon, 2008-06-16 at 18:07 +0300, Sasha Khapyorsky wrote: > > > On 07:48 Mon 16 Jun , Hal Rosenstock wrote: > > > > > > > > > > I would think so, but didn't check deeply. And in fact actual size of > > > > > such array can be limited by max supported number of MLIDs in a fabric > > > > > > > > Is that really a good idea to limit it in this way ? Guess it could be > > > > limited with some configurable option but even so if a device with a > > > > larger MFT comes along then the SM might not work. > > > > > > I meant something else - any multicast stuff will be processed in SM > > > after initial subnet discovery, so it is not a big problem to detect > > > max needed array size in run-time. > > > > Isn't a similar thing done in terms of unicast ? > > Yes, with cl_vector*() and it is *slow*. If 16K array of pointers is OK, is a 48K array ? If so, then this should be straightforward to change. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Mon Jun 16 09:05:46 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 19:05:46 +0300 Subject: [ofa-general] Re: [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <1213631552.5369.382.camel@cardanus.llnl.gov> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <20080616082415.GK24365@sashak.voltaire.com> <1213631552.5369.382.camel@cardanus.llnl.gov> Message-ID: <20080616160546.GJ13363@sashak.voltaire.com> On 08:52 Mon 16 Jun , Al Chu wrote: > > > > Something like: > > > > foreach switch reverse (higher is first) sorted by number of active links > > foreach port connected to the switch > > do_rounting() > > What do you mean by "foreach switch reverse"? "reverse sorted". So first routed ports will be ones connected to leaf switch with highest number of active links. > > IMO min hops algo could benefit from this feature just well. If so this > > simplifies implementation - patches 2,3,4 are not needed, code from patch > > 5 is going to osm_ucast_mgr.c. What do you think? > > Sounds like a plan. I will adjust when I adjust for the preserve-base- > lids patch too. Cool. Thanks. Sasha From vlad at dev.mellanox.co.il Mon Jun 16 09:08:33 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Mon, 16 Jun 2008 19:08:33 +0300 Subject: [ofa-general] cxgb3_0010_napi.patch fails to be applied Message-ID: <48569001.3070304@dev.mellanox.co.il> Hello Steve, I merged ofed-1.4 kernel tree with 2.6.26-rc6 kernel. Now ofed_makedist.sh fails: Failed executing /local/scm/ofed-1.4/linux-2.6/ofed_scripts/ofed_patch.sh --with-backport=2.6.16 >> /tmp/build-ofed_kernel-R21111/ofed_kernel-2.6.16.log Hunk #15 succeeded at 2461 (offset 25 lines). Hunk #16 succeeded at 2571 (offset 25 lines). Hunk #17 succeeded at 2583 (offset 25 lines). Hunk #18 succeeded at 2619 (offset 25 lines). Hunk #19 succeeded at 2644 (offset 25 lines). Hunk #20 succeeded at 2765 (offset 25 lines). 1 out of 20 hunks FAILED -- rejects in file drivers/net/cxgb3/sge.c Patch cxgb3_0010_napi.patch does not apply (enforce with -f) Please fix this issue and add required backports. git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Regards, Vladimir From sashak at voltaire.com Mon Jun 16 09:08:58 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 19:08:58 +0300 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <1213632395.14047.693.camel@hrosenstock-ws.xsigo.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> <1213630013.14047.685.camel@hrosenstock-ws.xsigo.com> <20080616155155.GH13363@sashak.voltaire.com> <1213632395.14047.693.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616160858.GK13363@sashak.voltaire.com> On 09:06 Mon 16 Jun , Hal Rosenstock wrote: > > If 16K array of pointers is OK, is a 48K array ? If so, then this should > be straightforward to change. Looks like a good idea for me. Sasha From chu11 at llnl.gov Mon Jun 16 09:16:43 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 16 Jun 2008 09:16:43 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <4856798A.8020006@dev.mellanox.co.il> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> Message-ID: <1213633003.5369.406.camel@cardanus.llnl.gov> On Mon, 2008-06-16 at 17:32 +0300, Yevgeny Kliteynik wrote: > Jeff, > > Jeff Becker wrote: > > Hi Al > > > > Al Chu wrote: > >> Hey Jeff, > >> > >> > >>> That works. The compute nodes need to talk to other compute nodes for > >>> MPI over one set of links, and they need to talk to the Lustre nodes > >>> for I/O, but over a different (disjoint) set of links. Thanks. > >>> > >> > >> Is there a strong belief that a different/disjoint set of links would be > >> beneficial? Sometime ago, Sasha and I iterated on a patch in which I > >> found out sometimes not all switch ports would be used. In this > >> particular case, a chunk of leaf switches were sometimes using only 11 > >> out of 12 uplinks. After the fix, mpigraph showed about 20% improvement > >> in MPI bandwidth. > >> > > Basically, we want to avoid situations where I/O and MPI contend for the > > same links, and get in each other's way. > > What about using different VLs for MPI and I/O? Adam Moody ran this idea by me sometime ago too and was something I thought of looking into later. (We are analyzing/dealing w/ routing first :-). I have no idea if different service levels can be configured into MPI implementations. I asked the Lustre people in my hallway, and it isn't currently configurable for Lustre. This isn't to say it's not doable, but would take some effort. Al > It won't buy more bandwidth, but it might prevent MPI and I/O from > congesting each other - they will share the wire according to the > priority that you will define. > > -- Yevgeny > > > -jeff -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From sashak at voltaire.com Mon Jun 16 09:17:09 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 19:17:09 +0300 Subject: [ofa-general] Re: [PATCHv4][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements In-Reply-To: <1213621671.14047.606.camel@hrosenstock-ws.xsigo.com> References: <1213621671.14047.606.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616161709.GL13363@sashak.voltaire.com> On 06:07 Mon 16 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: Some error message improvements > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hrosenstock at xsigo.com Mon Jun 16 09:23:21 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 09:23:21 -0700 Subject: [ofa-general] Re: [RFC] OpenSM and IPv6 Scalability Proposal In-Reply-To: <20080616160858.GK13363@sashak.voltaire.com> References: <1212843797.6127.242.camel@hrosenstock-ws.xsigo.com> <20080616063905.GD24365@sashak.voltaire.com> <1213624871.14047.643.camel@hrosenstock-ws.xsigo.com> <20080616144203.GC13363@sashak.voltaire.com> <1213627683.14047.658.camel@hrosenstock-ws.xsigo.com> <20080616150759.GE13363@sashak.voltaire.com> <1213630013.14047.685.camel@hrosenstock-ws.xsigo.com> <20080616155155.GH13363@sashak.voltaire.com> <1213632395.14047.693.camel@hrosenstock-ws.xsigo.com> <20080616160858.GK13363@sashak.voltaire.com> Message-ID: <1213633401.14047.700.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 19:08 +0300, Sasha Khapyorsky wrote: > On 09:06 Mon 16 Jun , Hal Rosenstock wrote: > > > > If 16K array of pointers is OK, is a 48K array ? If so, then this should > > be straightforward to change. > > Looks like a good idea for me. It would need to be based on min of the switch LFT sizes rather than max LID in use/persistent as it is now. Also, max_unicast_lid_ho similar to max_multicast_lid_ho exists and is supported now so the actual max optimization could be applied although from a practical standpoint this doesn't have any benefit in most if not all subnets (whereas the multicast arch max is not currently implemented). -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Mon Jun 16 09:21:36 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Jun 2008 19:21:36 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_qos_parser.y: TRIVIAL addition in log message In-Reply-To: <48566718.80909@dev.mellanox.co.il> References: <48566718.80909@dev.mellanox.co.il> Message-ID: <20080616162136.GM13363@sashak.voltaire.com> On 16:14 Mon 16 Jun , Yevgeny Kliteynik wrote: > Hi Sasha, > > Adding new line in one log (error) message. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. Sasha From sean.hefty at intel.com Mon Jun 16 09:24:47 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 16 Jun 2008 09:24:47 -0700 Subject: [ofa-general] RE: [PATCH RFC] librdmacm: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: <000001c8cfcd$7ebe5fb0$9d37170a@amr.corp.intel.com> >Assuming Jeff accepts them into his tree, does it makes sense to you to pull >his tree into yours and then push the patch sequence to Roland, or you >want me to push it to Roland and you will sign/ack the patches? We are now >at -rc6 and I want to bring this to the for-2.6.27 branch of Roland's this >week. I think this is Roland's call on whatever is best/easiest for him. But I'll review the patches and provide a reviewed-by line. - Sean From hrosenstock at xsigo.com Mon Jun 16 09:25:48 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 09:25:48 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213633003.5369.406.camel@cardanus.llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> Message-ID: <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > I asked the Lustre people in my hallway, and it isn't > currently configurable for Lustre. Wouldn't Lustre SL be inherited from partition based on underlying IPoIB interface ? -- Hal From weiny2 at llnl.gov Mon Jun 16 09:35:15 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 16 Jun 2008 09:35:15 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616093515.7f4f61cc.weiny2@llnl.gov> On Mon, 16 Jun 2008 09:25:48 -0700 Hal Rosenstock wrote: > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > I asked the Lustre people in my hallway, and it isn't > > currently configurable for Lustre. > > Wouldn't Lustre SL be inherited from partition based on underlying IPoIB > interface ? I am not quite sure what you mean here? Our Lustre sets up their own QP's via the RDMACM. So I believe we could set our SL and/or partition for those QP's separately from IPoIB via a modify_qp call; right? Ira > > -- Hal > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Mon Jun 16 09:38:58 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 09:38:58 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <20080616093515.7f4f61cc.weiny2@llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> Message-ID: <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: > On Mon, 16 Jun 2008 09:25:48 -0700 > Hal Rosenstock wrote: > > > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > > I asked the Lustre people in my hallway, and it isn't > > > currently configurable for Lustre. > > > > Wouldn't Lustre SL be inherited from partition based on underlying IPoIB > > interface ? > > I am not quite sure what you mean here? Our Lustre sets up their own QP's via > the RDMACM. So I believe we could set our SL and/or partition for those QP's > separately from IPoIB via a modify_qp call; right? RDMA CM does address resolution based on IP addresses and an SL can be associated with the outgoing IPoIB interface. -- Hal > > Ira > > > > > -- Hal > > > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sean.hefty at intel.com Mon Jun 16 09:40:40 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 16 Jun 2008 09:40:40 -0700 Subject: [ofa-general] RE: [PATCH RFC] librdmacm: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: <000201c8cfcf$b6759c00$9d37170a@amr.corp.intel.com> I've pulled this patch into my internal librdmacm git tree. It should make it into the next release, which will correspond with 2.6.27, assuming the kernel patches make it in. - Sean From weiny2 at llnl.gov Mon Jun 16 09:46:29 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 16 Jun 2008 09:46:29 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616094629.557bf892.weiny2@llnl.gov> On Mon, 16 Jun 2008 09:38:58 -0700 Hal Rosenstock wrote: > On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: > > On Mon, 16 Jun 2008 09:25:48 -0700 > > Hal Rosenstock wrote: > > > > > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > > > I asked the Lustre people in my hallway, and it isn't > > > > currently configurable for Lustre. > > > > > > Wouldn't Lustre SL be inherited from partition based on underlying IPoIB > > > interface ? > > > > I am not quite sure what you mean here? Our Lustre sets up their own QP's via > > the RDMACM. So I believe we could set our SL and/or partition for those QP's > > separately from IPoIB via a modify_qp call; right? > > RDMA CM does address resolution based on IP addresses and an SL can be > associated with the outgoing IPoIB interface. > Right, but does it _have_ to be the associated? I thought not. Ira From hrosenstock at xsigo.com Mon Jun 16 09:47:51 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 09:47:51 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <20080616094629.557bf892.weiny2@llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> <20080616094629.557bf892.weiny2@llnl.gov> Message-ID: <1213634871.14047.708.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 09:46 -0700, Ira Weiny wrote: > On Mon, 16 Jun 2008 09:38:58 -0700 > Hal Rosenstock wrote: > > > On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: > > > On Mon, 16 Jun 2008 09:25:48 -0700 > > > Hal Rosenstock wrote: > > > > > > > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > > > > I asked the Lustre people in my hallway, and it isn't > > > > > currently configurable for Lustre. > > > > > > > > Wouldn't Lustre SL be inherited from partition based on underlying IPoIB > > > > interface ? > > > > > > I am not quite sure what you mean here? Our Lustre sets up their own QP's via > > > the RDMACM. So I believe we could set our SL and/or partition for those QP's > > > separately from IPoIB via a modify_qp call; right? > > > > RDMA CM does address resolution based on IP addresses and an SL can be > > associated with the outgoing IPoIB interface. > > > > Right, but does it _have_ to be the associated? I thought not. Do you want a different SL from that ? There's a QoS syntax but I'm not sure how Lustre plays into that. -- Hal > Ira > From weiny2 at llnl.gov Mon Jun 16 09:58:40 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 16 Jun 2008 09:58:40 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <1213634871.14047.708.camel@hrosenstock-ws.xsigo.com> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> <20080616094629.557bf892.weiny2@llnl.gov> <1213634871.14047.708.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080616095840.37c7e17b.weiny2@llnl.gov> On Mon, 16 Jun 2008 09:47:51 -0700 Hal Rosenstock wrote: > On Mon, 2008-06-16 at 09:46 -0700, Ira Weiny wrote: > > On Mon, 16 Jun 2008 09:38:58 -0700 > > Hal Rosenstock wrote: > > > > > On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: > > > > On Mon, 16 Jun 2008 09:25:48 -0700 > > > > Hal Rosenstock wrote: > > > > > > > > > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > > > > > I asked the Lustre people in my hallway, and it isn't > > > > > > currently configurable for Lustre. > > > > > > > > > > Wouldn't Lustre SL be inherited from partition based on underlying IPoIB > > > > > interface ? > > > > > > > > I am not quite sure what you mean here? Our Lustre sets up their own QP's via > > > > the RDMACM. So I believe we could set our SL and/or partition for those QP's > > > > separately from IPoIB via a modify_qp call; right? > > > > > > RDMA CM does address resolution based on IP addresses and an SL can be > > > associated with the outgoing IPoIB interface. > > > > > > > Right, but does it _have_ to be the associated? I thought not. > > Do you want a different SL from that ? > Maybe, some MPI's may use the RDMACM as well (I think some already do). Therefore if you want Lustre and MPI to be on different SL's at least one of them will have to change from the "inherited" IPoIB SL. > There's a QoS syntax but I'm not > sure how Lustre plays into that. Just to be clear this is only a "thought experiment" at this point. We have not tried to do any of this for real, yet. ;-) We realized there might be many changes to various configurations and codes which may need to be done. But knowing that I/O is less dependent on latency than MPI it seems to follow that overall system performance could benefit from having MPI run at a higher priority than Lustre/NFS etc. Ira From chu11 at llnl.gov Mon Jun 16 10:21:21 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 16 Jun 2008 10:21:21 -0700 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <4854CFFC.7040508@dev.mellanox.co.il> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <4854CFFC.7040508@dev.mellanox.co.il> Message-ID: <1213636881.5369.438.camel@cardanus.llnl.gov> Hey Yevgeny, On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: > Hi Al, > > Al Chu wrote: > > Hey Sasha, > > > > This is a conceptually simple option I've developed for updn routing. > > > > Currently in updn routing, nodes/guids are routed on switches in a > > seemingly-random order, which I believe is due to internal data > > structure organization (i.e. cl_qmap_apply_func is called on > > port_guid_tbl) as well as how the fabric is scanned (it is logically > > scanned from a port perspective, but it may not be logical from a node > > perspective). I had a hypothesis that this was leading to increased > > contention in the network for MPI. > > > > For example, suppose we have 12 uplinks from a leaf switch to a spine > > switch. If we want to send data from this leaf switch to node[13-24], > > the up links we will send on are pretty random. It's because: > > > > A) node[13-24] are individually routed at seemingly-random points based > > on when they are called by cl_qmap_apply_func(). > > > > B) the ports chosen for routing are based on least used port usage. > > > > C) least used port usage is based on whatever was routed earlier on. > > > > So I developed this patch series, which supports an option called > > "guid_routing_order_file" which allows the user to input a file with a > > list of port_guids which will indicate the order in which guids are > > routed instead (naturally, those guids not listed are routed last). > > Great idea! Thanks. > I understand that this guid_routing_order_file is synchronized with > an MPI rank file, right? If not, then synchronizing them might give > even better results. Not quite sure what you mean by a MPI rank file. At LLNL, slurm is responsible for MPI ranks, so I order the guids in my file according to how slurm is configured for chosing MPI ranks. I will admit to being a novice to MPI's configuration (blindly accepting slurm MPI rankings). Is there an underlying file that MPI libs use for ranking knowledge? > Another idea: OpenSM can create such file (list, doesn't have to be > actual file) automatically, just by checking topologically-adjacent > leaf switches and their HCAs. Definitely a good idea. This patch set was just a "step one" kind of thing. > > > I list the port guids of the nodes of the cluster from node0 to nodeN, one > > per line in the file. By listing the nodes in this order, I believe we > > could get less contention in the network. In the example above, sending > > to node[13-24] should use all of the 12 uplinks, b/c the ports will be > > equally used b/c nodes[1-12] were routed beforehand in order. > > > > The results from some tests are pretty impressive when I do this. LMC=0 > > average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s > > when I use guid_routing_order. > > Can you compare this to the fat-tree routing? Conceptually, fat-tree > is doing the same - it routes LIDs on nodes in a topological order, so > it would be interesting to see the comparison. Actually I already did :-). w/ LMC=0. updn default - 391.374 MB/s updn w/ guid_routing_order - 573.678 MB/s ftree - 579.603 MB/s I later discovered that one of the internal ports of the cluster I'm testing on was broken (sLB of a 288 port), and think that is the cause of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as designed) seemed to be able to work around it properly, while updn (as currently implemented) couldn't. When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were able to do better on some tests than ftree. One example (I think these numbers are in microseconds. Lower is better): Alltoall 16K packets ftree - 415490.6919 updn normal (LMC=0) - 495460.5526 updn w/ ordered routing (LMC=0) - 416562.7417 updn w/ ordered routing (LMC=1) - 453153.7289 - this ^^^ result is quite odd. Not sure why. updn w/ ordered routing (LMC=2) - 3660132.1530 We are regularly debating what will be better overall at the end of the day. > Also, fat-tree produces the guid order file automatically, but nobody > used it yet as an input to produce MPI rank file. I didn't know about this option. How do you do this (just skimmed the manpage, didn't see anything)? I know about the --cn_guid_file. But since that file doesn't have to be ordered, that's why I created a different option (rather than have the cn_guid_file for both ftree and updn). Al > -- Yevgeny > > > A variety of other positive performance > > increases were found when doing other tests, other MPIs, and other LMCs > > if anyone is interested. > > > > BTW, I developed this patch series before your preserve-base-lid patch > > series. It will 100% conflict with the preserve-base-lid patch series. > > I will fix this patch series once the preserve-base-lids patch series is > > committed to git. I'm just looking for comments right now. > > > > Al > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From weiny2 at llnl.gov Mon Jun 16 10:24:30 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 16 Jun 2008 10:24:30 -0700 Subject: [ofa-general] Re: [PATCH v2] Add enum strings and *_str functions for enums In-Reply-To: <200806161705.41183.jackm@dev.mellanox.co.il> References: <20080415133548.414aeaea.weiny2@llnl.gov> <200806161705.41183.jackm@dev.mellanox.co.il> Message-ID: <20080616102430.505a2000.weiny2@llnl.gov> On Mon, 16 Jun 2008 17:05:40 +0300 Jack Morgenstein wrote: > On Saturday 19 April 2008 00:54, Roland Dreier wrote: > > Thanks, I added a man page and changed things a little and committed the > > following: > > > > commit 1c0b7ac0a6bbbe4d246ef4cf50ae31bde4929ba3 > > Author: Ira Weiny > > Date: Tue Apr 15 13:35:48 2008 -0700 > > > > Add functions to convert enum values to strings > > > > Add ibv_xxx_str() functions to convert node type, port state, event > > type and wc status enum values to strings. > > > > Signed-off-by: Ira K. Weiny > > Signed-off-by: Roland Dreier > > > > > The change below (in the output format of the port state string) > is causing us problems (with all sorts of scripts based upon ibv_devinfo). > This only surfaced now because we're only just now bringing up the OFED 1.4 tree > based upon the current libibverbs and kernel 2.6.26. > > Unfortunately, I missed this change when you posted it to the list in April. > > I assume that the only reason for the change below, aside from adding the defer state, was cosmetic. > Is it possible to change the output strings so that for previously defined port states, > the output remains what it was previously? Yes it was just cosmetic. I have no attachment to what the strings are, just that they are provided by the lib to be used by other ULP's such as MPI. > > (I don't see this as any different from other userspace library backwards-compatibility issues, > the change breaks existing scripts). > Sorry, I did not intend to break anything. I completely forgot that ibv_devinfo was a widely used tool. Patch included below Ira >From 5a624c6ee75d34dfa29580a71eb26c9f2a9990fb Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Mon, 16 Jun 2008 10:20:22 -0700 Subject: [PATCH] src/enum_strs.c: revert strings for port state to fix script problems. Signed-off-by: Ira K. Weiny --- src/enum_strs.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/enum_strs.c b/src/enum_strs.c index c57feaa..97f1081 100644 --- a/src/enum_strs.c +++ b/src/enum_strs.c @@ -51,15 +51,15 @@ const char *ibv_port_state_str(enum ibv_port_state port_state) { static const char *const port_state_str[] = { [IBV_PORT_NOP] = "no state change (NOP)", - [IBV_PORT_DOWN] = "down", - [IBV_PORT_INIT] = "init", - [IBV_PORT_ARMED] = "armed", - [IBV_PORT_ACTIVE] = "active", - [IBV_PORT_ACTIVE_DEFER] = "active defer" + [IBV_PORT_DOWN] = "PORT_DOWN", + [IBV_PORT_INIT] = "PORT_INIT", + [IBV_PORT_ARMED] = "PORT_ARMED", + [IBV_PORT_ACTIVE] = "PORT_ACTIVE", + [IBV_PORT_ACTIVE_DEFER] = "PORT_ACTIVE_DEFER" }; if (port_state < IBV_PORT_NOP || port_state > IBV_PORT_ACTIVE_DEFER) - return "unknown"; + return "invalid state"; return port_state_str[port_state]; } -- 1.5.1 From swise at opengridcomputing.com Mon Jun 16 10:56:10 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 16 Jun 2008 12:56:10 -0500 Subject: [ofa-general] [ANNOUNCE] chelsio libcxgb3 version 1.2.1 released Message-ID: <4856A93A.8010400@opengridcomputing.com> Chelsio iWARP fans, Version 1.2.1 of libcxgb3 is available at: http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.1.tar.gz This library is packaged along with ofed-1.3.1 and is required for 2.6.26 kernels and beyond. Also: The new ofed-1.3.1 / kernel.org 2.6.26 cxgb3 drivers require a firmware upgrade for the chelsio rnic. You can pull the firmware from: http://service.chelsio.com/drivers/firmware/t3/t3fw-6.0.0.bin.gz Unzip it and place it in /lib/firmware on your systems. Then the next time you reload and configure cxgb3 it will install the new firmware. Thanks, Steve. From chu11 at llnl.gov Mon Jun 16 10:56:52 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 16 Jun 2008 10:56:52 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <20080610025939.GK10301@sashak.voltaire.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> Message-ID: <1213639012.5369.443.camel@cardanus.llnl.gov> Hey Sasha, patch series seems to perform well. As a reminder mvapich 0.9.9 does not how to deal with multiple lids, openMPI 1.2.6 does. The "preserve lids" condition is labeled "PL". With LMC > 0, the preserve lids seems to do its job under mvapich 0.9.9, and atleast maintains (sometimes increases) performance of openmpi 1.2.6 with LMC > 0. Numbers are attached. I'd give my thumbs up to the patch. Lets commit it. Al On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: > Basically this addresses the problem described by Al Chu in: > > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html > > When base lid paths become completely disbalanced on a fabrics with > lmc > 0. > > One feedback was from Yiftah Shahar: > > "I think that our requirements should be that even when you are working > with LMC>0 then the base LID routing should not be affected. > One way to achieve this goal is to first run the base-LID routing (so > all base LID improvement will be also in LMC>0) and then start with the > other LIDs as round-robbing starting from the base-lid-port + 1 > according current routing algorithm rules (keeping min-hop, up/down...)." > > We had some discussion with Al and Yiftah about this and considered that > in addition to "pure" base lid paths preservation (which is good thing by > itself) proposed method solves original lid disbalancing problem as well. > > This patch is implementation of the idea above. > > Signed-off-by: Sasha Khapyorsky > --- > opensm/include/opensm/osm_switch.h | 4 + > opensm/opensm/osm_dump.c | 3 +- > opensm/opensm/osm_switch.c | 8 ++- > opensm/opensm/osm_ucast_mgr.c | 163 ++++++++++++++++-------------------- > 4 files changed, 86 insertions(+), 92 deletions(-) > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > index 0e9c5fa..c1521a6 100644 > --- a/opensm/include/opensm/osm_switch.h > +++ b/opensm/include/opensm/osm_switch.h > @@ -981,6 +981,7 @@ uint8_t > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > IN osm_port_t * p_port, > IN const uint16_t lid_ho, > + IN unsigned start_from, > IN const boolean_t ignore_existing, > IN const boolean_t dor); > /* > @@ -995,6 +996,9 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > * lid_ho > * [in] LID value (host order) for which to get a path advisory. > * > +* start_from > +* [in] Port number from where to start balance counting. > +* > * ignore_existing > * [in] Set to cause the switch to choose the optimal route > * regardless of existing paths. > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > index b96984b..60c6d25 100644 > --- a/opensm/opensm/osm_dump.c > +++ b/opensm/opensm/osm_dump.c > @@ -218,7 +218,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > else { > /* No LMC Optimization */ > best_port = osm_switch_recommend_path(p_sw, p_port, > - lid_ho, TRUE, dor); > + lid_ho, 1, TRUE, > + dor); > fprintf(file, "No %u hop path possible via port %u!", > best_hops, best_port); > } > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > index 58936e3..a9d13c8 100644 > --- a/opensm/opensm/osm_switch.c > +++ b/opensm/opensm/osm_switch.c > @@ -274,6 +274,7 @@ uint8_t > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > IN osm_port_t * p_port, > IN const uint16_t lid_ho, > + IN unsigned start_from, > IN const boolean_t ignore_existing, > IN const boolean_t dor) > { > @@ -294,6 +295,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > uint8_t port_num; > uint8_t num_ports; > uint32_t least_paths = 0xFFFFFFFF; > + unsigned i; > /* > The follwing will track the least paths if the > route should go through a new system/node > @@ -397,8 +399,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > */ > > /* port number starts with one and num_ports is 1 + num phys ports */ > - for (port_num = 1; port_num < num_ports; port_num++) { > - if (osm_switch_get_hop_count(p_sw, base_lid, port_num) != > + for (i = start_from; i < start_from + num_ports; i++) { > + port_num = i%num_ports; > + if (!port_num || > + osm_switch_get_hop_count(p_sw, base_lid, port_num) != > least_hops) > continue; > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > index c073037..2aae6d5 100644 > --- a/opensm/opensm/osm_ucast_mgr.c > +++ b/opensm/opensm/osm_ucast_mgr.c > @@ -208,7 +208,8 @@ find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, > static void > __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > IN osm_switch_t * const p_sw, > - IN osm_port_t * const p_port) > + IN osm_port_t * const p_port, > + IN unsigned lid_offset) > { > uint16_t min_lid_ho; > uint16_t max_lid_ho; > @@ -217,19 +218,14 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > boolean_t is_ignored_by_port_prof; > ib_net64_t node_guid; > struct osm_routing_engine *p_routing_eng; > - /* > - The following are temporary structures that will aid > - in providing better routing in LMC > 0 situations > - */ > - uint16_t lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > - struct osm_remote_node *p_remote_guid_used = NULL; > + unsigned start_from = 1; > > OSM_LOG_ENTER(p_mgr->p_log); > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > > - /* If the lids are zero - then there was some problem with the initialization. > - Don't handle this port. */ > + /* If the lids are zero - then there was some problem with > + * the initialization. Don't handle this port. */ > if (min_lid_ho == 0 || max_lid_ho == 0) { > OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A04: " > "Port 0x%" PRIx64 " has LID 0. An initialization " > @@ -238,16 +234,22 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > goto Exit; > } > > - if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { > + lid_ho = min_lid_ho + lid_offset; > + > + if (lid_ho > max_lid_ho) > + goto Exit; > + > + if (lid_offset) > + /* ignore potential overflow - it is handled in osm_switch.c */ > + start_from = osm_switch_get_port_by_lid(p_sw, lid_ho - 1) + 1; > + > + if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) > OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > - "Processing port 0x%" PRIx64 ", LIDs [0x%X,0x%X]\n", > - cl_ntoh64(osm_port_get_guid(p_port)), > + "Processing port 0x%" PRIx64 ", LID %u [0x%X,0x%X]\n", > + cl_ntoh64(osm_port_get_guid(p_port)), lid_ho, > min_lid_ho, max_lid_ho); > - } > > - /* > - TO DO - This should be runtime error, not a CL_ASSERT() > - */ > + /* TODO - This should be runtime error, not a CL_ASSERT() */ > CL_ASSERT(max_lid_ho < osm_switch_get_fwd_tbl_size(p_sw)); > > node_guid = osm_node_get_node_guid(p_sw->p_node); > @@ -260,80 +262,62 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > how best to distribute the LID range across the ports > that can reach those LIDs. > */ > - for (lid_ho = min_lid_ho; lid_ho <= max_lid_ho; lid_ho++) { > - /* Use the enhanced algorithm only for LMC > 0 */ > - if (lids_per_port > 1) { > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > - p_mgr->p_subn-> > - ignore_existing_lfts, > - p_mgr->is_dor); > - if (port > 0 && port != OSM_NO_PATH && p_port->priv) > - p_remote_guid_used = > - find_and_add_remote_sys(p_sw, port, > - p_port->priv); > + port = osm_switch_recommend_path(p_sw, p_port, lid_ho, start_from, > + p_mgr->p_subn->ignore_existing_lfts, > + p_mgr->is_dor); > + > + if (port == OSM_NO_PATH) { > + /* do not try to overwrite the ppro of non existing port ... */ > + is_ignored_by_port_prof = TRUE; > + > + /* Up/Down routing can cause unreachable routes between some > + switches so we do not report that as an error in that case */ > + if (!p_routing_eng->build_lid_matrices) { > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > + "No path to get to LID 0x%X from switch 0x%" > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > + /* trigger a new sweep - try again ... */ > + p_mgr->p_subn->subnet_initialization_error = TRUE; > } else > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > - p_mgr->p_subn-> > - ignore_existing_lfts, > - p_mgr->is_dor); > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > + "No path to get to LID 0x%X from switch 0x%" > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > + } else { > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > + "Routing LID 0x%X to port 0x%X" > + " for switch 0x%" PRIx64 "\n", > + lid_ho, port, cl_ntoh64(node_guid)); > > /* > - There might be no path to the target > + we would like to optionally ignore this port in equalization > + as in the case of the Mellanox Anafa Internal PCI TCA port > */ > - if (port == OSM_NO_PATH) { > - /* do not try to overwrite the ppro of non existing port ... */ > - is_ignored_by_port_prof = TRUE; > - > - /* Up/Down routing can cause unreachable routes between some > - switches so we do not report that as an error in that case */ > - if (!p_routing_eng->build_lid_matrices) { > - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > - "No path to get to LID 0x%X from switch 0x%" > - PRIx64 "\n", lid_ho, > - cl_ntoh64(node_guid)); > - /* trigger a new sweep - try again ... */ > - p_mgr->p_subn->subnet_initialization_error = > - TRUE; > - } else > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > - "No path to get to LID 0x%X from switch 0x%" > - PRIx64 "\n", lid_ho, > - cl_ntoh64(node_guid)); > - } else { > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > - "Routing LID 0x%X to port 0x%X" > - " for switch 0x%" PRIx64 "\n", > - lid_ho, port, cl_ntoh64(node_guid)); > - > - /* > - we would like to optionally ignore this port in equalization > - as in the case of the Mellanox Anafa Internal PCI TCA port > - */ > - is_ignored_by_port_prof = > - osm_port_prof_is_ignored_port(p_mgr->p_subn, > - node_guid, port); > - > - /* > - We also would ignore this route if the target lid is of a switch > - and the port_profile_switch_node is not TRUE > - */ > - if (!p_mgr->p_subn->opt.port_profile_switch_nodes) { > - is_ignored_by_port_prof |= > - (osm_node_get_type(p_port->p_node) == > - IB_NODE_TYPE_SWITCH); > - } > - } > + is_ignored_by_port_prof = > + osm_port_prof_is_ignored_port(p_mgr->p_subn, > + node_guid, port); > > /* > - We have selected the port for this LID. > - Write it to the forwarding tables. > + We also would ignore this route if the target lid is of > + a switch and the port_profile_switch_node is not TRUE > */ > - p_mgr->lft_buf[lid_ho] = port; > - if (!is_ignored_by_port_prof) { > - osm_switch_count_path(p_sw, port); > - if (p_remote_guid_used) > - p_remote_guid_used->forwarded_to++; > - } > + if (!p_mgr->p_subn->opt.port_profile_switch_nodes) > + is_ignored_by_port_prof |= > + (osm_node_get_type(p_port->p_node) == > + IB_NODE_TYPE_SWITCH); > + } > + > + /* > + We have selected the port for this LID. > + Write it to the forwarding tables. > + */ > + p_mgr->lft_buf[lid_ho] = port; > + if (!is_ignored_by_port_prof) { > + struct osm_remote_node *rem_node_used; > + osm_switch_count_path(p_sw, port); > + if (port > 0 && p_port->priv && > + (rem_node_used = find_and_add_remote_sys(p_sw, port, > + p_port->priv))) > + rem_node_used->forwarded_to++; > } > > Exit: > @@ -512,6 +496,7 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > osm_node_t *p_node; > osm_port_t *p_port; > const cl_qmap_t *p_port_tbl; > + unsigned i, lids_per_port; > > OSM_LOG_ENTER(p_mgr->p_log); > > @@ -538,12 +523,12 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > Iterate through every port setting LID routes for each > port based on base LID and LMC value. > */ > - > - for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > - p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > - p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) { > - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port); > - } > + lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > + for (i = 0; i < lids_per_port; i++) > + for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > + p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > + p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) > + __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port, i); > > osm_ucast_mgr_set_fwd_table(p_mgr, p_sw); > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: mpi_preserve_base_lids.xls Type: application/vnd.ms-excel Size: 20480 bytes Desc: not available URL: From hrosenstock at xsigo.com Mon Jun 16 11:09:06 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 16 Jun 2008 11:09:06 -0700 Subject: [ofa-general] opensm routing In-Reply-To: <20080616095840.37c7e17b.weiny2@llnl.gov> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> <20080616094629.557bf892.weiny2@llnl.gov> <1213634871.14047.708.camel@hrosenstock-ws.xsigo.com> <20080616095840.37c7e17b.weiny2@llnl.gov> Message-ID: <1213639746.14047.711.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 09:58 -0700, Ira Weiny wrote: > On Mon, 16 Jun 2008 09:47:51 -0700 > Hal Rosenstock wrote: > > > On Mon, 2008-06-16 at 09:46 -0700, Ira Weiny wrote: > > > On Mon, 16 Jun 2008 09:38:58 -0700 > > > Hal Rosenstock wrote: > > > > > > > On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: > > > > > On Mon, 16 Jun 2008 09:25:48 -0700 > > > > > Hal Rosenstock wrote: > > > > > > > > > > > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > > > > > > I asked the Lustre people in my hallway, and it isn't > > > > > > > currently configurable for Lustre. > > > > > > > > > > > > Wouldn't Lustre SL be inherited from partition based on underlying IPoIB > > > > > > interface ? > > > > > > > > > > I am not quite sure what you mean here? Our Lustre sets up their own QP's via > > > > > the RDMACM. So I believe we could set our SL and/or partition for those QP's > > > > > separately from IPoIB via a modify_qp call; right? > > > > > > > > RDMA CM does address resolution based on IP addresses and an SL can be > > > > associated with the outgoing IPoIB interface. > > > > > > > > > > Right, but does it _have_ to be the associated? I thought not. > > > > Do you want a different SL from that ? > > > > Maybe, some MPI's may use the RDMACM as well (I think some already do). > Therefore if you want Lustre and MPI to be on different SL's at least one of > them will have to change from the "inherited" IPoIB SL. Just use multiple (per ULP ?) IPoIB interfaces on different partitions with different SLs. -- Hal > > There's a QoS syntax but I'm not > > sure how Lustre plays into that. > > Just to be clear this is only a "thought experiment" at this point. We have > not tried to do any of this for real, yet. ;-) We realized there might be > many changes to various configurations and codes which may need to be done. > But knowing that I/O is less dependent on latency than MPI it seems to follow > that overall system performance could benefit from having MPI run at a higher > priority than Lustre/NFS etc. > > Ira > From swise at opengridcomputing.com Mon Jun 16 12:07:05 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 16 Jun 2008 14:07:05 -0500 Subject: [ofa-general] Re: cxgb3_0010_napi.patch fails to be applied In-Reply-To: <48569001.3070304@dev.mellanox.co.il> References: <48569001.3070304@dev.mellanox.co.il> Message-ID: <4856B9D9.7040602@opengridcomputing.com> Pull from: http://git.openfabrics.org/~swise/ofed-1.4 ofed_kernel Steve. Vladimir Sokolovsky wrote: > Hello Steve, > I merged ofed-1.4 kernel tree with 2.6.26-rc6 kernel. > > Now ofed_makedist.sh fails: > > Failed executing > /local/scm/ofed-1.4/linux-2.6/ofed_scripts/ofed_patch.sh > --with-backport=2.6.16 >> > /tmp/build-ofed_kernel-R21111/ofed_kernel-2.6.16.log > Hunk #15 succeeded at 2461 (offset 25 lines). > Hunk #16 succeeded at 2571 (offset 25 lines). > Hunk #17 succeeded at 2583 (offset 25 lines). > Hunk #18 succeeded at 2619 (offset 25 lines). > Hunk #19 succeeded at 2644 (offset 25 lines). > Hunk #20 succeeded at 2765 (offset 25 lines). > 1 out of 20 hunks FAILED -- rejects in file drivers/net/cxgb3/sge.c > Patch cxgb3_0010_napi.patch does not apply (enforce with -f) > > Please fix this issue and add required backports. > > git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git > git_branch: ofed_kernel > > Regards, > Vladimir From amar.mudrankit at qlogic.com Mon Jun 16 12:44:59 2008 From: amar.mudrankit at qlogic.com (Amar Mudrankit) Date: Tue, 17 Jun 2008 01:14:59 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <48537C68.1010207@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> <485148DA.2060508@trash.net> <48529027.1090309@trash.net> <48537C68.1010207@trash.net> Message-ID: >> I was looking at latest release of iproute2, in which I could not >> locate fill_xstats. Can you help me to locate this? > > ->fill_xstats is in the kernel, in iproute its ->print_xstats. I was looking at kernel code net/8021q/vlan_netlink.c on which I have following queries: a. Is it a correct file in kernel I m referring to? b. vlan_fill_info is a function to send parameters back to user level. What is the role of skb as an argument to function? Is some network operation expected? c. How exactly the kernel API ops->get_size comes into play? I guess user prog should first get the size of parameters that will be returned by driver before calling ops->fill_info. d. Can somebody briefly introduce struct nlattr? It looks like all parameters flow through this structure. Thanks and Regards, Amar S Mudrankit From vlad at dev.mellanox.co.il Mon Jun 16 12:53:19 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Mon, 16 Jun 2008 22:53:19 +0300 Subject: [ofa-general] Re: cxgb3_0010_napi.patch fails to be applied In-Reply-To: <4856B9D9.7040602@opengridcomputing.com> References: <48569001.3070304@dev.mellanox.co.il> <4856B9D9.7040602@opengridcomputing.com> Message-ID: <4856C4AF.6070905@dev.mellanox.co.il> Steve Wise wrote: > Pull from: > > http://git.openfabrics.org/~swise/ofed-1.4 ofed_kernel > > Steve. > Done, Regards, Vladimir From chu11 at llnl.gov Mon Jun 16 13:08:52 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 16 Jun 2008 13:08:52 -0700 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <1213636881.5369.438.camel@cardanus.llnl.gov> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <4854CFFC.7040508@dev.mellanox.co.il> <1213636881.5369.438.camel@cardanus.llnl.gov> Message-ID: <1213646932.5369.446.camel@cardanus.llnl.gov> On Mon, 2008-06-16 at 10:21 -0700, Al Chu wrote: > Hey Yevgeny, > > On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: > > Hi Al, > > > > Al Chu wrote: > > > Hey Sasha, > > > > > > This is a conceptually simple option I've developed for updn routing. > > > > > > Currently in updn routing, nodes/guids are routed on switches in a > > > seemingly-random order, which I believe is due to internal data > > > structure organization (i.e. cl_qmap_apply_func is called on > > > port_guid_tbl) as well as how the fabric is scanned (it is logically > > > scanned from a port perspective, but it may not be logical from a node > > > perspective). I had a hypothesis that this was leading to increased > > > contention in the network for MPI. > > > > > > For example, suppose we have 12 uplinks from a leaf switch to a spine > > > switch. If we want to send data from this leaf switch to node[13-24], > > > the up links we will send on are pretty random. It's because: > > > > > > A) node[13-24] are individually routed at seemingly-random points based > > > on when they are called by cl_qmap_apply_func(). > > > > > > B) the ports chosen for routing are based on least used port usage. > > > > > > C) least used port usage is based on whatever was routed earlier on. > > > > > > So I developed this patch series, which supports an option called > > > "guid_routing_order_file" which allows the user to input a file with a > > > list of port_guids which will indicate the order in which guids are > > > routed instead (naturally, those guids not listed are routed last). > > > > Great idea! > > Thanks. > > > I understand that this guid_routing_order_file is synchronized with > > an MPI rank file, right? If not, then synchronizing them might give > > even better results. > > Not quite sure what you mean by a MPI rank file. At LLNL, slurm is > responsible for MPI ranks, so I order the guids in my file according to > how slurm is configured for chosing MPI ranks. I will admit to being a > novice to MPI's configuration (blindly accepting slurm MPI rankings). > Is there an underlying file that MPI libs use for ranking knowledge? I spoke to one of our MPI guys. I wasn't aware that in some MPIs you can input a file to tell it how ranks should be assigned to nodes for MPI. I assume that's what you're talking about? Al > > Another idea: OpenSM can create such file (list, doesn't have to be > > actual file) automatically, just by checking topologically-adjacent > > leaf switches and their HCAs. > > Definitely a good idea. This patch set was just a "step one" kind of > thing. > > > > > > I list the port guids of the nodes of the cluster from node0 to nodeN, one > > > per line in the file. By listing the nodes in this order, I believe we > > > could get less contention in the network. In the example above, sending > > > to node[13-24] should use all of the 12 uplinks, b/c the ports will be > > > equally used b/c nodes[1-12] were routed beforehand in order. > > > > > > The results from some tests are pretty impressive when I do this. LMC=0 > > > average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s > > > when I use guid_routing_order. > > > > Can you compare this to the fat-tree routing? Conceptually, fat-tree > > is doing the same - it routes LIDs on nodes in a topological order, so > > it would be interesting to see the comparison. > > Actually I already did :-). w/ LMC=0. > > updn default - 391.374 MB/s > updn w/ guid_routing_order - 573.678 MB/s > ftree - 579.603 MB/s > > I later discovered that one of the internal ports of the cluster I'm > testing on was broken (sLB of a 288 port), and think that is the cause > of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as > designed) seemed to be able to work around it properly, while updn (as > currently implemented) couldn't. > > When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were able > to do better on some tests than ftree. One example (I think these > numbers are in microseconds. Lower is better): > > Alltoall 16K packets > ftree - 415490.6919 > updn normal (LMC=0) - 495460.5526 > updn w/ ordered routing (LMC=0) - 416562.7417 > updn w/ ordered routing (LMC=1) - 453153.7289 > - this ^^^ result is quite odd. Not sure why. > updn w/ ordered routing (LMC=2) - 3660132.1530 > > We are regularly debating what will be better overall at the end of the > day. > > > Also, fat-tree produces the guid order file automatically, but nobody > > used it yet as an input to produce MPI rank file. > > I didn't know about this option. How do you do this (just skimmed the > manpage, didn't see anything)? I know about the --cn_guid_file. But > since that file doesn't have to be ordered, that's why I created a > different option (rather than have the cn_guid_file for both ftree and > updn). > > Al > > > -- Yevgeny > > > > > A variety of other positive performance > > > increases were found when doing other tests, other MPIs, and other LMCs > > > if anyone is interested. > > > > > > BTW, I developed this patch series before your preserve-base-lid patch > > > series. It will 100% conflict with the preserve-base-lid patch series. > > > I will fix this patch series once the preserve-base-lids patch series is > > > committed to git. I'm just looking for comments right now. > > > > > > Al > > > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From kaber at trash.net Mon Jun 16 13:39:19 2008 From: kaber at trash.net (Patrick McHardy) Date: Mon, 16 Jun 2008 22:39:19 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <71d336490806120850x3b5e6172jf4d75850771d6349@mail.gmail.com> <485148DA.2060508@trash.net> <48529027.1090309@trash.net> <48537C68.1010207@trash.net> Message-ID: <4856CF77.8010705@trash.net> Amar Mudrankit wrote: >>> I was looking at latest release of iproute2, in which I could not >>> locate fill_xstats. Can you help me to locate this? >> ->fill_xstats is in the kernel, in iproute its ->print_xstats. > > I was looking at kernel code net/8021q/vlan_netlink.c on which I have > following queries: > > a. Is it a correct file in kernel I m referring to? Yes. > b. vlan_fill_info is a function to send parameters back to user > level. What is the role of skb as an argument to function? Is some > network operation expected? netlink is a (local) networking protocol, this is the message sent to userspace. > c. How exactly the kernel API ops->get_size comes into play? I guess > user prog should first get the size of parameters that will be > returned by driver before calling ops->fill_info. No, its used internally for skb allocation. It should return the size of all encapsulated netlink attributes that will be dumped by ->fill_info (in most cases, just a constant value). > d. Can somebody briefly introduce struct nlattr? It looks like all > parameters flow through this structure. Not much to say about it, nla_type contains the numerical attribute type, nla_len the length without padding. Data follows the attribute, followed by padding to a multiple of four. The netlink construction functions and macros from include/net/netlink.h take care of all of this for you. From enshroudedprsu4 at ls3517.wdf.sap-ag.de Mon Jun 16 14:16:56 2008 From: enshroudedprsu4 at ls3517.wdf.sap-ag.de (Ebony Keys) Date: Mon, 16 Jun 2008 18:16:56 -0300 Subject: [ofa-general] Turn your professional experience into the University Degree you deserve. m gtsl gucxxi Message-ID: <527004929.37560656461946@ls3517.wdf.sap-ag.de> Want the deg dp ree but can not find the ti ja me? WHAT A GRE qh AT ID ve EA! We pro ozd vide a co sfl ncept that will allow anyone with sufficient work experience to obtain a fully verifiable Un fr ivers tcs ity Deg khj ree. Ba tr chel ch ors, Ma qb ste kha rs or ev xv en a Do sh cto mde rate. Think of it, within four to six weeks, you too could be a co dbi lle cmj ge gr bnm adu xw ate. Many people sh fa are the same fru ryn stra bn tion, they are all doing the work of the person that has the de goe gree and the person that has the d zf egree is getting all the mo bqq ney. Do not you think that it is time you were paid fair comp tb ensa nu tion for the level of work you are already doing? This is your chance t be o fin jvx ally make the right move and receive your due benefits. If you are like most people, you are more than qu wr alif vk ied with your experience, but are lacking that pre up stig vx ious piece of paper known as a di ovt plo yt ma that is often the pas ag sp lk ort to suc jmh cess. CA qw LL US TODAY AND GIVE YOUR WORK EXP yg ERIE sz NCE THE CHANCE TO EARN YOU THE HIG jwd HER CO pwx MPENS rah ATION YOU DE bm SER xt VE! 1-5 jyh 2 tgq 0-2 yv 0 mn 3-0 kqu 21 uk 3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Mon Jun 16 14:45:55 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 16 Jun 2008 16:45:55 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: <4851CB5B.1060804@opengridcomputing.com> References: <20080612211302.25120.41155.stgit@dell3.ogc.int> <4851CB5B.1060804@opengridcomputing.com> Message-ID: <4856DF13.3090409@opengridcomputing.com> Steve Wise wrote: > > > Roland Dreier wrote: >>> - Add a new rdma ctl command called RDMA_GET_MIB to the low level >>> driver >>> to obtain the protocol mib from the hardware. >>> >>> - Add a sysfs entry to allow dumping these stats. >> >> This looks really useful for debugging/tuning etc... what MIBs are >> being dumped >> here? The standard TCP and IPv4 MIBs? >> > > Yes, this is a standard TCP MIB. > I take this back. These stats are a subset of the IP and TCP MIBs. I'd like to get something out in 2.6.27/ofed-1.4 for support reasons without opening the SNMP can of worms. So I'll define a standard struct iw_protocol_counters and include the chelsio stats. Other providers can expand if they care. I'll also add the core stuff to have sysfs.c create the attribute files, one per stat and query the device to get the values via a device get_iw_stats() call like you propose. The stats can live in /sys/class/infiniband/$device/protocol_counters/iwarp/*. How does that sound? From yiftahs at voltaire.com Mon Jun 16 14:53:50 2008 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Tue, 17 Jun 2008 00:53:50 +0300 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order"option for updn routing In-Reply-To: <1213646932.5369.446.camel@cardanus.llnl.gov> Message-ID: <39C75744D164D948A170E9792AF8E7CA013E7622@exil.voltaire.com> Al, Yevgeny, > > > I understand that this guid_routing_order_file is synchronized with > > > an MPI rank file, right? If not, then synchronizing them might give > > > even better results. > > > > Not quite sure what you mean by a MPI rank file. At LLNL, slurm is > > responsible for MPI ranks, so I order the guids in my file according to > > how slurm is configured for chosing MPI ranks. I will admit to being a > > novice to MPI's configuration (blindly accepting slurm MPI rankings). > > Is there an underlying file that MPI libs use for ranking knowledge? > > I spoke to one of our MPI guys. I wasn't aware that in some MPIs you > can input a file to tell it how ranks should be assigned to nodes for > MPI. I assume that's what you're talking about? > > Al Upcoming Open MPI 1.3 will have such capabilities of rank placement in a specific node and specific CPU, we will also have some decisions settings how to communicate with different HCAs in multi HCAs node (we also have these capabilities in VLT-MPI for more then 2 years now but it is going into EOL stage...). I think that more important then rank placement is communication pattern (i.e. some ranks communicate a lot and some does not send a single message) and this is far more complicated to do. Yiftah > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general- > bounces at lists.openfabrics.org] On Behalf Of Al Chu > Sent: Monday, June 16, 2008 23:09 > To: kliteyn at dev.mellanox.co.il > Cc: OpenIB > Subject: Re: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing- > order"option for updn routing > > On Mon, 2008-06-16 at 10:21 -0700, Al Chu wrote: > > Hey Yevgeny, > > > > On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: > > > Hi Al, > > > > > > Al Chu wrote: > > > > Hey Sasha, > > > > > > > > This is a conceptually simple option I've developed for updn routing. > > > > > > > > Currently in updn routing, nodes/guids are routed on switches in a > > > > seemingly-random order, which I believe is due to internal data > > > > structure organization (i.e. cl_qmap_apply_func is called on > > > > port_guid_tbl) as well as how the fabric is scanned (it is logically > > > > scanned from a port perspective, but it may not be logical from a > node > > > > perspective). I had a hypothesis that this was leading to increased > > > > contention in the network for MPI. > > > > > > > > For example, suppose we have 12 uplinks from a leaf switch to a > spine > > > > switch. If we want to send data from this leaf switch to node[13- > 24], > > > > the up links we will send on are pretty random. It's because: > > > > > > > > A) node[13-24] are individually routed at seemingly-random points > based > > > > on when they are called by cl_qmap_apply_func(). > > > > > > > > B) the ports chosen for routing are based on least used port usage. > > > > > > > > C) least used port usage is based on whatever was routed earlier on. > > > > > > > > So I developed this patch series, which supports an option called > > > > "guid_routing_order_file" which allows the user to input a file with > a > > > > list of port_guids which will indicate the order in which guids are > > > > routed instead (naturally, those guids not listed are routed last). > > > > > > Great idea! > > > > Thanks. > > > > > I understand that this guid_routing_order_file is synchronized with > > > an MPI rank file, right? If not, then synchronizing them might give > > > even better results. > > > > Not quite sure what you mean by a MPI rank file. At LLNL, slurm is > > responsible for MPI ranks, so I order the guids in my file according to > > how slurm is configured for chosing MPI ranks. I will admit to being a > > novice to MPI's configuration (blindly accepting slurm MPI rankings). > > Is there an underlying file that MPI libs use for ranking knowledge? > > I spoke to one of our MPI guys. I wasn't aware that in some MPIs you > can input a file to tell it how ranks should be assigned to nodes for > MPI. I assume that's what you're talking about? > > Al > > > > Another idea: OpenSM can create such file (list, doesn't have to be > > > actual file) automatically, just by checking topologically-adjacent > > > leaf switches and their HCAs. > > > > Definitely a good idea. This patch set was just a "step one" kind of > > thing. > > > > > > > > > I list the port guids of the nodes of the cluster from node0 to > nodeN, one > > > > per line in the file. By listing the nodes in this order, I believe > we > > > > could get less contention in the network. In the example above, > sending > > > > to node[13-24] should use all of the 12 uplinks, b/c the ports will > be > > > > equally used b/c nodes[1-12] were routed beforehand in order. > > > > > > > > The results from some tests are pretty impressive when I do this. > LMC=0 > > > > average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s > > > > when I use guid_routing_order. > > > > > > Can you compare this to the fat-tree routing? Conceptually, fat-tree > > > is doing the same - it routes LIDs on nodes in a topological order, so > > > it would be interesting to see the comparison. > > > > Actually I already did :-). w/ LMC=0. > > > > updn default - 391.374 MB/s > > updn w/ guid_routing_order - 573.678 MB/s > > ftree - 579.603 MB/s > > > > I later discovered that one of the internal ports of the cluster I'm > > testing on was broken (sLB of a 288 port), and think that is the cause > > of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as > > designed) seemed to be able to work around it properly, while updn (as > > currently implemented) couldn't. > > > > When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were able > > to do better on some tests than ftree. One example (I think these > > numbers are in microseconds. Lower is better): > > > > Alltoall 16K packets > > ftree - 415490.6919 > > updn normal (LMC=0) - 495460.5526 > > updn w/ ordered routing (LMC=0) - 416562.7417 > > updn w/ ordered routing (LMC=1) - 453153.7289 > > - this ^^^ result is quite odd. Not sure why. > > updn w/ ordered routing (LMC=2) - 3660132.1530 > > > > We are regularly debating what will be better overall at the end of the > > day. > > > > > Also, fat-tree produces the guid order file automatically, but nobody > > > used it yet as an input to produce MPI rank file. > > > > I didn't know about this option. How do you do this (just skimmed the > > manpage, didn't see anything)? I know about the --cn_guid_file. But > > since that file doesn't have to be ordered, that's why I created a > > different option (rather than have the cn_guid_file for both ftree and > > updn). > > > > Al > > > > > -- Yevgeny > > > > > > > A variety of other positive performance > > > > increases were found when doing other tests, other MPIs, and other > LMCs > > > > if anyone is interested. > > > > > > > > BTW, I developed this patch series before your preserve-base-lid > patch > > > > series. It will 100% conflict with the preserve-base-lid patch > series. > > > > I will fix this patch series once the preserve-base-lids patch > series is > > > > committed to git. I'm just looking for comments right now. > > > > > > > > Al > > > > > > > > -- > Albert Chu > chu11 at llnl.gov > 925-422-5311 > Computer Scientist > High Performance Systems Division > Lawrence Livermore National Laboratory > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general From roland.list at gmail.com Mon Jun 16 18:36:20 2008 From: roland.list at gmail.com (Roland Dreier) Date: Mon, 16 Jun 2008 18:36:20 -0700 Subject: [ofa-general] [ANNOUNCE] chelsio libcxgb3 version 1.2.1 released In-Reply-To: <4856A93A.8010400@opengridcomputing.com> References: <4856A93A.8010400@opengridcomputing.com> Message-ID: > Chelsio iWARP fans, > > Version 1.2.1 of libcxgb3 is available at: > > http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.1.tar.gz > > This library is packaged along with ofed-1.3.1 and is required for 2.6.26 > kernels and beyond. I didn't look at this yet, but can it work with pre-6.0 firmware too? Would be nice to be able to upgrade the library once and then work with both old and new kernels... also makes it much easier for distro packaging, since then the libcxgb3 package doesn't have to worry about exactly what kernel is present. - R. From roland.list at gmail.com Mon Jun 16 18:34:08 2008 From: roland.list at gmail.com (Roland Dreier) Date: Mon, 16 Jun 2008 18:34:08 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: <4856DF13.3090409@opengridcomputing.com> References: <20080612211302.25120.41155.stgit@dell3.ogc.int> <4851CB5B.1060804@opengridcomputing.com> <4856DF13.3090409@opengridcomputing.com> Message-ID: > I take this back. These stats are a subset of the IP and TCP MIBs. > I'd like to get something out in 2.6.27/ofed-1.4 for support reasons without > opening the SNMP can of worms. So I'll define a standard struct > iw_protocol_counters and include the chelsio stats. Other providers can > expand if they care. I'll also add the core stuff to have sysfs.c create > the attribute files, one per stat and query the device to get the values via > a device get_iw_stats() call like you propose. The stats can live in > /sys/class/infiniband/$device/protocol_counters/iwarp/*. > How does that sound? Sounds great except I'm not thrilled about the exact sysfs pathname... I guess iWARP devices always have one port (??) (physically multiport NICs show up as multiple devices? don't have good access to my system with a two-port rnic to check) The only point of all that being that $device/ports/$port may not make total sense. But still why not follow the convention we already have and just use $device/ports/1/counters/ and use the tcp and ip mib names rather than ib pma names. - R. From swise at opengridcomputing.com Mon Jun 16 18:51:02 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 16 Jun 2008 20:51:02 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: References: <20080612211302.25120.41155.stgit@dell3.ogc.int> <4851CB5B.1060804@opengridcomputing.com> <4856DF13.3090409@opengridcomputing.com> Message-ID: <48571886.3080104@opengridcomputing.com> Roland Dreier wrote: >> I take this back. These stats are a subset of the IP and TCP MIBs. > >> I'd like to get something out in 2.6.27/ofed-1.4 for support reasons without >> opening the SNMP can of worms. So I'll define a standard struct >> iw_protocol_counters and include the chelsio stats. Other providers can >> expand if they care. I'll also add the core stuff to have sysfs.c create >> the attribute files, one per stat and query the device to get the values via >> a device get_iw_stats() call like you propose. The stats can live in >> /sys/class/infiniband/$device/protocol_counters/iwarp/*. > >> How does that sound? > > Sounds great except I'm not thrilled about the exact sysfs pathname... > I guess iWARP devices always have one port (??) (physically multiport > NICs show up as multiple devices? don't have good access to my system > with a two-port rnic to check) The only point of all that being that > $device/ports/$port may not make total sense. > The TCP and IP protocol stats are global to the device. Just like the native stack's TCP/IP stats aren't per-nic, we can't report offload TCP stats per-port. Yes? Note per-port stats are reported via ethtool -S for iwarp devices (L2 stats). Steve. From swise at opengridcomputing.com Mon Jun 16 18:52:35 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 16 Jun 2008 20:52:35 -0500 Subject: [ofa-general] [ANNOUNCE] chelsio libcxgb3 version 1.2.1 released In-Reply-To: References: <4856A93A.8010400@opengridcomputing.com> Message-ID: <485718E3.3010903@opengridcomputing.com> As it stands now, it requires 6.0 firmware. Roland Dreier wrote: >> Chelsio iWARP fans, >> >> Version 1.2.1 of libcxgb3 is available at: >> >> http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.1.tar.gz >> >> This library is packaged along with ofed-1.3.1 and is required for 2.6.26 >> kernels and beyond. > > I didn't look at this yet, but can it work with pre-6.0 firmware too? > Would be nice to be able to upgrade the library once and then work > with both old and new kernels... also makes it much easier for distro > packaging, since then the libcxgb3 package doesn't have to worry > about exactly what kernel is present. > > - R. From swise at opengridcomputing.com Mon Jun 16 18:58:55 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 16 Jun 2008 20:58:55 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Add protocol statistics to iw_cxgb3. In-Reply-To: References: <20080612211302.25120.41155.stgit@dell3.ogc.int> <4851CB5B.1060804@opengridcomputing.com> <4856DF13.3090409@opengridcomputing.com> Message-ID: <48571A5F.1090206@opengridcomputing.com> > Sounds great except I'm not thrilled about the exact sysfs pathname... > I guess iWARP devices always have one port (??) (physically multiport > NICs show up as multiple devices? don't have good access to my system > with a two-port rnic to check) The chelsio device shows up as one RDMA device per adapter. So a 2 port chelsio device will have two netdev interfaces but one ib_device interface. Stevo From vlad at dev.mellanox.co.il Tue Jun 17 01:10:34 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Tue, 17 Jun 2008 11:10:34 +0300 Subject: [ofa-general] dapl compilation fails on Fedora 9 Message-ID: <4857717A.9020405@dev.mellanox.co.il> Hello Arlin, I got the following dapl compilation errors on Fedora 9: gcc -DHAVE_CONFIG_H -I. -I. -I. -I../libibverbs/include/infiniband -I../librdmacm/include -I../libibverbs/include -I../../dat/include -Wall -g -D_GNU_SOURCE -DOS_RELEASE=131078 -DDAT_EXTENSIONS -I./dat/include/ -I./dat/udat/ -I./dat/udat/linux -I./dat/common/ -g -O2 -MT dat_udat_libdat2_la-dat_strerror.lo -MD -MP -MF .deps/dat_udat_libdat2_la-dat_strerror.Tpo -c dat/common/dat_strerror.c -fPIC -DPIC -o .libs/dat_udat_libdat2_la-dat_strerror.o dat/common/dat_strerror.c: In function 'dat_strerror': dat/common/dat_strerror.c:621: error: 'NULL' undeclared (first use in this function) dat/common/dat_strerror.c:621: error: (Each undeclared identifier is reported only once dat/common/dat_strerror.c:621: error: for each function it appears in.) make[3]: Leaving directory `/tmp/ofa_1_4_dev_user-20080617-1038_check/src/userspace/dapl' Making all in test/dtest make[3]: *** [dat_udat_libdat2_la-dat_strerror.lo] Error 1 ... if gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I include -I mdep/linux -I ./../../dat/include -I../libibverbs/include/infiniband -I../librdmacm/include -I../libibverbs/include -I../../dat/include -DDAT_EXTENSIONS -g -O2 -MT dapltest-dapl_netaddr.o -MD -MP -MF ".deps/dapltest-dapl_netaddr.Tpo" -c -o dapltest-dapl_netaddr.o `test -f 'cmd/dapl_netaddr.c' || echo './'`cmd/dapl_netaddr.c; \ then mv -f ".deps/dapltest-dapl_netaddr.Tpo" ".deps/dapltest-dapl_netaddr.Po"; else rm -f ".deps/dapltest-dapl_netaddr.Tpo"; exit 1; fi cmd/dapl_netaddr.c: In function גDT_NetAddrLookupHostAddressג: cmd/dapl_netaddr.c:94: error: EAI_ADDRFAMILY undeclared (first use in this function) cmd/dapl_netaddr.c:94: error: (Each undeclared identifier is reported only once cmd/dapl_netaddr.c:94: error: for each function it appears in.) cmd/dapl_netaddr.c:99: error: EAI_NODATA undeclared (first use in this function) make[3]: Leaving directory `/tmp/ofa_1_4_dev_user-20080617-1038_check/src/userspace/dapl/test/dapltest' make[3]: *** [dapltest-dapl_netaddr.o] Error 1 make[2]: Leaving directory `/tmp/ofa_1_4_dev_user-20080617-1038_check/src/userspace/dapl' make[1]: Leaving directory `/tmp/ofa_1_4_dev_user-20080617-1038_check/src/userspace/dapl' make[2]: *** [all-recursive] Error 1 make[1]: *** [all] Error 2 Regards, Vladimir From kliteyn at dev.mellanox.co.il Tue Jun 17 01:17:01 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 11:17:01 +0300 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <1213646932.5369.446.camel@cardanus.llnl.gov> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <4854CFFC.7040508@dev.mellanox.co.il> <1213636881.5369.438.camel@cardanus.llnl.gov> <1213646932.5369.446.camel@cardanus.llnl.gov> Message-ID: <485772FD.4000206@dev.mellanox.co.il> Hi Al, Al Chu wrote: > On Mon, 2008-06-16 at 10:21 -0700, Al Chu wrote: >> Hey Yevgeny, >> >> On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: >>> Hi Al, >>> >>> Al Chu wrote: >>>> Hey Sasha, >>>> >>>> This is a conceptually simple option I've developed for updn routing. >>>> >>>> Currently in updn routing, nodes/guids are routed on switches in a >>>> seemingly-random order, which I believe is due to internal data >>>> structure organization (i.e. cl_qmap_apply_func is called on >>>> port_guid_tbl) as well as how the fabric is scanned (it is logically >>>> scanned from a port perspective, but it may not be logical from a node >>>> perspective). I had a hypothesis that this was leading to increased >>>> contention in the network for MPI. >>>> >>>> For example, suppose we have 12 uplinks from a leaf switch to a spine >>>> switch. If we want to send data from this leaf switch to node[13-24], >>>> the up links we will send on are pretty random. It's because: >>>> >>>> A) node[13-24] are individually routed at seemingly-random points based >>>> on when they are called by cl_qmap_apply_func(). >>>> >>>> B) the ports chosen for routing are based on least used port usage. >>>> >>>> C) least used port usage is based on whatever was routed earlier on. >>>> >>>> So I developed this patch series, which supports an option called >>>> "guid_routing_order_file" which allows the user to input a file with a >>>> list of port_guids which will indicate the order in which guids are >>>> routed instead (naturally, those guids not listed are routed last). >>> Great idea! >> Thanks. >> >>> I understand that this guid_routing_order_file is synchronized with >>> an MPI rank file, right? If not, then synchronizing them might give >>> even better results. >> Not quite sure what you mean by a MPI rank file. At LLNL, slurm is >> responsible for MPI ranks, so I order the guids in my file according to >> how slurm is configured for chosing MPI ranks. I will admit to being a >> novice to MPI's configuration (blindly accepting slurm MPI rankings). >> Is there an underlying file that MPI libs use for ranking knowledge? > > I spoke to one of our MPI guys. I wasn't aware that in some MPIs you > can input a file to tell it how ranks should be assigned to nodes for > MPI. I assume that's what you're talking about? Yes, that is what I was talking about. There is a host file, where you list all the hosts that MPI should use, and in some MPIs there is also a way to specify the order of MPI ranks that would be assigned to processes (I'm not an MPI expert, so I'm not sure about the terminology that I use). I know that MVAPICH is using the host order when assigning ranks, so the order of the cluster nodes listed in host file is important. Not sure about OpenMPI. >>> Another idea: OpenSM can create such file (list, doesn't have to be >>> actual file) automatically, just by checking topologically-adjacent >>> leaf switches and their HCAs. >> Definitely a good idea. This patch set was just a "step one" kind of >> thing. >> >>>> I list the port guids of the nodes of the cluster from node0 to nodeN, one >>>> per line in the file. By listing the nodes in this order, I believe we >>>> could get less contention in the network. In the example above, sending >>>> to node[13-24] should use all of the 12 uplinks, b/c the ports will be >>>> equally used b/c nodes[1-12] were routed beforehand in order. >>>> >>>> The results from some tests are pretty impressive when I do this. LMC=0 >>>> average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s >>>> when I use guid_routing_order. >>> Can you compare this to the fat-tree routing? Conceptually, fat-tree >>> is doing the same - it routes LIDs on nodes in a topological order, so >>> it would be interesting to see the comparison. >> Actually I already did :-). w/ LMC=0. >> >> updn default - 391.374 MB/s >> updn w/ guid_routing_order - 573.678 MB/s >> ftree - 579.603 MB/s >> >> I later discovered that one of the internal ports of the cluster I'm >> testing on was broken (sLB of a 288 port), and think that is the cause >> of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as >> designed) seemed to be able to work around it properly, while updn (as >> currently implemented) couldn't. >> >> When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were able >> to do better on some tests than ftree. One example (I think these >> numbers are in microseconds. Lower is better): >> >> Alltoall 16K packets >> ftree - 415490.6919 >> updn normal (LMC=0) - 495460.5526 >> updn w/ ordered routing (LMC=0) - 416562.7417 >> updn w/ ordered routing (LMC=1) - 453153.7289 >> - this ^^^ result is quite odd. Not sure why. >> updn w/ ordered routing (LMC=2) - 3660132.1530 >> >> We are regularly debating what will be better overall at the end of the >> day. >> >>> Also, fat-tree produces the guid order file automatically, but nobody >>> used it yet as an input to produce MPI rank file. >> I didn't know about this option. How do you do this (just skimmed the >> manpage, didn't see anything)? Right, it's missing there. I'll add this info. The file is /var/log/opensm-ftree-ca-order.dump. Small correction though - the file contains ordered list of HCA LIDs and their host names. It's not a problem to change it to have guids as well, but MPI doesn't need guids anyway. Note that the optimal order might be different depending on the current topology state and the location of the management node that runs OpenSM. >> I know about the --cn_guid_file. But >> since that file doesn't have to be ordered, that's why I created a >> different option (rather than have the cn_guid_file for both ftree and >> updn). Right, the cn file doesn't have to be ordered - ftree will order it by itself. The ordering is by topology-adjacent leaf switches. -- Yevgeny >> >> Al >> >>> -- Yevgeny >>> >>>> A variety of other positive performance >>>> increases were found when doing other tests, other MPIs, and other LMCs >>>> if anyone is interested. >>>> >>>> BTW, I developed this patch series before your preserve-base-lid patch >>>> series. It will 100% conflict with the preserve-base-lid patch series. >>>> I will fix this patch series once the preserve-base-lids patch series is >>>> committed to git. I'm just looking for comments right now. >>>> >>>> Al >>>> From ogerlitz at voltaire.com Tue Jun 17 01:28:15 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Jun 2008 11:28:15 +0300 Subject: [ofa-general] Re: [PATCH RFC] librdmacm: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: <000201c8cfcf$b6759c00$9d37170a@amr.corp.intel.com> References: <000201c8cfcf$b6759c00$9d37170a@amr.corp.intel.com> Message-ID: <4857759F.5020405@voltaire.com> Sean Hefty wrote: > I've pulled this patch into my internal librdmacm git tree. It should make it > into the next release, which will correspond with 2.6.27, assuming the kernel > patches make it in. OK, great. Or. From ogerlitz at voltaire.com Tue Jun 17 01:31:16 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Jun 2008 11:31:16 +0300 Subject: [ofa-general] Re: RDMA_CM_EVENT_ADDR_CHANGE notification merge In-Reply-To: <000001c8cfcd$7ebe5fb0$9d37170a@amr.corp.intel.com> References: <000001c8cfcd$7ebe5fb0$9d37170a@amr.corp.intel.com> Message-ID: <48577654.1010109@voltaire.com> Sean Hefty wrote: >> Assuming Jeff accepts them into his tree, does it makes sense to you to pull >> his tree into yours and then push the patch sequence to Roland, or you >> want me to push it to Roland and you will sign/ack the patches? We are now >> at -rc6 and I want to bring this to the for-2.6.27 branch of Roland's this >> week. > I think this is Roland's call on whatever is best/easiest for him. But I'll > review the patches and provide a reviewed-by line. Hi Roland, Does the method I suggested of pulling some patches from Jeff's tree over which my patches are dependent upon into the infiniband tree make sense to you? Or. From kliteyn at dev.mellanox.co.il Tue Jun 17 01:45:16 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 11:45:16 +0300 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order"option for updn routing In-Reply-To: <39C75744D164D948A170E9792AF8E7CA013E7622@exil.voltaire.com> References: <39C75744D164D948A170E9792AF8E7CA013E7622@exil.voltaire.com> Message-ID: <4857799C.8030800@dev.mellanox.co.il> Hi Yiftah, Yiftah Shahar wrote: > Al, Yevgeny, > >>>> I understand that this guid_routing_order_file is synchronized > with >>>> an MPI rank file, right? If not, then synchronizing them might give >>>> even better results. >>> Not quite sure what you mean by a MPI rank file. At LLNL, slurm is >>> responsible for MPI ranks, so I order the guids in my file according to >>> how slurm is configured for chosing MPI ranks. I will admit to being a >>> novice to MPI's configuration (blindly accepting slurm MPI rankings). >>> Is there an underlying file that MPI libs use for ranking knowledge? >> I spoke to one of our MPI guys. I wasn't aware that in some MPIs you >> can input a file to tell it how ranks should be assigned to nodes for >> MPI. I assume that's what you're talking about? >> >> Al > Upcoming Open MPI 1.3 will have such capabilities of rank placement in a > specific node and specific CPU, we will also have some decisions > settings how to communicate with different HCAs in multi HCAs node (we > also have these capabilities in VLT-MPI for more then 2 years now but it > is going into EOL stage...). > > I think that more important then rank placement is communication pattern > (i.e. some ranks communicate a lot and some does not send a single > message) and this is far more complicated to do. Both are important. In routing we are dealing with congestion when there is some communication that involves many nodes. However, the communication is usually not random - it has a pattern, and this pattern is affected by ranks. In some patterns (such as "shift") all the nodes are sending something at every pattern stage, in others (such as "recursive doubling") some nodes send all the time, and others rarely. In addition to that, there are optimizations that reduce IB communication even more by having mpi processes on the same host communicate in shared memory, and then have a single "representant" for IB for all the processes. I think that MVAPICH1 and OpenMPI are doing this. However, doesn't matter how optimized the pattern will be, in the end it has to transmit something on the wire, so if OpenSM won't produce a balanced routing, you might get a single congested wire that will delay each and every stage of the MPI communication pattern. Theoretically, the best result could be achieved if OpenSM and MPI would work together - OpenSM would produce some kind of list that would describe the topology order of the nodes, and MPI would somehow use this info when assigning ranks. -- Yevgeny > Yiftah > > >> -----Original Message----- >> From: general-bounces at lists.openfabrics.org [mailto:general- >> bounces at lists.openfabrics.org] On Behalf Of Al Chu >> Sent: Monday, June 16, 2008 23:09 >> To: kliteyn at dev.mellanox.co.il >> Cc: OpenIB >> Subject: Re: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing- >> order"option for updn routing >> >> On Mon, 2008-06-16 at 10:21 -0700, Al Chu wrote: >>> Hey Yevgeny, >>> >>> On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: >>>> Hi Al, >>>> >>>> Al Chu wrote: >>>>> Hey Sasha, >>>>> >>>>> This is a conceptually simple option I've developed for updn > routing. >>>>> Currently in updn routing, nodes/guids are routed on switches in > a >>>>> seemingly-random order, which I believe is due to internal data >>>>> structure organization (i.e. cl_qmap_apply_func is called on >>>>> port_guid_tbl) as well as how the fabric is scanned (it is > logically >>>>> scanned from a port perspective, but it may not be logical from > a >> node >>>>> perspective). I had a hypothesis that this was leading to > increased >>>>> contention in the network for MPI. >>>>> >>>>> For example, suppose we have 12 uplinks from a leaf switch to a >> spine >>>>> switch. If we want to send data from this leaf switch to > node[13- >> 24], >>>>> the up links we will send on are pretty random. It's because: >>>>> >>>>> A) node[13-24] are individually routed at seemingly-random > points >> based >>>>> on when they are called by cl_qmap_apply_func(). >>>>> >>>>> B) the ports chosen for routing are based on least used port > usage. >>>>> C) least used port usage is based on whatever was routed earlier > on. >>>>> So I developed this patch series, which supports an option > called >>>>> "guid_routing_order_file" which allows the user to input a file > with >> a >>>>> list of port_guids which will indicate the order in which guids > are >>>>> routed instead (naturally, those guids not listed are routed > last). >>>> Great idea! >>> Thanks. >>> >>>> I understand that this guid_routing_order_file is synchronized > with >>>> an MPI rank file, right? If not, then synchronizing them might > give >>>> even better results. >>> Not quite sure what you mean by a MPI rank file. At LLNL, slurm is >>> responsible for MPI ranks, so I order the guids in my file according > to >>> how slurm is configured for chosing MPI ranks. I will admit to > being a >>> novice to MPI's configuration (blindly accepting slurm MPI > rankings). >>> Is there an underlying file that MPI libs use for ranking knowledge? >> I spoke to one of our MPI guys. I wasn't aware that in some MPIs you >> can input a file to tell it how ranks should be assigned to nodes for >> MPI. I assume that's what you're talking about? >> >> Al >> >>>> Another idea: OpenSM can create such file (list, doesn't have to > be >>>> actual file) automatically, just by checking > topologically-adjacent >>>> leaf switches and their HCAs. >>> Definitely a good idea. This patch set was just a "step one" kind > of >>> thing. >>> >>>>> I list the port guids of the nodes of the cluster from node0 to >> nodeN, one >>>>> per line in the file. By listing the nodes in this order, I > believe >> we >>>>> could get less contention in the network. In the example above, >> sending >>>>> to node[13-24] should use all of the 12 uplinks, b/c the ports > will >> be >>>>> equally used b/c nodes[1-12] were routed beforehand in order. >>>>> >>>>> The results from some tests are pretty impressive when I do > this. >> LMC=0 >>>>> average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 > MB/s >>>>> when I use guid_routing_order. >>>> Can you compare this to the fat-tree routing? Conceptually, > fat-tree >>>> is doing the same - it routes LIDs on nodes in a topological > order, so >>>> it would be interesting to see the comparison. >>> Actually I already did :-). w/ LMC=0. >>> >>> updn default - 391.374 MB/s >>> updn w/ guid_routing_order - 573.678 MB/s >>> ftree - 579.603 MB/s >>> >>> I later discovered that one of the internal ports of the cluster I'm >>> testing on was broken (sLB of a 288 port), and think that is the > cause >>> of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as >>> designed) seemed to be able to work around it properly, while updn > (as >>> currently implemented) couldn't. >>> >>> When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were > able >>> to do better on some tests than ftree. One example (I think these >>> numbers are in microseconds. Lower is better): >>> >>> Alltoall 16K packets >>> ftree - 415490.6919 >>> updn normal (LMC=0) - 495460.5526 >>> updn w/ ordered routing (LMC=0) - 416562.7417 >>> updn w/ ordered routing (LMC=1) - 453153.7289 >>> - this ^^^ result is quite odd. Not sure why. >>> updn w/ ordered routing (LMC=2) - 3660132.1530 >>> >>> We are regularly debating what will be better overall at the end of > the >>> day. >>> >>>> Also, fat-tree produces the guid order file automatically, but > nobody >>>> used it yet as an input to produce MPI rank file. >>> I didn't know about this option. How do you do this (just skimmed > the >>> manpage, didn't see anything)? I know about the --cn_guid_file. > But >>> since that file doesn't have to be ordered, that's why I created a >>> different option (rather than have the cn_guid_file for both ftree > and >>> updn). >>> >>> Al >>> >>>> -- Yevgeny >>>> >>>>> A variety of other positive performance >>>>> increases were found when doing other tests, other MPIs, and > other >> LMCs >>>>> if anyone is interested. >>>>> >>>>> BTW, I developed this patch series before your preserve-base-lid >> patch >>>>> series. It will 100% conflict with the preserve-base-lid patch >> series. >>>>> I will fix this patch series once the preserve-base-lids patch >> series is >>>>> committed to git. I'm just looking for comments right now. >>>>> >>>>> Al >>>>> >> -- >> Albert Chu >> chu11 at llnl.gov >> 925-422-5311 >> Computer Scientist >> High Performance Systems Division >> Lawrence Livermore National Laboratory >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib- >> general > From kliteyn at dev.mellanox.co.il Tue Jun 17 02:10:03 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 12:10:03 +0300 Subject: [ofa-general] opensm routing In-Reply-To: <1213639746.14047.711.camel@hrosenstock-ws.xsigo.com> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> <20080616094629.557bf892.weiny2@llnl.gov> <1213634871.14047.708.camel@hrosenstock-ws.xsigo.com> <20080616095840.37c7e17b.weiny2@llnl.gov> <1213639746.14047.711.camel@hrosenstock-ws.xsigo.com> Message-ID: <48577F6B.5020306@dev.mellanox.co.il> Hal Rosenstock wrote: > On Mon, 2008-06-16 at 09:58 -0700, Ira Weiny wrote: >> On Mon, 16 Jun 2008 09:47:51 -0700 >> Hal Rosenstock wrote: >> >>> On Mon, 2008-06-16 at 09:46 -0700, Ira Weiny wrote: >>>> On Mon, 16 Jun 2008 09:38:58 -0700 >>>> Hal Rosenstock wrote: >>>> >>>>> On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: >>>>>> On Mon, 16 Jun 2008 09:25:48 -0700 >>>>>> Hal Rosenstock wrote: >>>>>> >>>>>>> On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: >>>>>>>> I asked the Lustre people in my hallway, and it isn't >>>>>>>> currently configurable for Lustre. >>>>>>> Wouldn't Lustre SL be inherited from partition based on underlying IPoIB >>>>>>> interface ? >>>>>> I am not quite sure what you mean here? Our Lustre sets up their own QP's via >>>>>> the RDMACM. So I believe we could set our SL and/or partition for those QP's >>>>>> separately from IPoIB via a modify_qp call; right? >>>>> RDMA CM does address resolution based on IP addresses and an SL can be >>>>> associated with the outgoing IPoIB interface. >>>>> >>>> Right, but does it _have_ to be the associated? I thought not. >>> Do you want a different SL from that ? >>> >> Maybe, some MPI's may use the RDMACM as well (I think some already do). >> Therefore if you want Lustre and MPI to be on different SL's at least one of >> them will have to change from the "inherited" IPoIB SL. > > Just use multiple (per ULP ?) IPoIB interfaces on different partitions > with different SLs. AFAIK latest MVAPICH and OpenMPI have a command line option that specifies which SL to use, so you can have MPI running on non-default SL, leaving Lustre intact on a default SL. Besides, if Lustre sends path queries to OpenSM (via RDMACM) when opening their own QPs, then Lustre can use other SLs too. SM can even assign different SLs to communication to Lustre metadata servers and object storage servers if they are running on separate hosts, all based on the path query. -- Yevgeny > -- Hal > >>> There's a QoS syntax but I'm not >>> sure how Lustre plays into that. >> >> Just to be clear this is only a "thought experiment" at this point. We have >> not tried to do any of this for real, yet. ;-) We realized there might be >> many changes to various configurations and codes which may need to be done. >> But knowing that I/O is less dependent on latency than MPI it seems to follow >> that overall system performance could benefit from having MPI run at a higher >> priority than Lustre/NFS etc. >> >> Ira >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From ogerlitz at voltaire.com Tue Jun 17 02:16:02 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Jun 2008 12:16:02 +0300 (IDT) Subject: [ofa-general] [PATCH V5 3/5] rdma/cma: simplify locking needed for serialization of callbacks In-Reply-To: References: Message-ID: > Or Gerlitz wrote: >> Sean Hefty wrote: >> I wasn't clear on this before, but we shouldn't need to take the spinlock here >> at all now. We needed it before in order to check the state and increment >> dev_remove in one operation. Once the spinlock was released the state could >> have changed, but dev_remove would have halted the device removal thread. Under >> the new method, device removal is halted while we hold the handler_mutex. > OK, got it. >> This can end up trying to acquire the mutex twice. We could change this to >> mutex_lock(); >> if (id_priv->state == CMA_ADDR_BOUND || id_priv->state == CMA_ADDR_RESOLVED) > OK, will do so. Hi Sean, I just want to make sure you are ok with the final version of the patch that changes the rdma-cm to use a mutex for serialization of callbacks on the ID. Below is the version I intend to push which addresses your last comments ---- The rdma-cm has some logic in place to make sure that callbacks on an ID are delivered to the consumer in serialized manner, specifically it has code to protect against the device removal racing with a callback now being delivered to the user. This patch simplifies this logic by using a mutex per ID instead of the wait queue and atomic variable, such that cma_disable remove is now called cma_disable_callback and cma_enable_remove is now just mutex_unlock. Signed-off-by: Or Gerlitz cma.c | 108 +++++++++++++++++++++++++++++++----------------------------------- 1 files changed, 51 insertions(+), 57 deletions(-) Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -130,8 +130,7 @@ struct rdma_id_private { struct completion comp; atomic_t refcount; - wait_queue_head_t wait_remove; - atomic_t dev_remove; + struct mutex handler_mutex; int backlog; int timeout_ms; @@ -355,26 +354,15 @@ static void cma_deref_id(struct rdma_id_ complete(&id_priv->comp); } -static int cma_disable_remove(struct rdma_id_private *id_priv, +static int cma_disable_callback(struct rdma_id_private *id_priv, enum cma_state state) { - unsigned long flags; - int ret; - - spin_lock_irqsave(&id_priv->lock, flags); - if (id_priv->state == state) { - atomic_inc(&id_priv->dev_remove); - ret = 0; - } else - ret = -EINVAL; - spin_unlock_irqrestore(&id_priv->lock, flags); - return ret; -} - -static void cma_enable_remove(struct rdma_id_private *id_priv) -{ - if (atomic_dec_and_test(&id_priv->dev_remove)) - wake_up(&id_priv->wait_remove); + mutex_lock(&id_priv->handler_mutex); + if (id_priv->state != state) { + mutex_unlock(&id_priv->handler_mutex); + return -EINVAL; + } + return 0; } static int cma_has_cm_dev(struct rdma_id_private *id_priv) @@ -399,8 +387,7 @@ struct rdma_cm_id *rdma_create_id(rdma_c mutex_init(&id_priv->qp_mutex); init_completion(&id_priv->comp); atomic_set(&id_priv->refcount, 1); - init_waitqueue_head(&id_priv->wait_remove); - atomic_set(&id_priv->dev_remove, 0); + mutex_init(&id_priv->handler_mutex); INIT_LIST_HEAD(&id_priv->listen_list); INIT_LIST_HEAD(&id_priv->mc_list); get_random_bytes(&id_priv->seq_num, sizeof id_priv->seq_num); @@ -927,7 +914,7 @@ static int cma_ib_handler(struct ib_cm_i struct rdma_cm_event event; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -984,12 +971,12 @@ static int cma_ib_handler(struct ib_cm_i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.ib = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -1101,7 +1088,7 @@ static int cma_req_handler(struct ib_cm_ int offset, ret; listen_id = cm_id->context; - if (cma_disable_remove(listen_id, CMA_LISTEN)) + if (cma_disable_callback(listen_id, CMA_LISTEN)) return -ECONNABORTED; memset(&event, 0, sizeof event); @@ -1122,7 +1109,7 @@ static int cma_req_handler(struct ib_cm_ goto out; } - atomic_inc(&conn_id->dev_remove); + mutex_lock(&conn_id->handler_mutex); mutex_lock(&lock); ret = cma_acquire_dev(conn_id); mutex_unlock(&lock); @@ -1144,7 +1131,7 @@ static int cma_req_handler(struct ib_cm_ !cma_is_ud_ps(conn_id->id.ps)) ib_send_cm_mra(cm_id, CMA_CM_MRA_SETTING, NULL, 0); mutex_unlock(&lock); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); goto out; } @@ -1153,11 +1140,11 @@ static int cma_req_handler(struct ib_cm_ release_conn_id: cma_exch(conn_id, CMA_DESTROYING); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(&conn_id->id); out: - cma_enable_remove(listen_id); + mutex_unlock(&listen_id->handler_mutex); return ret; } @@ -1223,7 +1210,7 @@ static int cma_iw_handler(struct iw_cm_i struct sockaddr_in *sin; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -1267,12 +1254,12 @@ static int cma_iw_handler(struct iw_cm_i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.iw = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -1288,7 +1275,7 @@ static int iw_conn_req_handler(struct iw struct ib_device_attr attr; listen_id = cm_id->context; - if (cma_disable_remove(listen_id, CMA_LISTEN)) + if (cma_disable_callback(listen_id, CMA_LISTEN)) return -ECONNABORTED; /* Create a new RDMA id for the new IW CM ID */ @@ -1300,19 +1287,19 @@ static int iw_conn_req_handler(struct iw goto out; } conn_id = container_of(new_cm_id, struct rdma_id_private, id); - atomic_inc(&conn_id->dev_remove); + mutex_lock(&conn_id->handler_mutex); conn_id->state = CMA_CONNECT; dev = ip_dev_find(&init_net, iw_event->local_addr.sin_addr.s_addr); if (!dev) { ret = -EADDRNOTAVAIL; - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } ret = rdma_copy_addr(&conn_id->id.route.addr.dev_addr, dev, NULL); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1321,7 +1308,7 @@ static int iw_conn_req_handler(struct iw ret = cma_acquire_dev(conn_id); mutex_unlock(&lock); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1337,7 +1324,7 @@ static int iw_conn_req_handler(struct iw ret = ib_query_device(conn_id->id.device, &attr); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1353,14 +1340,14 @@ static int iw_conn_req_handler(struct iw /* User wants to destroy the CM ID */ conn_id->cm_id.iw = NULL; cma_exch(conn_id, CMA_DESTROYING); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(&conn_id->id); } out: if (dev) dev_put(dev); - cma_enable_remove(listen_id); + mutex_unlock(&listen_id->handler_mutex); return ret; } @@ -1592,7 +1579,7 @@ static void cma_work_handler(struct work struct rdma_id_private *id_priv = work->id; int destroy = 0; - atomic_inc(&id_priv->dev_remove); + mutex_lock(&id_priv->handler_mutex); if (!cma_comp_exch(id_priv, work->old_state, work->new_state)) goto out; @@ -1601,7 +1588,7 @@ static void cma_work_handler(struct work destroy = 1; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); if (destroy) rdma_destroy_id(&id_priv->id); @@ -1764,7 +1751,7 @@ static void addr_handler(int status, str struct rdma_cm_event event; memset(&event, 0, sizeof event); - atomic_inc(&id_priv->dev_remove); + mutex_lock(&id_priv->handler_mutex); /* * Grab mutex to block rdma_destroy_id() from removing the device while @@ -1793,13 +1780,13 @@ static void addr_handler(int status, str if (id_priv->id.event_handler(&id_priv->id, &event)) { cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); rdma_destroy_id(&id_priv->id); return; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); } @@ -2126,7 +2113,7 @@ static int cma_sidr_rep_handler(struct i struct ib_cm_sidr_rep_event_param *rep = &ib_event->param.sidr_rep_rcvd; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -2167,12 +2154,12 @@ static int cma_sidr_rep_handler(struct i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.ib = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -2570,9 +2557,12 @@ static int cma_ib_mc_handler(int status, int ret; id_priv = mc->id_priv; - if (cma_disable_remove(id_priv, CMA_ADDR_BOUND) && - cma_disable_remove(id_priv, CMA_ADDR_RESOLVED)) + mutex_lock(&id_priv->handler_mutex); + if ((id_priv->state != CMA_ADDR_BOUND) && + (id_priv->state != CMA_ADDR_RESOLVED)) { + mutex_unlock(&id_priv->handler_mutex); return 0; + } mutex_lock(&id_priv->qp_mutex); if (!status && id_priv->id.qp) @@ -2596,12 +2586,12 @@ static int cma_ib_mc_handler(int status, ret = id_priv->id.event_handler(&id_priv->id, &event); if (ret) { cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return 0; } - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return 0; } @@ -2760,22 +2750,26 @@ static int cma_remove_id_dev(struct rdma { struct rdma_cm_event event; enum cma_state state; - + int ret = 0; + /* Record that we want to remove the device */ state = cma_exch(id_priv, CMA_DEVICE_REMOVAL); if (state == CMA_DESTROYING) return 0; cma_cancel_operation(id_priv, state); - wait_event(id_priv->wait_remove, !atomic_read(&id_priv->dev_remove)); + mutex_lock(&id_priv->handler_mutex); /* Check for destruction from another callback. */ if (!cma_comp(id_priv, CMA_DEVICE_REMOVAL)) - return 0; + goto out; memset(&event, 0, sizeof event); event.event = RDMA_CM_EVENT_DEVICE_REMOVAL; - return id_priv->id.event_handler(&id_priv->id, &event); + ret = id_priv->id.event_handler(&id_priv->id, &event); +out: + mutex_unlock(&id_priv->handler_mutex); + return ret; } static void cma_process_remove(struct cma_device *cma_dev) From vlad at lists.openfabrics.org Tue Jun 17 02:24:44 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 17 Jun 2008 02:24:44 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080617-0200 daily build status Message-ID: <20080617092444.90F32E61C62@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.21.1 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080617-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From yiftahs at voltaire.com Tue Jun 17 02:45:27 2008 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Tue, 17 Jun 2008 12:45:27 +0300 Subject: [ofa-general] opensm routing In-Reply-To: <1213639746.14047.711.camel@hrosenstock-ws.xsigo.com> Message-ID: <39C75744D164D948A170E9792AF8E7CA013E7733@exil.voltaire.com> Ira, > > Maybe, some MPI's may use the RDMACM as well (I think some already do). > > Therefore if you want Lustre and MPI to be on different SL's at least > one of > > them will have to change from the "inherited" IPoIB SL. > > Just use multiple (per ULP ?) IPoIB interfaces on different partitions > with different SLs. Open MPI support service level even without rdma_cm. You can use 'btl_openib_ib_service_level' the default value is 0 but you can change it to (0<=SL<=15) Yiftah > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general- > bounces at lists.openfabrics.org] On Behalf Of Hal Rosenstock > Sent: Monday, June 16, 2008 21:09 > To: Ira Weiny > Cc: moody20 at llnl.gov; general at lists.openfabrics.org > Subject: Re: [ofa-general] opensm routing > > On Mon, 2008-06-16 at 09:58 -0700, Ira Weiny wrote: > > On Mon, 16 Jun 2008 09:47:51 -0700 > > Hal Rosenstock wrote: > > > > > On Mon, 2008-06-16 at 09:46 -0700, Ira Weiny wrote: > > > > On Mon, 16 Jun 2008 09:38:58 -0700 > > > > Hal Rosenstock wrote: > > > > > > > > > On Mon, 2008-06-16 at 09:35 -0700, Ira Weiny wrote: > > > > > > On Mon, 16 Jun 2008 09:25:48 -0700 > > > > > > Hal Rosenstock wrote: > > > > > > > > > > > > > On Mon, 2008-06-16 at 09:16 -0700, Al Chu wrote: > > > > > > > > I asked the Lustre people in my hallway, and it isn't > > > > > > > > currently configurable for Lustre. > > > > > > > > > > > > > > Wouldn't Lustre SL be inherited from partition based on > underlying IPoIB > > > > > > > interface ? > > > > > > > > > > > > I am not quite sure what you mean here? Our Lustre sets up > their own QP's via > > > > > > the RDMACM. So I believe we could set our SL and/or partition > for those QP's > > > > > > separately from IPoIB via a modify_qp call; right? > > > > > > > > > > RDMA CM does address resolution based on IP addresses and an SL > can be > > > > > associated with the outgoing IPoIB interface. > > > > > > > > > > > > > Right, but does it _have_ to be the associated? I thought not. > > > > > > Do you want a different SL from that ? > > > > > > > Maybe, some MPI's may use the RDMACM as well (I think some already do). > > Therefore if you want Lustre and MPI to be on different SL's at least > one of > > them will have to change from the "inherited" IPoIB SL. > > Just use multiple (per ULP ?) IPoIB interfaces on different partitions > with different SLs. > > -- Hal > > > > There's a QoS syntax but I'm not > > > sure how Lustre plays into that. > > > > Just to be clear this is only a "thought experiment" at this point. We > have > > not tried to do any of this for real, yet. ;-) We realized there might > be > > many changes to various configurations and codes which may need to be > done. > > But knowing that I/O is less dependent on latency than MPI it seems to > follow > > that overall system performance could benefit from having MPI run at a > higher > > priority than Lustre/NFS etc. > > > > Ira > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general From ogerlitz at voltaire.com Tue Jun 17 02:55:18 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Jun 2008 12:55:18 +0300 (IDT) Subject: [ofa-general] [PATCH V5 4/5] rdma/cma: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: >Or Gerlitz wrote: >> Sean Hefty wrote: >> We should probably skip id_priv->state == CMA_DEVICE_REMOVAL as well. > OK >> Is KERN_ERR what we want here? > no, I think we can do well with warning or info level >> Should this be goto (mutex_unlock) instead? > yes it would be better to have it this way Hi Sean, Also here, I want to make sure you are ok with the final version of the patch that changes the rdma-cm to deliver the addr change event. So this is the version I intend to push which addresses your last comments ---- RDMA_CM_EVENT_ADDR_CHANGE event can be used by rdma-cm consumers that wish to have their RDMA sessions always use the same links (eg ) as the IP stack does. In the current code, this does not happen when bonding is used and fail-over happened, but the IB link used by an already existing session is operating fine. Use netevent notification for sensing that a change has happened in the IP stack, then scan the rdma-cm IDs list to see if there is an ID that is "misaligned" in that respect with the IP stack, and deliver RDMA_CM_EVENT_ADDR_CHANGE for this ID. The user can act on the event or just ignore it Signed-off-by: Or Gerlitz drivers/infiniband/core/cma.c | 90 ++++++++++++++++++++++++++++++++++++++++++ include/rdma/rdma_cm.h | 3 - 2 files changed, 92 insertions(+), 1 deletion(-) Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -168,6 +168,12 @@ struct cma_work { struct rdma_cm_event event; }; +struct cma_ndev_work { + struct work_struct work; + struct rdma_id_private *id; + struct rdma_cm_event event; +}; + union cma_ip_addr { struct in6_addr ip6; struct { @@ -1595,6 +1601,29 @@ out: kfree(work); } +static void cma_ndev_work_handler(struct work_struct *_work) +{ + struct cma_ndev_work *work = container_of(_work, struct cma_ndev_work, work); + struct rdma_id_private *id_priv = work->id; + int destroy = 0; + + mutex_lock(&id_priv->handler_mutex); + if (id_priv->state == CMA_DESTROYING || + id_priv->state == CMA_DEVICE_REMOVAL) + goto out; + + if (id_priv->id.event_handler(&id_priv->id, &work->event)) { + cma_exch(id_priv, CMA_DESTROYING); + destroy = 1; + } +out: + mutex_unlock(&id_priv->handler_mutex); + cma_deref_id(id_priv); + if (destroy) + rdma_destroy_id(&id_priv->id); + kfree(work); +} + static int cma_resolve_ib_route(struct rdma_id_private *id_priv, int timeout_ms) { struct rdma_route *route = &id_priv->id.route; @@ -2723,6 +2752,64 @@ void rdma_leave_multicast(struct rdma_cm } EXPORT_SYMBOL(rdma_leave_multicast); +static int cma_netdev_change(struct net_device *ndev, struct rdma_id_private *id_priv) +{ + struct rdma_dev_addr *dev_addr; + struct cma_ndev_work *work; + + dev_addr = &id_priv->id.route.addr.dev_addr; + + if (!memcmp(dev_addr->src_dev_name, ndev->name, IFNAMSIZ) && + memcmp(dev_addr->src_dev_addr, ndev->dev_addr, ndev->addr_len)) { + printk(KERN_INFO "RDMA CM addr change for ndev %s used by id %p\n", + ndev->name, &id_priv->id); + work = kzalloc(sizeof *work, GFP_KERNEL); + if (!work) + return -ENOMEM; + INIT_WORK(&work->work, cma_ndev_work_handler); + work->id = id_priv; + work->event.event = RDMA_CM_EVENT_ADDR_CHANGE; + atomic_inc(&id_priv->refcount); + queue_work(cma_wq, &work->work); + } + + return 0; +} + +static int cma_netdev_callback(struct notifier_block *self, unsigned long event, + void *ctx) +{ + struct net_device *ndev = (struct net_device *)ctx; + struct cma_device *cma_dev; + struct rdma_id_private *id_priv; + int ret = NOTIFY_DONE; + + if (dev_net(ndev) != &init_net) + return NOTIFY_DONE; + + if (event != NETDEV_BONDING_FAILOVER) + return NOTIFY_DONE; + + if (!(ndev->flags & IFF_MASTER) || !(ndev->priv_flags & IFF_BONDING)) + return NOTIFY_DONE; + + mutex_lock(&lock); + list_for_each_entry(cma_dev, &dev_list, list) + list_for_each_entry(id_priv, &cma_dev->id_list, list) { + ret = cma_netdev_change(ndev, id_priv); + if (ret) + goto out; + } + +out: + mutex_unlock(&lock); + return ret; +} + +static struct notifier_block cma_nb = { + .notifier_call = cma_netdev_callback +}; + static void cma_add_one(struct ib_device *device) { struct cma_device *cma_dev; @@ -2831,6 +2918,7 @@ static int cma_init(void) ib_sa_register_client(&sa_client); rdma_addr_register_client(&addr_client); + register_netdevice_notifier(&cma_nb); ret = ib_register_client(&cma_client); if (ret) @@ -2838,6 +2926,7 @@ static int cma_init(void) return 0; err: + unregister_netdevice_notifier(&cma_nb); rdma_addr_unregister_client(&addr_client); ib_sa_unregister_client(&sa_client); destroy_workqueue(cma_wq); @@ -2847,6 +2936,7 @@ err: static void cma_cleanup(void) { ib_unregister_client(&cma_client); + unregister_netdevice_notifier(&cma_nb); rdma_addr_unregister_client(&addr_client); ib_sa_unregister_client(&sa_client); destroy_workqueue(cma_wq); Index: infiniband/include/rdma/rdma_cm.h =================================================================== --- infiniband.orig/include/rdma/rdma_cm.h +++ infiniband/include/rdma/rdma_cm.h @@ -57,7 +57,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_ADDR_CHANGE }; enum rdma_port_space { From 46nichols at kbcom.com Tue Jun 17 01:33:39 2008 From: 46nichols at kbcom.com (hilton shigeo) Date: Tue, 17 Jun 2008 08:33:39 +0000 Subject: [ofa-general] Best of Adidas, Dsquared, UGG Message-ID: <000501c8d063$05b8d36c$0ccb7f90@tulnnwxs> The most popular and generous de luxe store for shoes and bags is just one click away. Estimated by tens thousands of grateful customers worldwide, we carry dozens of famous brands including: Prada Bally Bikkembergs Paul Smith UGG Here you willc see tens thousands of owsome designs for shoes, and leather products, at at temp't1ng priceE. Sale close this week, so visit web site today and start pampering yourself and your loved ones. - Visit our site: www.bemiatel[DOT]com (copy this link then replace "[DOT]" to ".") From ogerlitz at voltaire.com Tue Jun 17 03:40:30 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Jun 2008 13:40:30 +0300 Subject: [ofa-general] opensm routing In-Reply-To: <48577F6B.5020306@dev.mellanox.co.il> References: <484EFB6A.2030907@nasa.gov> <1213198145.5369.254.camel@cardanus.llnl.gov> <485000CC.8070502@nasa.gov> <1213289317.5369.279.camel@cardanus.llnl.gov> <485158D6.4050201@nasa.gov> <1213292984.5369.292.camel@cardanus.llnl.gov> <48516793.3090708@nasa.gov> <4856798A.8020006@dev.mellanox.co.il> <1213633003.5369.406.camel@cardanus.llnl.gov> <1213633548.14047.703.camel@hrosenstock-ws.xsigo.com> <20080616093515.7f4f61cc.weiny2@llnl.gov> <1213634338.14047.706.camel@hrosenstock-ws.xsigo.com> <20080616094629.557bf892.weiny2@llnl.gov> <1213634871.14047.708.camel@hrosenstock-ws.xsigo.com> <20080616095840.37c7e17b.weiny2@llnl.gov> <1213639746.14047.711.camel@hrosenstock-ws.xsigo.com> <48577F6B.5020306@dev.mellanox.co.il> Message-ID: <4857949E.8020009@voltaire.com> Yevgeny Kliteynik wrote: > Besides, if Lustre sends path queries to OpenSM (via RDMACM) when > opening their own QPs, then Lustre can use other SLs too. SM can even > assign different SLs to communication to Lustre metadata servers and > object storage servers if they are running on separate hosts, all > based on the path query. Indeed, since Lustre uses the RDMA-CM which in turn does its path query in a manner you can use either of the existing mechanisms in openSM to assign SL to this QP: 1. port group based (eg based on the SGID, DGID from the query) 2. pkey based (eg based on the PKEY) 3. SID based 4. TOS based 5. you name it... ~Everything happens under the cover of the rdma-cm, that is your app need to do nothing such that would be provided by the rdma-cm in the path query, and the rdma-cm uses the SL from the returned path when your app calls rdma_create_qp. Only if you want non zero TOS to be in the query, rdma_set_service_type has to be called before rdma_resolve_route. The approach is that the host provides the information, the IB sys admin design a QoS setting for the fabric and apply it through the mechanisms of the openSM, please see the cool presentation of Liran Liss from Sonoma @ http://www.openfabrics.org/archives/spring2008sonoma/Tuesday/qos_sonoma08_ofa_v1.ppt and let us know if you have any further questions. For MPIs which don't use path query the IB sys admin have instruct the users which SL to manually make the ranks of this job use, which is by far much less convenient. Or. From kliteyn at dev.mellanox.co.il Tue Jun 17 03:59:00 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 13:59:00 +0300 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <485772FD.4000206@dev.mellanox.co.il> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <4854CFFC.7040508@dev.mellanox.co.il> <1213636881.5369.438.camel@cardanus.llnl.gov> <1213646932.5369.446.camel@cardanus.llnl.gov> <485772FD.4000206@dev.mellanox.co.il> Message-ID: <485798F4.40201@dev.mellanox.co.il> Yevgeny Kliteynik wrote: > Hi Al, > > Al Chu wrote: >> On Mon, 2008-06-16 at 10:21 -0700, Al Chu wrote: >>> Hey Yevgeny, >>> >>> On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: >>>> Hi Al, >>>> >>>> Al Chu wrote: >>>>> Hey Sasha, >>>>> >>>>> This is a conceptually simple option I've developed for updn routing. >>>>> >>>>> Currently in updn routing, nodes/guids are routed on switches in a >>>>> seemingly-random order, which I believe is due to internal data >>>>> structure organization (i.e. cl_qmap_apply_func is called on >>>>> port_guid_tbl) as well as how the fabric is scanned (it is logically >>>>> scanned from a port perspective, but it may not be logical from a node >>>>> perspective). I had a hypothesis that this was leading to increased >>>>> contention in the network for MPI. >>>>> >>>>> For example, suppose we have 12 uplinks from a leaf switch to a spine >>>>> switch. If we want to send data from this leaf switch to node[13-24], >>>>> the up links we will send on are pretty random. It's because: >>>>> >>>>> A) node[13-24] are individually routed at seemingly-random points >>>>> based >>>>> on when they are called by cl_qmap_apply_func(). >>>>> >>>>> B) the ports chosen for routing are based on least used port usage. >>>>> >>>>> C) least used port usage is based on whatever was routed earlier on. >>>>> >>>>> So I developed this patch series, which supports an option called >>>>> "guid_routing_order_file" which allows the user to input a file with a >>>>> list of port_guids which will indicate the order in which guids are >>>>> routed instead (naturally, those guids not listed are routed last). >>>> Great idea! >>> Thanks. >>> >>>> I understand that this guid_routing_order_file is synchronized with >>>> an MPI rank file, right? If not, then synchronizing them might give >>>> even better results. >>> Not quite sure what you mean by a MPI rank file. At LLNL, slurm is >>> responsible for MPI ranks, so I order the guids in my file according to >>> how slurm is configured for chosing MPI ranks. I will admit to being a >>> novice to MPI's configuration (blindly accepting slurm MPI rankings). >>> Is there an underlying file that MPI libs use for ranking knowledge? >> >> I spoke to one of our MPI guys. I wasn't aware that in some MPIs you >> can input a file to tell it how ranks should be assigned to nodes for >> MPI. I assume that's what you're talking about? > > Yes, that is what I was talking about. > There is a host file, where you list all the hosts that MPI should use, > and in some MPIs there is also a way to specify the order of MPI ranks > that would be assigned to processes (I'm not an MPI expert, so I'm not > sure about the terminology that I use). > I know that MVAPICH is using the host order when assigning ranks, so > the order of the cluster nodes listed in host file is important. > Not sure about OpenMPI. > >>>> Another idea: OpenSM can create such file (list, doesn't have to be >>>> actual file) automatically, just by checking topologically-adjacent >>>> leaf switches and their HCAs. >>> Definitely a good idea. This patch set was just a "step one" kind of >>> thing. >>> >>>>> I list the port guids of the nodes of the cluster from node0 to >>>>> nodeN, one >>>>> per line in the file. By listing the nodes in this order, I >>>>> believe we >>>>> could get less contention in the network. In the example above, >>>>> sending >>>>> to node[13-24] should use all of the 12 uplinks, b/c the ports will be >>>>> equally used b/c nodes[1-12] were routed beforehand in order. >>>>> >>>>> The results from some tests are pretty impressive when I do this. >>>>> LMC=0 >>>>> average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s >>>>> when I use guid_routing_order. >>>> Can you compare this to the fat-tree routing? Conceptually, fat-tree >>>> is doing the same - it routes LIDs on nodes in a topological order, so >>>> it would be interesting to see the comparison. >>> Actually I already did :-). w/ LMC=0. >>> >>> updn default - 391.374 MB/s >>> updn w/ guid_routing_order - 573.678 MB/s >>> ftree - 579.603 MB/s >>> >>> I later discovered that one of the internal ports of the cluster I'm >>> testing on was broken (sLB of a 288 port), and think that is the cause >>> of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as >>> designed) seemed to be able to work around it properly, while updn (as >>> currently implemented) couldn't. >>> >>> When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were able >>> to do better on some tests than ftree. One example (I think these >>> numbers are in microseconds. Lower is better): >>> >>> Alltoall 16K packets >>> ftree - 415490.6919 >>> updn normal (LMC=0) - 495460.5526 >>> updn w/ ordered routing (LMC=0) - 416562.7417 >>> updn w/ ordered routing (LMC=1) - 453153.7289 >>> - this ^^^ result is quite odd. Not sure why. >>> updn w/ ordered routing (LMC=2) - 3660132.1530 >>> >>> We are regularly debating what will be better overall at the end of the >>> day. >>> >>>> Also, fat-tree produces the guid order file automatically, but nobody >>>> used it yet as an input to produce MPI rank file. >>> I didn't know about this option. How do you do this (just skimmed the >>> manpage, didn't see anything)? > > Right, it's missing there. I'll add this info. Nope, it's there: "The algorithm also dumps compute node ordering file (opensm-ftree-ca-order.dump) in the same directory where the OpenSM log resides. This ordering file provides the CN order that may be used to create efficient communication pattern, that will match the routing tables." -- Yevgeny > The file is /var/log/opensm-ftree-ca-order.dump. > Small correction though - the file contains ordered list of HCA LIDs > and their host names. It's not a problem to change it to have guids > as well, but MPI doesn't need guids anyway. > Note that the optimal order might be different depending on the current > topology state and the location of the management node that runs OpenSM. > >>> I know about the --cn_guid_file. But >>> since that file doesn't have to be ordered, that's why I created a >>> different option (rather than have the cn_guid_file for both ftree and >>> updn). > > Right, the cn file doesn't have to be ordered - ftree will order it > by itself. The ordering is by topology-adjacent leaf switches. > > -- Yevgeny > >>> >>> Al >>> >>>> -- Yevgeny >>>> >>>>> A variety of other positive performance >>>>> increases were found when doing other tests, other MPIs, and other >>>>> LMCs >>>>> if anyone is interested. >>>>> >>>>> BTW, I developed this patch series before your preserve-base-lid patch >>>>> series. It will 100% conflict with the preserve-base-lid patch >>>>> series. >>>>> I will fix this patch series once the preserve-base-lids patch >>>>> series is >>>>> committed to git. I'm just looking for comments right now. >>>>> >>>>> Al >>>>> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From kliteyn at dev.mellanox.co.il Tue Jun 17 04:05:42 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 14:05:42 +0300 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c : TRIVIAL fixes in log handling Message-ID: <48579A86.5070508@dev.mellanox.co.il> Hi Sasha, OSM_LOG_ENTER() at the beginning of the function and OSM_LOG_EXIT() at the end of the function were missing. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_ucast_ftree.c | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index dc61130..3ee2bdc 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -1260,6 +1260,8 @@ static void __osm_ftree_fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) FILE *p_hca_ordering_file; char *filename = "opensm-ftree-ca-order.dump"; + OSM_LOG_ENTER(&p_ftree->p_osm->log); + snprintf(path, sizeof(path), "%s/%s", p_ftree->p_osm->subn.opt.dump_files_dir, filename); p_hca_ordering_file = fopen(path, "w"); @@ -1267,8 +1269,7 @@ static void __osm_ftree_fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB01: " "cannot open file \'%s\': %s\n", filename, strerror(errno)); - OSM_LOG_EXIT(&p_ftree->p_osm->log); - return; + goto Exit; } /* for each leaf switch (in indexing order) */ @@ -1309,6 +1310,10 @@ static void __osm_ftree_fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) /* done going through all the leaf switches */ fclose(p_hca_ordering_file); + +Exit: + OSM_LOG_EXIT(&p_ftree->p_osm->log); + return; } /* __osm_ftree_fabric_dump_hca_ordering() */ /***************************************************/ -- 1.5.1.4 From jackm at dev.mellanox.co.il Tue Jun 17 04:11:02 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Tue, 17 Jun 2008 14:11:02 +0300 Subject: [ofa-general] [PATCH] ib_core: fix is_closed flag check in ib_uverbs_async_handler Message-ID: <200806171411.03001.jackm@dev.mellanox.co.il> ib_core: fix async event handler "is_closed" check The is_closed flag is checked incorrectly in ib_uverbs_async_handler(). As a result, no async events were passed to applications. Found by: Ronni Zimmerman Signed-off-by: Jack Morgenstein diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c index f806da1..caed42b 100644 --- a/drivers/infiniband/core/uverbs_main.c +++ b/drivers/infiniband/core/uverbs_main.c @@ -423,7 +423,7 @@ static void ib_uverbs_async_handler(struct ib_uverbs_file *file, unsigned long flags; spin_lock_irqsave(&file->async_file->lock, flags); - if (!file->async_file->is_closed) { + if (file->async_file->is_closed) { spin_unlock_irqrestore(&file->async_file->lock, flags); return; } From kliteyn at dev.mellanox.co.il Tue Jun 17 04:09:29 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 14:09:29 +0300 Subject: [ofa-general] [PATCH] opensm/man/opensm.8.in: TRIVIAL addition in fat-tree description Message-ID: <48579B69.2080903@dev.mellanox.co.il> Hi Sasha, Mentioning in the man page that fat-tree can be used on "symmetrical or almost symmetrical" trees, instead of the previous "symmetrical" only. Signed-off-by: Yevgeny Kliteynik --- opensm/man/opensm.8.in | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index d34b9c8..b5ffd24 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -539,10 +539,10 @@ loop in the subnet. 3. Fat Tree Unicast routing algorithm - this algorithm optimizes routing for congestion-free "shift" communication pattern. -It should be chosen if a subnet is a symmetrical Fat Trees of various types, -not just K-ary-N-Trees: non-constant K, not fully staffed, any Constant -Bisectional Bandwidth (CBB) ratio. Similar to UPDN, Fat Tree routing is -constrained to ranking rules. +It should be chosen if a subnet is a symmetrical or almost symmetrical +fat-tree of various types, not just K-ary-N-Trees: non-constant K, not +fully staffed, any Constant Bisectional Bandwidth (CBB) ratio. +Similar to UPDN, Fat Tree routing is constrained to ranking rules. 4. LASH unicast routing algorithm - uses Infiniband virtual layers (SL) to provide deadlock-free shortest-path routing while also -- 1.5.1.4 From hrosenstock at xsigo.com Tue Jun 17 05:06:44 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 05:06:44 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/opensm/osm_multicast.h: Cosmetic changes Message-ID: <1213704404.14047.723.camel@hrosenstock-ws.xsigo.com> opensm/include/opensm/osm_multicast.h: Cosmetic changes Formatting changes and fix typo Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index 57e5ef5..e79e288 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -81,7 +81,7 @@ BEGIN_C_DECLS * Steve King, Intel * *********/ -/****f* IBA Base: OpneSM: Multicast Group/osm_get_mcast_req_type_str +/****f* IBA Base: OpenSM: Multicast Group/osm_get_mcast_req_type_str * NAME * osm_get_mcast_req_type_str * @@ -126,15 +126,16 @@ typedef struct osm_mcast_mgr_ctxt { * FIELDS * * mlid -* The network ordered LID of this Multicast Group (must be >= 0xC000). +* The network ordered LID of this Multicast Group +* (must be >= 0xC000). * * req_type * The type of the request that caused this call -* (multicast create/join/leave). +* (multicast create/join/leave). * * port_guid * The port guid of the port that is being added/removed from -* the multicast group due to this call. +* the multicast group due to this call. * * SEE ALSO *********/ From john.russo at qlogic.com Tue Jun 17 05:57:38 2008 From: john.russo at qlogic.com (John Russo) Date: Tue, 17 Jun 2008 07:57:38 -0500 Subject: [ofa-general] QLogic vNIC Kernel Submission Message-ID: <99863D2ED484D449811D97A4C44C9CBD7C50F5@EPEXCH2.qlogic.org> QLogic has been attempting to submit our virtual NIC (vNIC) driver to the Linux kernel for several months. We have made changes to the code based on the feedback we have received over four rounds of submissions. Among the feedback we received during this process was a request to alter our code to use a single value per file for configuration of our driver through sysfs interface. After spending much time and effort to complete this change to our design we re-submitted the driver only to receive a response suggesting that we change once again from this interface to a different API interface called rtnl_link. Needless to say I am very frustrated with this process. This new API interface would require substantial changes to our code. QLogic has met the initial request to move to a single valued sysfs interface and I would hope that this new request will be waived and will not be a roadblock to inclusion of our driver to the kernel. __________________________ John F. Russo Manager, Engineering QLogic Corporation 780 Fifth Avenue, Suite 140 King of Prussia, PA 19406 Direct: 610-233-4866 Main: 610-233-4800 Fax: 610-233-4777 Cell: 610-246-9903 Email: John.Russo at qlogic.com www.qlogic.com True success is the undeniable truth that we have proved ourselves. -Joe Luppino-Esposito -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 3677 bytes Desc: image001.jpg URL: From hrosenstock at xsigo.com Tue Jun 17 05:58:01 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 05:58:01 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_sa_mcmember_record.c: Cosmetic changes Message-ID: <1213707481.14047.731.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Cosmetic changes Cosmetic changes to log message, formatting, and comments Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index caa50d7..5ddefa6 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -244,7 +244,7 @@ __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) osm_mgrp_t *p_mgrp = __get_mgrp_by_mlid(sa, mlid); /* Remove MGRP only if osm_mcm_port_t count is 0 and - * Not a well known group */ + not a well known group */ if (p_mgrp && cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && p_mgrp->well_known == FALSE) { cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, @@ -1039,7 +1039,7 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, (osm_mgrp_t *) cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Found previous group for mlid:0x%04x - " - "Need to destroy it\n", + "Destroying it first\n", cl_ntoh16(mlid)); cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, (cl_map_item_t *) p_prev_mgrp); @@ -1665,8 +1665,8 @@ __osm_sa_mcm_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, sizeof(uint16_t))) goto Exit; - /* if the requester physical port doesn't have the pkey that is defined for - the group - exit. */ + /* if the requester physical port doesn't have the pkey that is defined + for the group - exit. */ if (!osm_physp_has_pkey(sa->p_log, p_mgrp->mcmember_rec.pkey, p_req_physp)) goto Exit; From hrosenstock at xsigo.com Tue Jun 17 06:02:59 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 06:02:59 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_mcast_mgr.c: Cosmetic comment format change Message-ID: <1213707779.14047.735.camel@hrosenstock-ws.xsigo.com> opensm/osm_mcast_mgr.c: Cosmetic comment format change Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index 75afbbb..edfaabf 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -1191,9 +1191,11 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) p_mgrp = (osm_mgrp_t *) cl_qmap_head(p_mcast_tbl); while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(p_mcast_tbl)) { - /* We reached here due to some change that caused a heavy sweep + /* + We reached here due to some change that caused a heavy sweep of the subnet. Not due to a specific multicast request. - So the request type is subnet_change and the port guid is 0. */ + So the request type is subnet_change and the port guid is 0. + */ p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); mcast_mgr_process_mgrp(sm, p_mgrp, OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, 0); From ogerlitz at voltaire.com Tue Jun 17 06:03:24 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Jun 2008 16:03:24 +0300 (IDT) Subject: [ofa-general] local-sa diff between ofed 1.2 and 1.3 Message-ID: Hi Sean, I took a look on the local-sa and it turns that its been through quite a big change between the last two ofed releases (below). Does this diff relates to the structural change of the local-sa being now embedded in the ib-sa module? is it documented somewhere (eg by change-log of patch/es in a git tree)? looking at your rdma-dev git tree I didn't evidence to this change... Or. --- OFED-1.2/SOURCES/ofa_kernel-1.2/drivers/infiniband/core/local_sa.c 2008-06-17 07:43:55.000000000 +0000 +++ ofed-1.3.1/OFED-1.3.1/SOURCES/ofa_kernel-1.3.1/drivers/infiniband/core/local_sa.c 2008-06-17 07:41:13.000000000 +0000 @@ -1,5 +1,5 @@ /* - * Copyright (c) 2006 Intel Corporation.  All rights reserved. + * Copyright (c) 2006 Intel Corporation.  All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -34,113 +34,162 @@ #include #include #include -#include +#include +#include #include +#include +#include #include -#include +#include +#include "sa.h" MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("InfiniBand subnet administration caching"); MODULE_LICENSE("Dual BSD/GPL"); -static int retry_timer = 5000; /* 5 sec */ -module_param(retry_timer, int, 0444); -MODULE_PARM_DESC(retry_timer, "Time in ms between retried requests."); - -static int retries = 3; -module_param(retries, int, 0444); -MODULE_PARM_DESC(retries, "Number of times to retry a request."); - -static unsigned long cache_timeout = 15 * 60 * 1000; /* 15 min */ -module_param(cache_timeout, ulong, 0444); -MODULE_PARM_DESC(cache_timeout, "Time in ms between cache updates. " - "Set to 0 to disable cache."); - -static unsigned long hold_time = 30 * 1000; /* 30 sec */ -module_param(hold_time, ulong, 0444); -MODULE_PARM_DESC(hold_timer, "Minimal time in ms between cache updates."); - -static unsigned long update_delay = 3000; /* 3 sec */ -module_param(update_delay, ulong, 0444); -MODULE_PARM_DESC(update_delay, "Delay in ms between an event and an update."); - enum { - IB_MAX_PATHS_PER_DEST = 0x7F + SA_DB_MAX_PATHS_PER_DEST = 0x7F, + SA_DB_MIN_RETRY_TIMER = 4000, /* 4 sec */ + SA_DB_MAX_RETRY_TIMER = 256000 /* 256 sec */ }; -static unsigned long paths_per_dest = IB_MAX_PATHS_PER_DEST; -module_param(paths_per_dest, ulong, 0444); +static int set_paths_per_dest(const char *val, struct kernel_param *kp); +static unsigned long paths_per_dest = 0; +module_param_call(paths_per_dest, set_paths_per_dest, param_get_ulong, + &paths_per_dest, 0644); MODULE_PARM_DESC(paths_per_dest, "Maximum number of paths to retrieve " "to each destination (DGID). Set to 0 " "to disable cache."); -static void sa_db_add_one(struct ib_device *device); -static void sa_db_remove_one(struct ib_device *device); +static int set_subscribe_inform_info(const char *val, struct kernel_param *kp); +static char subscribe_inform_info = 1; +module_param_call(subscribe_inform_info, set_subscribe_inform_info, + param_get_bool, &subscribe_inform_info, 0644); +MODULE_PARM_DESC(subscribe_inform_info, + "Subscribe for SA InformInfo/Notice events."); + +static int do_refresh(const char *val, struct kernel_param *kp); +module_param_call(refresh, do_refresh, NULL, NULL, 0200); + +static unsigned long retry_timer = SA_DB_MIN_RETRY_TIMER; + +enum sa_db_lookup_method { + SA_DB_LOOKUP_LEAST_USED, + SA_DB_LOOKUP_RANDOM +}; + +static int set_lookup_method(const char *val, struct kernel_param *kp); +static int get_lookup_method(char *buf, struct kernel_param *kp); +static unsigned long lookup_method; +module_param_call(lookup_method, set_lookup_method, get_lookup_method, + &lookup_method, 0644); +MODULE_PARM_DESC(lookup_method, "Method used to return path records when " + "multiple paths exist to a given destination."); + +static void sa_db_add_dev(struct ib_device *device); +static void sa_db_remove_dev(struct ib_device *device); static struct ib_client sa_db_client = { .name = "local_sa", - .add = sa_db_add_one, - .remove = sa_db_remove_one + .add = sa_db_add_dev, + .remove = sa_db_remove_dev }; static LIST_HEAD(dev_list); -static DECLARE_RWSEM(lock); -static unsigned long hold_time, update_delay; +static DEFINE_MUTEX(lock); +static rwlock_t rwlock; static struct workqueue_struct *sa_wq; +static struct ib_sa_client sa_client; + +enum sa_db_state { + SA_DB_IDLE, + SA_DB_REFRESH, + SA_DB_DESTROY +}; struct sa_db_port { - struct sa_db_device *dev; - struct ib_mad_agent *agent; - struct rb_root paths; - unsigned long update_time; - int update; - struct delayed_work work; - union ib_gid gid; - int port_num; + struct sa_db_device *dev; + struct ib_mad_agent *agent; + /* Limit number of outstanding MADs to SA to reduce SA flooding */ + struct ib_mad_send_buf *msg; + u16 sm_lid; + u8 sm_sl; + struct ib_inform_info *in_info; + struct ib_inform_info *out_info; + struct rb_root paths; + struct list_head update_list; + unsigned long update_id; + enum sa_db_state state; + struct work_struct work; + union ib_gid gid; + int port_num; }; struct sa_db_device { - struct list_head list; - struct ib_device *device; + struct list_head list; + struct ib_device *device; struct ib_event_handler event_handler; - struct sa_db_port port[0]; + int start_port; + int port_count; + struct sa_db_port port[0]; }; struct ib_sa_iterator { struct ib_sa_iterator *next; }; +struct ib_sa_attr_iter { + struct ib_sa_iterator *iter; + unsigned long flags; +}; + struct ib_sa_attr_list { struct ib_sa_iterator iter; struct ib_sa_iterator *tail; - int update; + int update_id; union ib_gid gid; struct rb_node node; }; struct ib_path_rec_info { - struct ib_sa_iterator iter; /* keep first for ib_get_next_sa_attr */ + struct ib_sa_iterator iter; /* keep first */ struct ib_sa_path_rec rec; + unsigned long lookups; }; -struct ib_sa_iter { - struct ib_mad_recv_wc *recv_wc; - struct ib_mad_recv_buf *recv_buf; - int attr_size; - int attr_offset; - int data_offset; - int data_left; - void *attr; - u8 attr_data[0]; +struct ib_sa_mad_iter { + struct ib_mad_recv_wc *recv_wc; + struct ib_mad_recv_buf *recv_buf; + int attr_size; + int attr_offset; + int data_offset; + int data_left; + void *attr; + u8 attr_data[0]; }; -static void send_handler(struct ib_mad_agent *agent, - struct ib_mad_send_wc *mad_send_wc) -{ - ib_destroy_ah(mad_send_wc->send_buf->ah); - ib_free_send_mad(mad_send_wc->send_buf); -} +enum sa_update_type { + SA_UPDATE_FULL, + SA_UPDATE_ADD, + SA_UPDATE_REMOVE +}; + +struct update_info { + struct list_head list; + union ib_gid gid; + enum sa_update_type type; +}; + +struct sa_path_request { + struct work_struct work; + struct ib_sa_client *client; + void (*callback)(int, struct ib_sa_path_rec *, void *); + void *context; + struct ib_sa_path_rec path_rec; +}; + +static void process_updates(struct sa_db_port *port); static void free_attr_list(struct ib_sa_attr_list *attr_list) { @@ -153,23 +202,44 @@ static void free_attr_list(struct ib_sa_ attr_list->tail = &attr_list->iter; } +static void remove_attr(struct rb_root *root, struct ib_sa_attr_list *attr_list) +{ + rb_erase(&attr_list->node, root); + free_attr_list(attr_list); + kfree(attr_list); +} + static void remove_all_attrs(struct rb_root *root) { struct rb_node *node, *next_node; struct ib_sa_attr_list *attr_list; + write_lock_irq(&rwlock); for (node = rb_first(root); node; node = next_node) { next_node = rb_next(node); - rb_erase(node, root); attr_list = rb_entry(node, struct ib_sa_attr_list, node); - free_attr_list(attr_list); - kfree(attr_list); + remove_attr(root, attr_list); } + write_unlock_irq(&rwlock); } -static struct ib_sa_attr_list * insert_attr_list(struct rb_root *root, - struct ib_sa_attr_list - *attr_list) +static void remove_old_attrs(struct rb_root *root, unsigned long update_id) +{ + struct rb_node *node, *next_node; + struct ib_sa_attr_list *attr_list; + + write_lock_irq(&rwlock); + for (node = rb_first(root); node; node = next_node) { + next_node = rb_next(node); + attr_list = rb_entry(node, struct ib_sa_attr_list, node); + if (attr_list->update_id != update_id) + remove_attr(root, attr_list); + } + write_unlock_irq(&rwlock); +} + +static struct ib_sa_attr_list *insert_attr_list(struct rb_root *root, + struct ib_sa_attr_list *attr_list) { struct rb_node **link = &root->rb_node; struct rb_node *parent = NULL; @@ -193,8 +263,7 @@ static struct ib_sa_attr_list * insert_a return NULL; } -static struct ib_sa_attr_list * find_attr_list(struct rb_root *root, - u8 *gid) +static struct ib_sa_attr_list *find_attr_list(struct rb_root *root, u8 *gid) { struct rb_node *node = root->rb_node; struct ib_sa_attr_list *attr_list; @@ -213,46 +282,47 @@ static struct ib_sa_attr_list * find_att return NULL; } -static int insert_attr(struct rb_root *root, int update, void *key, +static int insert_attr(struct rb_root *root, unsigned long update_id, void *key, struct ib_sa_iterator *iter) { struct ib_sa_attr_list *attr_list; void *err; + write_lock_irq(&rwlock); attr_list = find_attr_list(root, key); if (!attr_list) { + write_unlock_irq(&rwlock); attr_list = kmalloc(sizeof *attr_list, GFP_KERNEL); if (!attr_list) return -ENOMEM; attr_list->iter.next = NULL; attr_list->tail = &attr_list->iter; - attr_list->update = update; + attr_list->update_id = update_id; memcpy(attr_list->gid.raw, key, sizeof attr_list->gid); + write_lock_irq(&rwlock); err = insert_attr_list(root, attr_list); if (err) { + write_unlock_irq(&rwlock); kfree(attr_list); return PTR_ERR(err); } - } else if (attr_list->update != update) { + } else if (attr_list->update_id != update_id) { free_attr_list(attr_list); - attr_list->update = update; + attr_list->update_id = update_id; } - /* - * Assume that the SA returned the best attribute first, and insert - * attributes on the tail. - */ attr_list->tail->next = iter; iter->next = NULL; attr_list->tail = iter; + write_unlock_irq(&rwlock); return 0; } -static struct ib_sa_iter *ib_sa_iter_create(struct ib_mad_recv_wc *mad_recv_wc) +static struct ib_sa_mad_iter *ib_sa_iter_create(struct ib_mad_recv_wc *mad_recv_wc) { - struct ib_sa_iter *iter; + struct ib_sa_mad_iter *iter; struct ib_sa_mad *mad = (struct ib_sa_mad *) mad_recv_wc->recv_buf.mad; int attr_size, attr_offset; @@ -273,12 +343,12 @@ static struct ib_sa_iter *ib_sa_iter_cre return iter; } -static void ib_sa_iter_free(struct ib_sa_iter *iter) +static void ib_sa_iter_free(struct ib_sa_mad_iter *iter) { kfree(iter); } -static void *ib_sa_iter_next(struct ib_sa_iter *iter) +static void *ib_sa_iter_next(struct ib_sa_mad_iter *iter) { struct ib_sa_mad *mad; int left, offset = 0; @@ -319,67 +389,48 @@ out: } /* - * Copy a path record from a received MAD and insert it into our index. - * The path record in the MAD is in network order, so must be swapped. It - * can also span multiple MADs, just to make our life hard. + * Copy path records from a received response and insert them into our cache. + * A path record in the MADs are in network order, packed, and may + * span multiple MAD buffers, just to make our life hard. */ -static void update_path_rec(struct sa_db_port *port, - struct ib_mad_recv_wc *mad_recv_wc) +static void update_path_db(struct sa_db_port *port, + struct ib_mad_recv_wc *mad_recv_wc, + enum sa_update_type type) { - struct ib_sa_iter *iter; + struct ib_sa_mad_iter *iter; struct ib_path_rec_info *path_info; void *attr; + int ret; iter = ib_sa_iter_create(mad_recv_wc); if (IS_ERR(iter)) return; - down_write(&lock); - port->update++; + port->update_id += (type == SA_UPDATE_FULL); + while ((attr = ib_sa_iter_next(iter)) && (path_info = kmalloc(sizeof *path_info, GFP_KERNEL))) { ib_sa_unpack_attr(&path_info->rec, attr, IB_SA_ATTR_PATH_REC); - if (insert_attr(&port->paths, port->update, - path_info->rec.dgid.raw, - &path_info->iter)) { + + ret = insert_attr(&port->paths, port->update_id, + path_info->rec.dgid.raw, &path_info->iter); + if (ret) { kfree(path_info); break; } } - up_write(&lock); ib_sa_iter_free(iter); -} - -static void recv_handler(struct ib_mad_agent *mad_agent, - struct ib_mad_recv_wc *mad_recv_wc) -{ - struct ib_sa_mad *mad = (void *) mad_recv_wc->recv_buf.mad; - - if (mad->mad_hdr.status) - goto done; - switch (cpu_to_be16(mad->mad_hdr.attr_id)) { - case IB_SA_ATTR_PATH_REC: - update_path_rec(mad_agent->context, mad_recv_wc); - break; - default: - break; - } -done: - ib_free_recv_mad(mad_recv_wc); + if (type == SA_UPDATE_FULL) + remove_old_attrs(&port->paths, port->update_id); } -static struct ib_mad_send_buf* get_sa_msg(struct sa_db_port *port) +static struct ib_mad_send_buf *get_sa_msg(struct sa_db_port *port, + struct update_info *update) { - struct ib_port_attr port_attr; - struct ib_ah_attr ah_attr; - struct ib_mad_send_buf *msg; - int ret; - - ret = ib_query_port(port->dev->device, port->port_num, &port_attr); - if (ret || port_attr.state != IB_PORT_ACTIVE) - return NULL; + struct ib_ah_attr ah_attr; + struct ib_mad_send_buf *msg; msg = ib_create_send_mad(port->agent, 1, 0, 0, IB_MGMT_SA_HDR, IB_MGMT_SA_DATA, GFP_KERNEL); @@ -387,8 +438,8 @@ static struct ib_mad_send_buf* get_sa_ms return NULL; memset(&ah_attr, 0, sizeof ah_attr); - ah_attr.dlid = port_attr.sm_lid; - ah_attr.sl = port_attr.sm_sl; + ah_attr.dlid = port->sm_lid; + ah_attr.sl = port->sm_sl; ah_attr.port_num = port->port_num; msg->ah = ib_create_ah(port->agent->qp->pd, &ah_attr); @@ -398,8 +449,9 @@ static struct ib_mad_send_buf* get_sa_ms } msg->timeout_ms = retry_timer; - msg->retries = retries; + msg->retries = 0; msg->context[0] = port; + msg->context[1] = update; return msg; } @@ -411,6 +463,7 @@ static __be64 form_tid(u32 hi_tid) } static void format_path_req(struct sa_db_port *port, + struct update_info *update, struct ib_mad_send_buf *msg) { struct ib_sa_mad *mad = msg->mad; @@ -426,198 +479,749 @@ static void format_path_req(struct sa_db mad->sa_hdr.comp_mask = IB_SA_PATH_REC_SGID | IB_SA_PATH_REC_NUMB_PATH; path_rec.sgid = port->gid; - path_rec.numb_path = paths_per_dest; + path_rec.numb_path = (u8) paths_per_dest; + + if (update->type == SA_UPDATE_ADD) { + mad->sa_hdr.comp_mask |= IB_SA_PATH_REC_DGID; + memcpy(&path_rec.dgid, &update->gid, sizeof path_rec.dgid); + } + ib_sa_pack_attr(mad->data, &path_rec, IB_SA_ATTR_PATH_REC); } -static void update_cache(struct work_struct *work) +static int send_query(struct sa_db_port *port, + struct update_info *update) { - struct sa_db_port *port; - struct ib_mad_send_buf *msg; + int ret; - port = container_of(work, typeof(*port), work.work); - msg = get_sa_msg(port); - if (!msg) - return; + port->msg = get_sa_msg(port, update); + if (!port->msg) + return -ENOMEM; - format_path_req(port, msg); + format_path_req(port, update, port->msg); - if (ib_post_send_mad(msg, NULL)) { - ib_destroy_ah(msg->ah); - ib_free_send_mad(msg); - return; + ret = ib_post_send_mad(port->msg, NULL); + if (ret) + goto err; + + return 0; + +err: + ib_destroy_ah(port->msg->ah); + ib_free_send_mad(port->msg); + return ret; +} + +static void add_update(struct sa_db_port *port, u8 *gid, + enum sa_update_type type) +{ + struct update_info *update; + + update = kmalloc(sizeof *update, GFP_KERNEL); + if (update) { + if (gid) + memcpy(&update->gid, gid, sizeof update->gid); + update->type = type; + list_add(&update->list, &port->update_list); } - /* - * We record the time that we requested the update, rather than use the - * time that the update occurred. This allows us to generate a new - * update if an event occurs while we're still processing this one. - */ - port->update_time = jiffies; - queue_delayed_work(sa_wq, &port->work, cache_timeout); + if (port->state == SA_DB_IDLE) { + port->state = SA_DB_REFRESH; + process_updates(port); + } } -static void schedule_update(struct sa_db_port *port) +static void clean_update_list(struct sa_db_port *port) { - unsigned long time, delay; + struct update_info *update; - if (!paths_per_dest) + while (!list_empty(&port->update_list)) { + update = list_entry(port->update_list.next, + struct update_info, list); + list_del(&update->list); + kfree(update); + } +} + +static int notice_handler(int status, struct ib_inform_info *info, + struct ib_sa_notice *notice) +{ + struct sa_db_port *port = info->context; + struct ib_sa_notice_data_gid *gid_data; + struct ib_inform_info **pinfo; + enum sa_update_type type; + + if (info->trap_number == IB_SA_SM_TRAP_GID_IN_SERVICE) { + pinfo = &port->in_info; + type = SA_UPDATE_ADD; + } else { + pinfo = &port->out_info; + type = SA_UPDATE_REMOVE; + } + + mutex_lock(&lock); + if (port->state == SA_DB_DESTROY || !*pinfo) { + mutex_unlock(&lock); + return 0; + } + + if (notice) { + gid_data = (struct ib_sa_notice_data_gid *) + ¬ice->data_details; + add_update(port, gid_data->gid, type); + mutex_unlock(&lock); + } else if (status == -ENETRESET) { + *pinfo = NULL; + mutex_unlock(&lock); + } else { + if (status) + *pinfo = ERR_PTR(-EINVAL); + port->state = SA_DB_IDLE; + clean_update_list(port); + mutex_unlock(&lock); + queue_work(sa_wq, &port->work); + } + + return status; +} + +static int reg_in_info(struct sa_db_port *port) +{ + int ret = 0; + + port->in_info = ib_sa_register_inform_info(&sa_client, + port->dev->device, + port->port_num, + IB_SA_SM_TRAP_GID_IN_SERVICE, + GFP_KERNEL, notice_handler, + port); + if (IS_ERR(port->in_info)) + ret = PTR_ERR(port->in_info); + + return ret; +} + +static int reg_out_info(struct sa_db_port *port) +{ + int ret = 0; + + port->out_info = ib_sa_register_inform_info(&sa_client, + port->dev->device, + port->port_num, + IB_SA_SM_TRAP_GID_OUT_OF_SERVICE, + GFP_KERNEL, notice_handler, + port); + if (IS_ERR(port->out_info)) + ret = PTR_ERR(port->out_info); + + return ret; +} + +static void unsubscribe_port(struct sa_db_port *port) +{ + if (port->in_info && !IS_ERR(port->in_info)) + ib_sa_unregister_inform_info(port->in_info); + + if (port->out_info && !IS_ERR(port->out_info)) + ib_sa_unregister_inform_info(port->out_info); + + port->out_info = NULL; + port->in_info = NULL; + +} + +static void cleanup_port(struct sa_db_port *port) +{ + unsubscribe_port(port); + + clean_update_list(port); + remove_all_attrs(&port->paths); +} + +static int update_port_info(struct sa_db_port *port) +{ + struct ib_port_attr port_attr; + int ret; + + ret = ib_query_port(port->dev->device, port->port_num, &port_attr); + if (ret) + return ret; + + if (port_attr.state != IB_PORT_ACTIVE) + return -ENODATA; + + port->sm_lid = port_attr.sm_lid; + port->sm_sl = port_attr.sm_sl; + return 0; +} + +static void process_updates(struct sa_db_port *port) +{ + struct update_info *update; + struct ib_sa_attr_list *attr_list; + int ret; + + if (!paths_per_dest || update_port_info(port)) { + cleanup_port(port); + goto out; + } + + /* Event registration is an optimization, so ignore failures. */ + if (subscribe_inform_info) { + if (!port->out_info) { + ret = reg_out_info(port); + if (!ret) + return; + } + + if (!port->in_info) { + ret = reg_in_info(port); + if (!ret) + return; + } + } else + unsubscribe_port(port); + + while (!list_empty(&port->update_list)) { + update = list_entry(port->update_list.next, + struct update_info, list); + + if (update->type == SA_UPDATE_REMOVE) { + write_lock_irq(&rwlock); + attr_list = find_attr_list(&port->paths, + update->gid.raw); + if (attr_list) + remove_attr(&port->paths, attr_list); + write_unlock_irq(&rwlock); + } else { + ret = send_query(port, update); + if (!ret) + return; + + } + list_del(&update->list); + kfree(update); + } +out: + port->state = SA_DB_IDLE; +} + +static void refresh_port_db(struct sa_db_port *port) +{ + if (port->state == SA_DB_DESTROY) return; - time = jiffies; - if (time_after(time, port->update_time + hold_time)) - delay = update_delay; - else - delay = port->update_time + hold_time - time; + if (port->state == SA_DB_REFRESH) { + clean_update_list(port); + ib_cancel_mad(port->agent, port->msg); + } - cancel_delayed_work(&port->work); - queue_delayed_work(sa_wq, &port->work, delay); + add_update(port, NULL, SA_UPDATE_FULL); } -static void handle_event(struct ib_event_handler *event_handler, - struct ib_event *event) +static void refresh_dev_db(struct sa_db_device *dev) +{ + int i; + + for (i = 0; i < dev->port_count; i++) + refresh_port_db(&dev->port[i]); +} + +static void refresh_db(void) { struct sa_db_device *dev; - dev = container_of(event_handler, typeof(*dev), event_handler); - if (event->event == IB_EVENT_PORT_ERR || - event->event == IB_EVENT_PORT_ACTIVE || - event->event == IB_EVENT_LID_CHANGE || - event->event == IB_EVENT_PKEY_CHANGE || - event->event == IB_EVENT_SM_CHANGE || - event->event == IB_EVENT_CLIENT_REREGISTER) - schedule_update(&dev->port[event->element.port_num - 1]); + list_for_each_entry(dev, &dev_list, list) + refresh_dev_db(dev); } -int ib_get_path_rec(struct ib_device *device, u8 port_num, union ib_gid *sgid, - union ib_gid *dgid, u16 pkey, struct ib_sa_path_rec *rec) +static int do_refresh(const char *val, struct kernel_param *kp) { - struct ib_sa_iterator *iter; - struct ib_sa_path_rec *path; - int ret = -ENODATA; + mutex_lock(&lock); + refresh_db(); + mutex_unlock(&lock); + return 0; +} - iter = ib_create_path_iter(device, port_num, dgid); - if (IS_ERR(iter)) - return PTR_ERR(iter); +static int get_lookup_method(char *buf, struct kernel_param *kp) +{ + return sprintf(buf, + "%c %d round robin\n" + "%c %d random", + (lookup_method == SA_DB_LOOKUP_LEAST_USED) ? '*' : ' ', + SA_DB_LOOKUP_LEAST_USED, + (lookup_method == SA_DB_LOOKUP_RANDOM) ? '*' : ' ', + SA_DB_LOOKUP_RANDOM); +} - for (path = ib_get_next_sa_attr(&iter); path; - path = ib_get_next_sa_attr(&iter)) { - if (pkey == path->pkey && - !memcmp(sgid, path->sgid.raw, sizeof *sgid)) { - memcpy(rec, path, sizeof *rec); - ret = 0; - break; - } +static int set_lookup_method(const char *val, struct kernel_param *kp) +{ + unsigned long method; + int ret = 0; + + method = simple_strtoul(val, NULL, 0); + + switch (method) { + case SA_DB_LOOKUP_LEAST_USED: + case SA_DB_LOOKUP_RANDOM: + lookup_method = method; + break; + default: + ret = -EINVAL; + break; } - ib_free_sa_iter(iter); return ret; } -EXPORT_SYMBOL(ib_get_path_rec); -struct ib_sa_iterator *ib_create_path_iter(struct ib_device *device, - u8 port_num, union ib_gid *dgid) +static int set_paths_per_dest(const char *val, struct kernel_param *kp) +{ + int ret; + + mutex_lock(&lock); + ret = param_set_ulong(val, kp); + if (ret) + goto out; + + if (paths_per_dest > SA_DB_MAX_PATHS_PER_DEST) + paths_per_dest = SA_DB_MAX_PATHS_PER_DEST; + refresh_db(); +out: + mutex_unlock(&lock); + return ret; +} + +static int set_subscribe_inform_info(const char *val, struct kernel_param *kp) +{ + int ret; + + ret = param_set_bool(val, kp); + if (ret) + return ret; + + return do_refresh(val, kp); +} + +static void port_work_handler(struct work_struct *work) +{ + struct sa_db_port *port; + + port = container_of(work, typeof(*port), work); + mutex_lock(&lock); + refresh_port_db(port); + mutex_unlock(&lock); +} + +static void handle_event(struct ib_event_handler *event_handler, + struct ib_event *event) +{ + struct sa_db_device *dev; + struct sa_db_port *port; + + dev = container_of(event_handler, typeof(*dev), event_handler); + port = &dev->port[event->element.port_num - dev->start_port]; + + switch (event->event) { + case IB_EVENT_PORT_ERR: + case IB_EVENT_LID_CHANGE: + case IB_EVENT_SM_CHANGE: + case IB_EVENT_CLIENT_REREGISTER: + case IB_EVENT_PKEY_CHANGE: + case IB_EVENT_PORT_ACTIVE: + queue_work(sa_wq, &port->work); + break; + default: + break; + } +} + +static void ib_free_path_iter(struct ib_sa_attr_iter *iter) +{ + read_unlock_irqrestore(&rwlock, iter->flags); +} + +static int ib_create_path_iter(struct ib_device *device, u8 port_num, + union ib_gid *dgid, struct ib_sa_attr_iter *iter) { struct sa_db_device *dev; struct sa_db_port *port; struct ib_sa_attr_list *list; - int ret; - down_read(&lock); dev = ib_get_client_data(device, &sa_db_client); - if (!dev) { - ret = -ENODEV; - goto err; - } - port = &dev->port[port_num - 1]; + if (!dev) + return -ENODEV; + port = &dev->port[port_num - dev->start_port]; + + read_lock_irqsave(&rwlock, iter->flags); list = find_attr_list(&port->paths, dgid->raw); if (!list) { - ret = -ENODATA; - goto err; + ib_free_path_iter(iter); + return -ENODATA; } - return &list->iter; -err: - up_read(&lock); - return ERR_PTR(ret); + iter->iter = &list->iter; + return 0; +} + +static struct ib_sa_path_rec *ib_get_next_path(struct ib_sa_attr_iter *iter) +{ + struct ib_path_rec_info *next_path; + + iter->iter = iter->iter->next; + if (iter->iter) { + next_path = container_of(iter->iter, struct ib_path_rec_info, iter); + return &next_path->rec; + } else + return NULL; } -EXPORT_SYMBOL(ib_create_path_iter); -void ib_free_sa_iter(struct ib_sa_iterator *iter) +static int cmp_rec(struct ib_sa_path_rec *src, + struct ib_sa_path_rec *dst, ib_sa_comp_mask comp_mask) { - up_read(&lock); + /* DGID check already done */ + if (comp_mask & IB_SA_PATH_REC_SGID && + memcmp(&src->sgid, &dst->sgid, sizeof src->sgid)) + return -EINVAL; + if (comp_mask & IB_SA_PATH_REC_DLID && src->dlid != dst->dlid) + return -EINVAL; + if (comp_mask & IB_SA_PATH_REC_SLID && src->slid != dst->slid) + return -EINVAL; + if (comp_mask & IB_SA_PATH_REC_RAW_TRAFFIC && + src->raw_traffic != dst->raw_traffic) + return -EINVAL; + + if (comp_mask & IB_SA_PATH_REC_FLOW_LABEL && + src->flow_label != dst->flow_label) + return -EINVAL; + if (comp_mask & IB_SA_PATH_REC_HOP_LIMIT && + src->hop_limit != dst->hop_limit) + return -EINVAL; + if (comp_mask & IB_SA_PATH_REC_TRAFFIC_CLASS && + src->traffic_class != dst->traffic_class) + return -EINVAL; + if (comp_mask & IB_SA_PATH_REC_REVERSIBLE && + dst->reversible && !src->reversible) + return -EINVAL; + /* Numb path check already done */ + if (comp_mask & IB_SA_PATH_REC_PKEY && src->pkey != dst->pkey) + return -EINVAL; + + if (comp_mask & IB_SA_PATH_REC_SL && src->sl != dst->sl) + return -EINVAL; + + if (ib_sa_check_selector(comp_mask, IB_SA_PATH_REC_MTU_SELECTOR, + IB_SA_PATH_REC_MTU, dst->mtu_selector, + src->mtu, dst->mtu)) + return -EINVAL; + if (ib_sa_check_selector(comp_mask, IB_SA_PATH_REC_RATE_SELECTOR, + IB_SA_PATH_REC_RATE, dst->rate_selector, + src->rate, dst->rate)) + return -EINVAL; + if (ib_sa_check_selector(comp_mask, + IB_SA_PATH_REC_PACKET_LIFE_TIME_SELECTOR, + IB_SA_PATH_REC_PACKET_LIFE_TIME, + dst->packet_life_time_selector, + src->packet_life_time, dst->packet_life_time)) + return -EINVAL; + + return 0; +} + +static struct ib_sa_path_rec *get_random_path(struct ib_sa_attr_iter *iter, + struct ib_sa_path_rec *req_path, + ib_sa_comp_mask comp_mask) +{ + struct ib_sa_path_rec *path, *rand_path = NULL; + int num, count = 0; + + for (path = ib_get_next_path(iter); path; + path = ib_get_next_path(iter)) { + if (!cmp_rec(path, req_path, comp_mask)) { + get_random_bytes(&num, sizeof num); + if ((num % ++count) == 0) + rand_path = path; + } + } + + return rand_path; } -EXPORT_SYMBOL(ib_free_sa_iter); -void *ib_get_next_sa_attr(struct ib_sa_iterator **iter) +static struct ib_sa_path_rec *get_next_path(struct ib_sa_attr_iter *iter, + struct ib_sa_path_rec *req_path, + ib_sa_comp_mask comp_mask) { - *iter = (*iter)->next; - if (*iter) - return ((void *)(*iter)) + sizeof(**iter); - else + struct ib_path_rec_info *cur_path, *next_path = NULL; + struct ib_sa_path_rec *path; + unsigned long lookups = ~0; + + for (path = ib_get_next_path(iter); path; + path = ib_get_next_path(iter)) { + if (!cmp_rec(path, req_path, comp_mask)) { + + cur_path = container_of(iter->iter, struct ib_path_rec_info, + iter); + if (cur_path->lookups < lookups) { + lookups = cur_path->lookups; + next_path = cur_path; + } + } + } + + if (next_path) { + next_path->lookups++; + return &next_path->rec; + } else return NULL; } -EXPORT_SYMBOL(ib_get_next_sa_attr); -static void sa_db_add_one(struct ib_device *device) +static void report_path(struct work_struct *work) +{ + struct sa_path_request *req; + + req = container_of(work, struct sa_path_request, work); + req->callback(0, &req->path_rec, req->context); + ib_sa_client_put(req->client); + kfree(req); +} + +/** + * ib_sa_path_rec_get - Start a Path get query + * @client:SA client + * @device:device to send query on + * @port_num: port number to send query on + * @rec:Path Record to send in query + * @comp_mask:component mask to send in query + * @timeout_ms:time to wait for response + * @gfp_mask:GFP mask to use for internal allocations + * @callback:function called when query completes, times out or is + * canceled + * @context:opaque user context passed to callback + * @sa_query:query context, used to cancel query + * + * Send a Path Record Get query to the SA to look up a path. The + * callback function will be called when the query completes (or + * fails); status is 0 for a successful response, -EINTR if the query + * is canceled, -ETIMEDOUT is the query timed out, or -EIO if an error + * occurred sending the query. The resp parameter of the callback is + * only valid if status is 0. + * + * If the return value of ib_sa_path_rec_get() is negative, it is an + * error code. Otherwise it is a query ID that can be used to cancel + * the query. + */ +int ib_sa_path_rec_get(struct ib_sa_client *client, + struct ib_device *device, u8 port_num, + struct ib_sa_path_rec *rec, + ib_sa_comp_mask comp_mask, + int timeout_ms, gfp_t gfp_mask, + void (*callback)(int status, + struct ib_sa_path_rec *resp, + void *context), + void *context, + struct ib_sa_query **sa_query) +{ + struct sa_path_request *req; + struct ib_sa_attr_iter iter; + struct ib_sa_path_rec *path_rec; + int ret; + + if (!paths_per_dest) + goto query_sa; + + if (!(comp_mask & IB_SA_PATH_REC_DGID) || + !(comp_mask & IB_SA_PATH_REC_NUMB_PATH) || rec->numb_path != 1) + goto query_sa; + + req = kmalloc(sizeof *req, gfp_mask); + if (!req) + goto query_sa; + + ret = ib_create_path_iter(device, port_num, &rec->dgid, &iter); + if (ret) + goto free_req; + + if (lookup_method == SA_DB_LOOKUP_RANDOM) + path_rec = get_random_path(&iter, rec, comp_mask); + else + path_rec = get_next_path(&iter, rec, comp_mask); + + if (!path_rec) + goto free_iter; + + memcpy(&req->path_rec, path_rec, sizeof *path_rec); + ib_free_path_iter(&iter); + + INIT_WORK(&req->work, report_path); + req->client = client; + req->callback = callback; + req->context = context; + + ib_sa_client_get(client); + queue_work(sa_wq, &req->work); + *sa_query = ERR_PTR(-EEXIST); + return 0; + +free_iter: + ib_free_path_iter(&iter); +free_req: + kfree(req); +query_sa: + return ib_sa_path_rec_query(client, device, port_num, rec, comp_mask, + timeout_ms, gfp_mask, callback, context, + sa_query); +} +EXPORT_SYMBOL(ib_sa_path_rec_get); + +static void recv_handler(struct ib_mad_agent *mad_agent, + struct ib_mad_recv_wc *mad_recv_wc) +{ + struct sa_db_port *port; + struct update_info *update; + struct ib_mad_send_buf *msg; + enum sa_update_type type; + + msg = (struct ib_mad_send_buf *) (unsigned long) mad_recv_wc->wc->wr_id; + port = msg->context[0]; + update = msg->context[1]; + + mutex_lock(&lock); + if (port->state == SA_DB_DESTROY || + update != list_entry(port->update_list.next, + struct update_info, list)) { + mutex_unlock(&lock); + } else { + type = update->type; + mutex_unlock(&lock); + update_path_db(mad_agent->context, mad_recv_wc, type); + } + + ib_free_recv_mad(mad_recv_wc); +} + +static void send_handler(struct ib_mad_agent *agent, + struct ib_mad_send_wc *mad_send_wc) +{ + struct ib_mad_send_buf *msg; + struct sa_db_port *port; + struct update_info *update; + int ret; + + msg = mad_send_wc->send_buf; + port = msg->context[0]; + update = msg->context[1]; + + mutex_lock(&lock); + if (port->state == SA_DB_DESTROY) + goto unlock; + + if (update == list_entry(port->update_list.next, + struct update_info, list)) { + + if (mad_send_wc->status == IB_WC_RESP_TIMEOUT_ERR && + msg->timeout_ms < SA_DB_MAX_RETRY_TIMER) { + + msg->timeout_ms <<= 1; + ret = ib_post_send_mad(msg, NULL); + if (!ret) { + mutex_unlock(&lock); + return; + } + } + list_del(&update->list); + kfree(update); + } + process_updates(port); +unlock: + mutex_unlock(&lock); + + ib_destroy_ah(msg->ah); + ib_free_send_mad(msg); +} + +static int init_port(struct sa_db_device *dev, int port_num) +{ + struct sa_db_port *port; + int ret; + + port = &dev->port[port_num - dev->start_port]; + port->dev = dev; + port->port_num = port_num; + INIT_WORK(&port->work, port_work_handler); + port->paths = RB_ROOT; + INIT_LIST_HEAD(&port->update_list); + + ret = ib_get_cached_gid(dev->device, port_num, 0, &port->gid); + if (ret) + return ret; + + port->agent = ib_register_mad_agent(dev->device, port_num, IB_QPT_GSI, + NULL, IB_MGMT_RMPP_VERSION, + send_handler, recv_handler, port); + if (IS_ERR(port->agent)) + ret = PTR_ERR(port->agent); + + return ret; +} + +static void destroy_port(struct sa_db_port *port) +{ + mutex_lock(&lock); + port->state = SA_DB_DESTROY; + mutex_unlock(&lock); + + ib_unregister_mad_agent(port->agent); + cleanup_port(port); + flush_workqueue(sa_wq); +} + +static void sa_db_add_dev(struct ib_device *device) { struct sa_db_device *dev; struct sa_db_port *port; - int i; + int s, e, i, ret; if (rdma_node_get_transport(device->node_type) != RDMA_TRANSPORT_IB) return; - dev = kmalloc(sizeof *dev + device->phys_port_cnt * sizeof *port, - GFP_KERNEL); + if (device->node_type == RDMA_NODE_IB_SWITCH) { + s = e = 0; + } else { + s = 1; + e = device->phys_port_cnt; + } + + dev = kzalloc(sizeof *dev + (e - s + 1) * sizeof *port, GFP_KERNEL); if (!dev) return; - for (i = 1; i <= device->phys_port_cnt; i++) { - port = &dev->port[i-1]; - port->dev = dev; - port->port_num = i; - port->update_time = jiffies - hold_time; - port->update = 0; - INIT_DELAYED_WORK(&port->work, update_cache); - port->paths = RB_ROOT; - - if (ib_get_cached_gid(device, i, 0, &port->gid)) - goto err; - - port->agent = ib_register_mad_agent(device, i, IB_QPT_GSI, - NULL, IB_MGMT_RMPP_VERSION, - send_handler, recv_handler, - port); - if (IS_ERR(port->agent)) + dev->start_port = s; + dev->port_count = e - s + 1; + dev->device = device; + for (i = 0; i < dev->port_count; i++) { + ret = init_port(dev, s + i); + if (ret) goto err; } - dev->device = device; ib_set_client_data(device, &sa_db_client, dev); - down_write(&lock); + INIT_IB_EVENT_HANDLER(&dev->event_handler, device, handle_event); + + mutex_lock(&lock); list_add_tail(&dev->list, &dev_list); - up_write(&lock); + refresh_dev_db(dev); + mutex_unlock(&lock); - /* Initialization must be complete before cache updates can occur. */ - INIT_IB_EVENT_HANDLER(&dev->event_handler, device, handle_event); ib_register_event_handler(&dev->event_handler); - - /* Force an update now. */ - for (i = 1; i <= device->phys_port_cnt; i++) - schedule_update(&dev->port[i-1]); return; err: - while (--i) { - ib_unregister_mad_agent(dev->port[i-1].agent); - remove_all_attrs(&dev->port[i-1].paths); - } + while (i--) + destroy_port(&dev->port[i]); kfree(dev); } -static void sa_db_remove_one(struct ib_device *device) +static void sa_db_remove_dev(struct ib_device *device) { struct sa_db_device *dev; int i; @@ -627,53 +1231,43 @@ static void sa_db_remove_one(struct ib_d return; ib_unregister_event_handler(&dev->event_handler); - for (i = 0; i < device->phys_port_cnt; i++) - cancel_delayed_work(&dev->port[i].work); flush_workqueue(sa_wq); - for (i = 0; i < device->phys_port_cnt; i++) { - ib_unregister_mad_agent(dev->port[i].agent); - remove_all_attrs(&dev->port[i].paths); - } + for (i = 0; i < dev->port_count; i++) + destroy_port(&dev->port[i]); - down_write(&lock); + mutex_lock(&lock); list_del(&dev->list); - up_write(&lock); + mutex_unlock(&lock); + kfree(dev); } -static int __init sa_db_init(void) +int sa_db_init(void) { int ret; - cache_timeout = msecs_to_jiffies(cache_timeout); - hold_time = msecs_to_jiffies(hold_time); - update_delay = msecs_to_jiffies(update_delay); - - if (!cache_timeout) - paths_per_dest = 0; - else if (paths_per_dest > IB_MAX_PATHS_PER_DEST) - paths_per_dest = IB_MAX_PATHS_PER_DEST; - + rwlock_init(&rwlock); sa_wq = create_singlethread_workqueue("local_sa"); if (!sa_wq) return -ENOMEM; + ib_sa_register_client(&sa_client); ret = ib_register_client(&sa_db_client); if (ret) goto err; + return 0; err: + ib_sa_unregister_client(&sa_client); destroy_workqueue(sa_wq); return ret; } -static void __exit sa_db_cleanup(void) +void sa_db_cleanup(void) { ib_unregister_client(&sa_db_client); + ib_sa_unregister_client(&sa_client); destroy_workqueue(sa_wq); } - -module_init(sa_db_init); -module_exit(sa_db_cleanup); From john.russo at qlogic.com Tue Jun 17 06:06:03 2008 From: john.russo at qlogic.com (John Russo) Date: Tue, 17 Jun 2008 08:06:03 -0500 Subject: [ofa-general] FW: QLogic vNIC Kernel Submission Message-ID: <99863D2ED484D449811D97A4C44C9CBD7C50F7@EPEXCH2.qlogic.org> It looks as if my original email was "scrubbed" before it made the mailing list so I am resending it... QLogic has been attempting to submit our virtual NIC (vNIC) driver to the Linux kernel for several months.  We have made changes to the code based on the feedback we have received over four rounds of submissions. Among the feedback we received during this process was a request to alter our code to use a single value per file for configuration of our driver through sysfs interface.  After spending much time and effort to complete this change to our design we re-submitted the driver only to receive a response suggesting that we change once again from this interface to a different API interface called rtnl_link.  Needless to say I am very frustrated with this process. This new API interface would require substantial changes to our code. QLogic has met the initial request to move to a single valued sysfs interface and I would hope that this new request will be waived and will not be a roadblock to inclusion of our driver to the kernel. From hrosenstock at xsigo.com Tue Jun 17 06:42:56 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 06:42:56 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_lid_mgr.c: Cosmetic formatting changes Message-ID: <1213710176.14047.740.camel@hrosenstock-ws.xsigo.com> opensm/osm_lid_mgr.c: Cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index 1ad9ef0..7472870 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -173,9 +173,9 @@ static void __osm_lid_mgr_validate_db(IN osm_lid_mgr_t * p_mgr) || (p_item->guid == 0) || (max_lid > p_mgr->p_subn->max_unicast_lid_ho)) { OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0312: " - "Illegal LID range [0x%x:0x%x] for guid:0x%016" - PRIx64 "\n", min_lid, max_lid, - p_item->guid); + "Illegal LID range [0x%x:0x%x] for " + "guid:0x%016" PRIx64 "\n", min_lid, + max_lid, p_item->guid); lids_ok = FALSE; } else if ((min_lid != max_lid) && ((min_lid & lmc_mask) != min_lid)) { @@ -212,8 +212,9 @@ static void __osm_lid_mgr_validate_db(IN osm_lid_mgr_t * p_mgr) (p_mgr->p_g2l, p_item->guid)) OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0315: " - "failed to delete entry for guid:0x%016" - PRIx64 "\n", p_item->guid); + "failed to delete entry for " + "guid:0x%016" PRIx64 "\n", + p_item->guid); } else { /* mark it was visited */ for (lid = min_lid; lid <= max_lid; lid++) @@ -262,12 +263,14 @@ osm_lid_mgr_init(IN osm_lid_mgr_t * const p_mgr, IN osm_sm_t *sm) if (osm_db_restore(p_mgr->p_g2l)) { if (p_mgr->p_subn->opt.exit_on_fatal) { osm_log(p_mgr->p_log, OSM_LOG_SYS, - "FATAL: Error restoring Guid-to-Lid persistent database\n"); + "FATAL: Error restoring Guid-to-Lid " + "persistent database\n"); status = IB_ERROR; goto Exit; } else - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0317: " - "Error restoring Guid-to-Lid persistent database\n"); + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, + "ERR 0317: Error restoring Guid-to-Lid " + "persistent database\n"); } /* we need to make sure we did not get duplicates with @@ -337,11 +340,13 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) cl_ptr_vector_set(p_persistent_vec, lid, NULL); } else { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Honor current guid2lid file when coming out of standby\n"); + "Honor current guid2lid file when coming out " + "of standby\n"); osm_db_clear(p_mgr->p_g2l); if (osm_db_restore(p_mgr->p_g2l)) OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0306: " - "Error restoring Guid-to-Lid persistent database. Ignoring it\n"); + "Error restoring Guid-to-Lid " + "persistent database. Ignoring it\n"); } } @@ -397,8 +402,9 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) (db_max_lid - db_min_lid + 1 < num_lids))) { /* Not aligned, or not wide enough, then remove the entry */ OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Cleaning persistent entry for guid:0x%016" - PRIx64 " illegal range:[0x%x:0x%x]\n", + "Cleaning persistent entry for guid:" + "0x%016" PRIx64 " illegal range:" + "[0x%x:0x%x]\n", cl_ntoh64(osm_port_get_guid(p_port)), db_min_lid, db_max_lid); osm_db_guid2lid_delete(p_mgr->p_g2l, @@ -452,8 +458,8 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) if ((lid <= max_persistent_lid) && cl_ptr_vector_get(p_persistent_vec, lid)) { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%04x is not free as its mapped by the persistent db\n", - lid); + "0x%04x is not free as its mapped by the " + "persistent db\n", lid); is_free = FALSE; } else { /* check this is a discovered port */ @@ -478,8 +484,9 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) &db_min_lid, &db_max_lid)) { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%04x is free as it was discovered " - "but mapped by the persistent db to [0x%04x:0x%04x]\n", + "0x%04x is free as it was " + "discovered but mapped by the " + "persistent db to [0x%04x:0x%04x]\n", lid, db_min_lid, db_max_lid); } else { /* can the port keep its assignment ? */ @@ -507,8 +514,8 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) /* The lid cannot be used */ OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%04x is free as it was discovered " - "but not aligned\n", + "0x%04x is free as it was " + "discovered but not aligned\n", lid); } else { /* check that all needed lids are not persistently mapped */ @@ -579,10 +586,11 @@ AfterScanningLids: (osm_lid_mgr_range_t *) malloc(sizeof(osm_lid_mgr_range_t)); /* The p_range can be NULL in one of 2 cases: - 1. If max_defined_lid == 0. In this case, we want the entire range. - 2. If all lids discovered in the loop where mapped. In this case, - no free range exists and we want to define it after the last - mapped lid. + 1. If max_defined_lid == 0. In this case, we want the + entire range. + 2. If all lids discovered in the loop where mapped. In this + case, no free range exists and we want to define it after the + last mapped lid. */ if (p_range) p_range->min_lid = lid; @@ -771,8 +779,8 @@ __osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, goto Exit; } else { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%016" PRIx64 - " with lid:0x%04x does not match its known lid:0x%04x\n", + "0x%016" PRIx64 " with lid:0x%04x " + "does not match its known lid:0x%04x\n", guid, cl_ntoh16(osm_port_get_base_lid(p_port)), min_lid); __osm_lid_mgr_cleanup_discovered_port_lid_range(p_mgr, @@ -809,13 +817,13 @@ __osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, goto NewLidSet; } else OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%016" PRIx64 - " existing lid range:[0x%x:0x%x] is not free\n", + "0x%016" PRIx64 " existing lid " + "range:[0x%x:0x%x] is not free\n", guid, min_lid, min_lid + num_lids - 1); } else OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%016" PRIx64 - " existing lid range:[0x%x:0x%x] is not lmc aligned\n", + "0x%016" PRIx64 " existing lid range:" + "0x%x:0x%x] is not lmc aligned\n", guid, min_lid, min_lid + num_lids - 1); } @@ -1033,8 +1041,8 @@ __osm_lid_mgr_set_physp_pi(IN osm_lid_mgr_t * const p_mgr, if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Sending Link Down to GUID 0x%016" - PRIx64 - "port %d due to op_vls or mtu change. MTU:%u,%u VL_CAP:%u,%u\n", + PRIx64 "port %d due to op_vls or " + "mtu change. MTU:%u,%u VL_CAP:%u,%u\n", cl_ntoh64(osm_physp_get_port_guid (p_physp)), port_num, mtu, ib_port_info_get_neighbor_mtu(p_old_pi), @@ -1043,8 +1051,9 @@ __osm_lid_mgr_set_physp_pi(IN osm_lid_mgr_t * const p_mgr, ); /* - we need to make sure the internal DB will follow the fact the remote - port is also going through "down" state into "init"... + we need to make sure the internal DB will follow the + fact that the remote port is also going through + "down" state into "init"... */ __osm_lid_mgr_set_remote_pi_state_to_init(p_mgr, p_physp); @@ -1090,11 +1099,11 @@ __osm_lid_mgr_set_physp_pi(IN osm_lid_mgr_t * const p_mgr, context.pi_context.active_transition = FALSE; /* - We need to set the cli_rereg bit when we are in first_time_master_sweep for - ports supporting the ClientReregistration Vol1 (v1.2) p811 14.4.11 - Also, if this port was just now discovered, then we should also set the - cli_rereg bit. We know that the port was just discovered if its is_new - field is set. + We need to set the cli_rereg bit when we are in first_time_master_sweep + for ports supporting the ClientReregistration Vol1 (v1.2) p811 14.4.11 + Also, if this port was just now discovered, then we should also set + the cli_rereg bit. We know that the port was just discovered if its + is_new field is set. */ if ((p_mgr->p_subn->first_time_master_sweep == TRUE || p_port->is_new) && !p_mgr->p_subn->opt.no_clients_rereg @@ -1279,15 +1288,17 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr) } /* - get the port lid range - we need to send it on first active sweep or - if there was a change (the result of the __osm_lid_mgr_get_port_lid) + get the port lid range - we need to send it on first active + sweep or if there was a change (the result of + __osm_lid_mgr_get_port_lid) */ lid_changed = __osm_lid_mgr_get_port_lid(p_mgr, p_port, &min_lid_ho, &max_lid_ho); - /* we can call the function to update the port info as it known to - look for any field change and will only send an updated if required */ + /* we can call the function to update the port info as it known + to look for any field change and will only send an updated + if required */ OSM_LOG(p_mgr->p_log, OSM_LOG_VERBOSE, "Assigned port 0x%016" PRIx64 ", LID [0x%X,0x%X]\n", cl_ntoh64(port_guid), From tziporet at mellanox.co.il Tue Jun 17 07:17:26 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Tue, 17 Jun 2008 17:17:26 +0300 Subject: [ofa-general] OFED June 16 2008 meeting summary In-Reply-To: <6C2C79E72C305246B504CBA17B5500C90440EF3B@mtlexch01.mtl.com> Message-ID: <6C2C79E72C305246B504CBA17B5500C904466C81@mtlexch01.mtl.com> OFED meeting summary for June 16 2008 ====================================== Summary: ======== * OFED 1.4: Decided to move the feature freeze to middle of July (instead of end of June). Release schedule: * Alpha: Jul 15, 08 - feature freeze * Beta: Jul 30, 08 - kernel 2.6.27-rcX * RC1: Aug 13, 08 * RC2: Aug 27, 08 * RC3-RC5/6 - every 5-10 days o Latest RC to be used in OFA interop event * GA: Oct 06 08 * EWG chair and co-chair: Gopal will be the chair and he will work closely to bring Cisco contribution to the EWG. Tziporet is nominated as EWG co-chair. * Libraries: Going to define OFED public libraries, for which we commit to stable API, and which are private. * OFA web site: the development web page are under changes. See more details below. Details: ======== 1. OFED 1.4 status: - Kernel: - Merged with 2.6.26-rc6 - SRP still failing - fixed today - qlogic_vnic, iSER and nes - under work of the maintainers - New components: iSER target - under work NFSoRDMA - no update from Jeff yet - Userspace status: - libiverbs - took latest changes from Roland's tree - other components - moved to master git trees 2. Close feature freeze date for OFED 1.4: After a short discussion all participant agreed that is make more sense to have the code freeze on middle of July. See release schedule above. 3. Electing a chair and co-chair to the EWG: Gopal has joined the meeting and stated his wish to actively participate in the meetings and find a way for him and Cisco to contribute to the EWG deliverables and work. In addition the group decided that Tziporet form Mellanox will be nominated as co-chair to continue and lead the EWG OFED activity. Gopal and Tziporet will meet and define the collaboration between them. 4. OFED libraries backward compatibility: Since we found that some "private" libraries were used as public, and thus changes in their API caused end customer issues we decided that we need to define which libraries are public and which are private. Tziporet will prepare a list of libraries. In the next meeting we will define the public and private libraries. This list will be documented in the new development web page and will also be added to OFED docs. 5. OFA Web: Woody updated on the changes on the developers' web page of OFA. Everybody is welcome to look at the new web pages and send suggestion for improvements. Woody will send an explanation to the list how to update and add documents to the doc directory Tziporet will maintain the roadmap page. Comment: the Linux download page is not organized and not pointing to the OFED releases. Tziporet -- Tziporet Koren Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 380) Mobile: +972 (54) 468 9426 E-mail: tziporet at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From hrosenstock at xsigo.com Tue Jun 17 07:26:33 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 07:26:33 -0700 Subject: [ofa-general] [PATCH] ibutils: fixing seg. fault in ibis_gsi_mad_ctrl.c In-Reply-To: <48356525.9010700@dev.mellanox.co.il> References: <48356525.9010700@dev.mellanox.co.il> Message-ID: <1213712793.13056.29.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-05-22 at 15:20 +0300, Yevgeny Kliteynik wrote: > Hi Oren, > > Fixing seg fault in allocation of gsi management class vector. Was this ever committed ? Should it be applied for both trunk and ofed_1_3 ? -- Hal > > Signed-off-by: Yevgeny Kliteynik > --- > ibis/src/ibis_gsi_mad_ctrl.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/ibis/src/ibis_gsi_mad_ctrl.c b/ibis/src/ibis_gsi_mad_ctrl.c > index 356d33d..bfb5fe6 100644 > --- a/ibis/src/ibis_gsi_mad_ctrl.c > +++ b/ibis/src/ibis_gsi_mad_ctrl.c > @@ -731,7 +731,7 @@ ibis_gsi_mad_ctrl_set_class_attr_cb( > size = cl_vector_get_size(&p_ctrl->class_vector); > if (size <= mad_class) > { > - cl_status = cl_vector_set_size(&p_ctrl->class_vector,mad_class); > + cl_status = cl_vector_set_size(&p_ctrl->class_vector,mad_class+1); > > if( cl_status != CL_SUCCESS) > { From sashak at voltaire.com Tue Jun 17 07:25:45 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 17:25:45 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213639012.5369.443.camel@cardanus.llnl.gov> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213639012.5369.443.camel@cardanus.llnl.gov> Message-ID: <20080617142545.GA25554@sashak.voltaire.com> Hi Al, On 10:56 Mon 16 Jun , Al Chu wrote: > > patch series seems to perform well. As a reminder mvapich 0.9.9 does > not how to deal with multiple lids, openMPI 1.2.6 does. The "preserve > lids" condition is labeled "PL". > > With LMC > 0, the preserve lids seems to do its job under mvapich 0.9.9, > and atleast maintains (sometimes increases) performance of openmpi 1.2.6 > with LMC > 0. Numbers are attached. The numbers look fine. Thanks for testing this. > I'd give my thumbs up to the patch. Lets commit it. Already done. Sasha From hrosenstock at xsigo.com Tue Jun 17 07:34:01 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 07:34:01 -0700 Subject: [ofa-general] [PATCH] ibutils: fixing seg. fault in ibis_gsi_mad_ctrl.c In-Reply-To: <1213712793.13056.29.camel@hrosenstock-ws.xsigo.com> References: <48356525.9010700@dev.mellanox.co.il> <1213712793.13056.29.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213713241.13056.31.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-17 at 07:26 -0700, Hal Rosenstock wrote: > On Thu, 2008-05-22 at 15:20 +0300, Yevgeny Kliteynik wrote: > > Hi Oren, > > > > Fixing seg fault in allocation of gsi management class vector. > > Was this ever committed ? Should it be applied for both trunk and > ofed_1_3 ? Looks like it's there for trunk but not ofed_1_3. -- Hal > -- Hal > > > > > Signed-off-by: Yevgeny Kliteynik > > --- > > ibis/src/ibis_gsi_mad_ctrl.c | 2 +- > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > diff --git a/ibis/src/ibis_gsi_mad_ctrl.c b/ibis/src/ibis_gsi_mad_ctrl.c > > index 356d33d..bfb5fe6 100644 > > --- a/ibis/src/ibis_gsi_mad_ctrl.c > > +++ b/ibis/src/ibis_gsi_mad_ctrl.c > > @@ -731,7 +731,7 @@ ibis_gsi_mad_ctrl_set_class_attr_cb( > > size = cl_vector_get_size(&p_ctrl->class_vector); > > if (size <= mad_class) > > { > > - cl_status = cl_vector_set_size(&p_ctrl->class_vector,mad_class); > > + cl_status = cl_vector_set_size(&p_ctrl->class_vector,mad_class+1); > > > > if( cl_status != CL_SUCCESS) > > { > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Tue Jun 17 07:51:54 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 07:51:54 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213639012.5369.443.camel@cardanus.llnl.gov> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213639012.5369.443.camel@cardanus.llnl.gov> Message-ID: <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> Al, On Mon, 2008-06-16 at 10:56 -0700, Al Chu wrote: > Hey Sasha, > > patch series seems to perform well. As a reminder mvapich 0.9.9 does > not how to deal with multiple lids, openMPI 1.2.6 does. The "preserve > lids" condition is labeled "PL". > > With LMC > 0, the preserve lids seems to do its job under mvapich 0.9.9, > and atleast maintains (sometimes increases) performance of openmpi 1.2.6 > with LMC > 0. Numbers are attached. Maybe I'm not reading this right but here's what I see: Other than the average bandwidth numbers at the top which show PL > non PL, the ones in the table seem to be the other way around. Do those numbers indicate bandwidth ? Also, you mentioned that mvapich doesn't know how to deal with LMC yet the numbers seem to go up as LMC does. -- Hal > I'd give my thumbs up to the patch. Lets commit it. > > Al > > On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: > > Basically this addresses the problem described by Al Chu in: > > > > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html > > > > When base lid paths become completely disbalanced on a fabrics with > > lmc > 0. > > > > One feedback was from Yiftah Shahar: > > > > "I think that our requirements should be that even when you are working > > with LMC>0 then the base LID routing should not be affected. > > One way to achieve this goal is to first run the base-LID routing (so > > all base LID improvement will be also in LMC>0) and then start with the > > other LIDs as round-robbing starting from the base-lid-port + 1 > > according current routing algorithm rules (keeping min-hop, up/down...)." > > > > We had some discussion with Al and Yiftah about this and considered that > > in addition to "pure" base lid paths preservation (which is good thing by > > itself) proposed method solves original lid disbalancing problem as well. > > > > This patch is implementation of the idea above. > > > > Signed-off-by: Sasha Khapyorsky > > --- > > opensm/include/opensm/osm_switch.h | 4 + > > opensm/opensm/osm_dump.c | 3 +- > > opensm/opensm/osm_switch.c | 8 ++- > > opensm/opensm/osm_ucast_mgr.c | 163 ++++++++++++++++-------------------- > > 4 files changed, 86 insertions(+), 92 deletions(-) > > > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > > index 0e9c5fa..c1521a6 100644 > > --- a/opensm/include/opensm/osm_switch.h > > +++ b/opensm/include/opensm/osm_switch.h > > @@ -981,6 +981,7 @@ uint8_t > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN osm_port_t * p_port, > > IN const uint16_t lid_ho, > > + IN unsigned start_from, > > IN const boolean_t ignore_existing, > > IN const boolean_t dor); > > /* > > @@ -995,6 +996,9 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > * lid_ho > > * [in] LID value (host order) for which to get a path advisory. > > * > > +* start_from > > +* [in] Port number from where to start balance counting. > > +* > > * ignore_existing > > * [in] Set to cause the switch to choose the optimal route > > * regardless of existing paths. > > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > > index b96984b..60c6d25 100644 > > --- a/opensm/opensm/osm_dump.c > > +++ b/opensm/opensm/osm_dump.c > > @@ -218,7 +218,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > > else { > > /* No LMC Optimization */ > > best_port = osm_switch_recommend_path(p_sw, p_port, > > - lid_ho, TRUE, dor); > > + lid_ho, 1, TRUE, > > + dor); > > fprintf(file, "No %u hop path possible via port %u!", > > best_hops, best_port); > > } > > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > > index 58936e3..a9d13c8 100644 > > --- a/opensm/opensm/osm_switch.c > > +++ b/opensm/opensm/osm_switch.c > > @@ -274,6 +274,7 @@ uint8_t > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > IN osm_port_t * p_port, > > IN const uint16_t lid_ho, > > + IN unsigned start_from, > > IN const boolean_t ignore_existing, > > IN const boolean_t dor) > > { > > @@ -294,6 +295,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > uint8_t port_num; > > uint8_t num_ports; > > uint32_t least_paths = 0xFFFFFFFF; > > + unsigned i; > > /* > > The follwing will track the least paths if the > > route should go through a new system/node > > @@ -397,8 +399,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > */ > > > > /* port number starts with one and num_ports is 1 + num phys ports */ > > - for (port_num = 1; port_num < num_ports; port_num++) { > > - if (osm_switch_get_hop_count(p_sw, base_lid, port_num) != > > + for (i = start_from; i < start_from + num_ports; i++) { > > + port_num = i%num_ports; > > + if (!port_num || > > + osm_switch_get_hop_count(p_sw, base_lid, port_num) != > > least_hops) > > continue; > > > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > > index c073037..2aae6d5 100644 > > --- a/opensm/opensm/osm_ucast_mgr.c > > +++ b/opensm/opensm/osm_ucast_mgr.c > > @@ -208,7 +208,8 @@ find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, > > static void > > __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > IN osm_switch_t * const p_sw, > > - IN osm_port_t * const p_port) > > + IN osm_port_t * const p_port, > > + IN unsigned lid_offset) > > { > > uint16_t min_lid_ho; > > uint16_t max_lid_ho; > > @@ -217,19 +218,14 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > boolean_t is_ignored_by_port_prof; > > ib_net64_t node_guid; > > struct osm_routing_engine *p_routing_eng; > > - /* > > - The following are temporary structures that will aid > > - in providing better routing in LMC > 0 situations > > - */ > > - uint16_t lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > > - struct osm_remote_node *p_remote_guid_used = NULL; > > + unsigned start_from = 1; > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > > > > - /* If the lids are zero - then there was some problem with the initialization. > > - Don't handle this port. */ > > + /* If the lids are zero - then there was some problem with > > + * the initialization. Don't handle this port. */ > > if (min_lid_ho == 0 || max_lid_ho == 0) { > > OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A04: " > > "Port 0x%" PRIx64 " has LID 0. An initialization " > > @@ -238,16 +234,22 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > goto Exit; > > } > > > > - if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { > > + lid_ho = min_lid_ho + lid_offset; > > + > > + if (lid_ho > max_lid_ho) > > + goto Exit; > > + > > + if (lid_offset) > > + /* ignore potential overflow - it is handled in osm_switch.c */ > > + start_from = osm_switch_get_port_by_lid(p_sw, lid_ho - 1) + 1; > > + > > + if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) > > OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > - "Processing port 0x%" PRIx64 ", LIDs [0x%X,0x%X]\n", > > - cl_ntoh64(osm_port_get_guid(p_port)), > > + "Processing port 0x%" PRIx64 ", LID %u [0x%X,0x%X]\n", > > + cl_ntoh64(osm_port_get_guid(p_port)), lid_ho, > > min_lid_ho, max_lid_ho); > > - } > > > > - /* > > - TO DO - This should be runtime error, not a CL_ASSERT() > > - */ > > + /* TODO - This should be runtime error, not a CL_ASSERT() */ > > CL_ASSERT(max_lid_ho < osm_switch_get_fwd_tbl_size(p_sw)); > > > > node_guid = osm_node_get_node_guid(p_sw->p_node); > > @@ -260,80 +262,62 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > how best to distribute the LID range across the ports > > that can reach those LIDs. > > */ > > - for (lid_ho = min_lid_ho; lid_ho <= max_lid_ho; lid_ho++) { > > - /* Use the enhanced algorithm only for LMC > 0 */ > > - if (lids_per_port > 1) { > > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > - p_mgr->p_subn-> > > - ignore_existing_lfts, > > - p_mgr->is_dor); > > - if (port > 0 && port != OSM_NO_PATH && p_port->priv) > > - p_remote_guid_used = > > - find_and_add_remote_sys(p_sw, port, > > - p_port->priv); > > + port = osm_switch_recommend_path(p_sw, p_port, lid_ho, start_from, > > + p_mgr->p_subn->ignore_existing_lfts, > > + p_mgr->is_dor); > > + > > + if (port == OSM_NO_PATH) { > > + /* do not try to overwrite the ppro of non existing port ... */ > > + is_ignored_by_port_prof = TRUE; > > + > > + /* Up/Down routing can cause unreachable routes between some > > + switches so we do not report that as an error in that case */ > > + if (!p_routing_eng->build_lid_matrices) { > > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > > + "No path to get to LID 0x%X from switch 0x%" > > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > > + /* trigger a new sweep - try again ... */ > > + p_mgr->p_subn->subnet_initialization_error = TRUE; > > } else > > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > - p_mgr->p_subn-> > > - ignore_existing_lfts, > > - p_mgr->is_dor); > > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > + "No path to get to LID 0x%X from switch 0x%" > > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > > + } else { > > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > + "Routing LID 0x%X to port 0x%X" > > + " for switch 0x%" PRIx64 "\n", > > + lid_ho, port, cl_ntoh64(node_guid)); > > > > /* > > - There might be no path to the target > > + we would like to optionally ignore this port in equalization > > + as in the case of the Mellanox Anafa Internal PCI TCA port > > */ > > - if (port == OSM_NO_PATH) { > > - /* do not try to overwrite the ppro of non existing port ... */ > > - is_ignored_by_port_prof = TRUE; > > - > > - /* Up/Down routing can cause unreachable routes between some > > - switches so we do not report that as an error in that case */ > > - if (!p_routing_eng->build_lid_matrices) { > > - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > > - "No path to get to LID 0x%X from switch 0x%" > > - PRIx64 "\n", lid_ho, > > - cl_ntoh64(node_guid)); > > - /* trigger a new sweep - try again ... */ > > - p_mgr->p_subn->subnet_initialization_error = > > - TRUE; > > - } else > > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > - "No path to get to LID 0x%X from switch 0x%" > > - PRIx64 "\n", lid_ho, > > - cl_ntoh64(node_guid)); > > - } else { > > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > - "Routing LID 0x%X to port 0x%X" > > - " for switch 0x%" PRIx64 "\n", > > - lid_ho, port, cl_ntoh64(node_guid)); > > - > > - /* > > - we would like to optionally ignore this port in equalization > > - as in the case of the Mellanox Anafa Internal PCI TCA port > > - */ > > - is_ignored_by_port_prof = > > - osm_port_prof_is_ignored_port(p_mgr->p_subn, > > - node_guid, port); > > - > > - /* > > - We also would ignore this route if the target lid is of a switch > > - and the port_profile_switch_node is not TRUE > > - */ > > - if (!p_mgr->p_subn->opt.port_profile_switch_nodes) { > > - is_ignored_by_port_prof |= > > - (osm_node_get_type(p_port->p_node) == > > - IB_NODE_TYPE_SWITCH); > > - } > > - } > > + is_ignored_by_port_prof = > > + osm_port_prof_is_ignored_port(p_mgr->p_subn, > > + node_guid, port); > > > > /* > > - We have selected the port for this LID. > > - Write it to the forwarding tables. > > + We also would ignore this route if the target lid is of > > + a switch and the port_profile_switch_node is not TRUE > > */ > > - p_mgr->lft_buf[lid_ho] = port; > > - if (!is_ignored_by_port_prof) { > > - osm_switch_count_path(p_sw, port); > > - if (p_remote_guid_used) > > - p_remote_guid_used->forwarded_to++; > > - } > > + if (!p_mgr->p_subn->opt.port_profile_switch_nodes) > > + is_ignored_by_port_prof |= > > + (osm_node_get_type(p_port->p_node) == > > + IB_NODE_TYPE_SWITCH); > > + } > > + > > + /* > > + We have selected the port for this LID. > > + Write it to the forwarding tables. > > + */ > > + p_mgr->lft_buf[lid_ho] = port; > > + if (!is_ignored_by_port_prof) { > > + struct osm_remote_node *rem_node_used; > > + osm_switch_count_path(p_sw, port); > > + if (port > 0 && p_port->priv && > > + (rem_node_used = find_and_add_remote_sys(p_sw, port, > > + p_port->priv))) > > + rem_node_used->forwarded_to++; > > } > > > > Exit: > > @@ -512,6 +496,7 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > > osm_node_t *p_node; > > osm_port_t *p_port; > > const cl_qmap_t *p_port_tbl; > > + unsigned i, lids_per_port; > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > @@ -538,12 +523,12 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > > Iterate through every port setting LID routes for each > > port based on base LID and LMC value. > > */ > > - > > - for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > > - p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > > - p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) { > > - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port); > > - } > > + lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > > + for (i = 0; i < lids_per_port; i++) > > + for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > > + p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > > + p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) > > + __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port, i); > > > > osm_ucast_mgr_set_fwd_table(p_mgr, p_sw); > > From mdidomenico4 at gmail.com Tue Jun 17 08:07:14 2008 From: mdidomenico4 at gmail.com (Michael Di Domenico) Date: Tue, 17 Jun 2008 11:07:14 -0400 Subject: [ofa-general] IB network problem - "Unknown remote side for node" Message-ID: Can anyone tell me what i likely causing this? The SM seems to be in a loop entering/exiting, and the "Unknown remote side" comes up with a different list of port each time it cycles. I though it was bad cables, but since it keeps changing, that seems unlikely. Thanks - Michael Jun 17 04:05:51 681669 [AAF0D060] -> OpenSM Rev:openib-2.0.5 OpenIB svn 9905 Jun 17 04:05:51 681723 [AAF0D060] -> OpenSM Rev:openib-2.0.5 OpenIB svn 9905 Jun 17 04:05:51 686837 [AAF0D060] -> osm_vendor_bind: Binding to port 0x2c9030000792e Jun 17 04:05:51 689709 [AAF0D060] -> osm_vendor_bind: Binding to port 0x2c9030000792e Jun 17 04:05:52 785232 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff00510d port 4. Adding to light sweep sampling list Jun 17 04:05:52 785273 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][13] Jun 17 04:05:52 785281 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff00510d port 6. Adding to light sweep sampling list Jun 17 04:05:52 785290 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][13] Jun 17 04:05:52 785296 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff00510d port 8. Adding to light sweep sampling list Jun 17 04:05:52 785305 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][13] Jun 17 04:05:52 785312 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff00510d port 10. Adding to light sweep sampling list Jun 17 04:05:52 785319 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][13] Jun 17 04:05:52 785325 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff00510d port 12. Adding to light sweep sampling list Jun 17 04:05:52 785334 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][13] Jun 17 04:05:52 785352 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff005118 port 8. Adding to light sweep sampling list Jun 17 04:05:52 785359 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][13][4] Jun 17 04:05:52 785424 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff00507a port 8. Adding to light sweep sampling list Jun 17 04:05:52 785431 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][13][6] Jun 17 04:05:52 785444 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff005094 port 8. Adding to light sweep sampling list Jun 17 04:05:52 785454 [46409940] -> Directed Path Dump of 2 hop path: Path = [0][2][3] Jun 17 04:05:52 785467 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff005095 port 10. Adding to light sweep sampling list Jun 17 04:05:52 785475 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][8][2] Jun 17 04:05:52 785501 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050a0 port 8. Adding to light sweep sampling list Jun 17 04:05:52 785509 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][13][8] Jun 17 04:05:52 785536 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050af port 8. Adding to light sweep sampling list Jun 17 04:05:52 785545 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][13][C] Jun 17 04:05:52 785553 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050b0 port 8. Adding to light sweep sampling list Jun 17 04:05:52 785561 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][13][A] Jun 17 04:05:52 785602 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050c1 port 4. Adding to light sweep sampling list Jun 17 04:05:52 785631 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][D] Jun 17 04:05:52 785645 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050c4 port 4. Adding to light sweep sampling list Jun 17 04:05:52 785651 [46409940] -> Directed Path Dump of 3 hop path: Path = [0][2][3][8] Jun 17 04:05:52 785662 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050c6 port 4. Adding to light sweep sampling list Jun 17 04:05:52 785671 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][D][4] Jun 17 04:05:52 785686 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050d2 port 2. Adding to light sweep sampling list Jun 17 04:05:52 785694 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][8] Jun 17 04:05:52 785702 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050d2 port 12. Adding to light sweep sampling list Jun 17 04:05:52 785710 [46409940] -> Directed Path Dump of 4 hop path: Path = [0][2][3][11][8] Jun 17 04:05:52 785741 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x000b8cffff0050fa port 10. Adding to light sweep sampling list Jun 17 04:05:52 785748 [46409940] -> Directed Path Dump of 5 hop path: Path = [0][2][3][11][8][C] Jun 17 04:05:52 785774 [46409940] -> Entering MASTER state Jun 17 04:05:52 786126 [46409940] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0x000000000000e80f,0x0002c9030000792e Jun 17 04:05:52 786254 [46409940] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0x000000000000e80f,0x0002c9030000792e Jun 17 04:05:56 691253 [AAF0D060] -> Exiting SM -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.kirch at oracle.com Tue Jun 17 08:22:02 2008 From: olaf.kirch at oracle.com (Olaf Kirch) Date: Tue, 17 Jun 2008 17:22:02 +0200 Subject: [ofa-general] Several RDS patches Message-ID: <200806171722.02780.olaf.kirch@oracle.com> Hi Vlad, during testing of RDS, a number of issues came up that I had to address. Some of them were bugs, others were usability issues. I submitted the following patches to git://git.openfabrics.org/~okir/ofed_1_3/linux-2.6.git code-drop-20080617 d9fd0d04ba31951bcd7fe4393bccfe98ab8a45d5 RDS: introduce rds_send_get_message 9906c6480cedb05905d23c696671bcde9894b6ed RDS: Track whether RDMA S/G list is DMA mapped db7a8ed76ac77e6a5352c8d41d28d9fac59e953d RDS: Another fix for RDMA signalling These fix another issue with signaling the completion of RDMA operations. 2665a94825a234fdd989ed98ae24c88bd3415368 RDS: Use vmalloc in rds_ib_setup_qp On a loaded system, re-establishing a broken RDS connection would fail because we tried to kmalloc the send and receive ring, which would result in an order-5 allocation. 1c3446c39af17e77fe7d39f32388e4c6462ee550 RDS: increase size of stats counters The stats counters interface to user space uses uint64_t, but inside the kernel, only "unsigned long" was used. On 32bit boxes this caused counters to be truncated. 1955124f7def27d55e98bfaff0c8da1438f2cca6 RDS: Expose info on IB connections c2d8ba51f925cddca73562611ead156e62c3f93a RDS: Extend IB connection info to include settings People asked for a way to identify the IB port a RDS connection would use when the application does the load balancing. 804de7023165fd3fe89af6aac94843f195a23e72 RDS: Wake up send worker after TX queue full condition is cleared When encountering a full IB send queue, we back off for 2 jiffies. When the send CQ handler frees up several entries, we want to kick the send worker quickly. The existing code had a bug in it. 23d9822407b96538950e9e3dff5a7dfdbb464efb RDS: Send notification for cancelled RDMA ops We need notifications for RDMA operations that got canceled using RDS_CANCEL_SENT_TO 5e210431f65089062302c6e5ae9e998d67b9a980 RDS: Add mr_pool_depleted stats counter Track how often we were unable to initiate a RDMA because we couldn't allocate a FMR from our pool. b99faff27c2c4e4b1058580645d997ce82d87ec0 RDS: rds-info -k should display inode number In order to identify the process holding a specific RDS socket, we want rds-info -k to show the socket's inode number, so that we can search /proc/*/fd for it eabede450f3d540605882727a3b71a116919d5c1 RDS: Clarify comment on rds_poll Cosmetic - Rick complained about a misleading comment in rds_poll :) ------------------------ The rds-info changes go with several patches to rds-tools, which I also pushed to my tree: git://git.openfabrics.org/~okir/ofed_1_3/rds-tools.git code-drop-20080617 e40149599d03c057b99df78608794dbb8803d732 rds-info: introduce address printing functions 4f84065a0b3aae50c2ee346d4b8935b654f9f64c rds-info: add option to display IB connections This implements display of RDS/IB connections 15eb9f7dfe6724857c80a698cf339146088c1bc8 rds-info: Add -v option 4b90f7329dc146f01ca5b91eda924bcf0f8aef3f rds-info: Extend IB connection info to include settings This displays per-connection IB settings such as max_send_wr if -v is given 794d3504649e5686342232cdad59b1cc6b445b35 rds-info -k should display inode number Display the inode number of the RDS socket. Can you please pull these into OFED-1.3? Olaf -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play okir at lst.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax From sashak at voltaire.com Tue Jun 17 08:57:20 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 18:57:20 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213639012.5369.443.camel@cardanus.llnl.gov> <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080617155720.GB25554@sashak.voltaire.com> Hi Hal, On 07:51 Tue 17 Jun , Hal Rosenstock wrote: > > Also, you mentioned that mvapich doesn't know how to deal with LMC yet > the numbers seem to go up as LMC does. Right, it is because there is better routing now for base LIDs which are used by mvapich. Sasha From sashak at voltaire.com Tue Jun 17 09:09:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 19:09:21 +0300 Subject: [ofa-general] IB network problem - "Unknown remote side for node" In-Reply-To: References: Message-ID: <20080617160921.GC25554@sashak.voltaire.com> On 11:07 Tue 17 Jun , Michael Di Domenico wrote: > Can anyone tell me what i likely causing this? The SM seems to be in a loop > entering/exiting, I don't know why it is looping, are you use -o (run once) option? > and the "Unknown remote side" comes up with a different > list of port each time it cycles. I though it was bad cables, but since it > keeps changing, that seems unlikely. > > Thanks > - Michael > > > Jun 17 04:05:51 681669 [AAF0D060] -> OpenSM Rev:openib-2.0.5 OpenIB svn 9905 > Jun 17 04:05:51 681723 [AAF0D060] -> OpenSM Rev:openib-2.0.5 OpenIB svn 9905 > Jun 17 04:05:51 686837 [AAF0D060] -> osm_vendor_bind: Binding to port > 0x2c9030000792e > Jun 17 04:05:51 689709 [AAF0D060] -> osm_vendor_bind: Binding to port > 0x2c9030000792e > Jun 17 04:05:52 785232 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff00510d port 4. Adding to light sweep > sampling list > Jun 17 04:05:52 785273 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785281 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff00510d port 6. Adding to light sweep > sampling list > Jun 17 04:05:52 785290 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785296 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff00510d port 8. Adding to light sweep > sampling list But it is always after that node Path = [0][2][3], so likely this part of subnet is slow in discovery or have a cabling problems. Also note that you are using two years old version of OpenSM. Sasha > Jun 17 04:05:52 785305 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785312 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff00510d port 10. Adding to light sweep > sampling list > Jun 17 04:05:52 785319 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785325 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff00510d port 12. Adding to light sweep > sampling list > Jun 17 04:05:52 785334 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785352 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff005118 port 8. Adding to light sweep > sampling list > Jun 17 04:05:52 785359 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][4] > Jun 17 04:05:52 785424 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff00507a port 8. Adding to light sweep > sampling list > Jun 17 04:05:52 785431 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][6] > Jun 17 04:05:52 785444 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff005094 port 8. Adding to light sweep > sampling list > Jun 17 04:05:52 785454 [46409940] -> Directed Path Dump of 2 hop path: > Path = [0][2][3] > Jun 17 04:05:52 785467 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff005095 port 10. Adding to light sweep > sampling list > Jun 17 04:05:52 785475 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][8][2] > Jun 17 04:05:52 785501 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050a0 port 8. Adding to light sweep > sampling list > Jun 17 04:05:52 785509 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][8] > Jun 17 04:05:52 785536 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050af port 8. Adding to light sweep > sampling list > Jun 17 04:05:52 785545 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][C] > Jun 17 04:05:52 785553 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050b0 port 8. Adding to light sweep > sampling list > Jun 17 04:05:52 785561 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][A] > Jun 17 04:05:52 785602 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050c1 port 4. Adding to light sweep > sampling list > Jun 17 04:05:52 785631 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][D] > Jun 17 04:05:52 785645 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050c4 port 4. Adding to light sweep > sampling list > Jun 17 04:05:52 785651 [46409940] -> Directed Path Dump of 3 hop path: > Path = [0][2][3][8] > Jun 17 04:05:52 785662 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050c6 port 4. Adding to light sweep > sampling list > Jun 17 04:05:52 785671 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][D][4] > Jun 17 04:05:52 785686 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050d2 port 2. Adding to light sweep > sampling list > Jun 17 04:05:52 785694 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][8] > Jun 17 04:05:52 785702 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050d2 port 12. Adding to light sweep > sampling list > Jun 17 04:05:52 785710 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][8] > Jun 17 04:05:52 785741 [46409940] -> osm_drop_mgr_process: ERR 0108: Unknown > remote side for node 0x000b8cffff0050fa port 10. Adding to light sweep > sampling list > Jun 17 04:05:52 785748 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][8][C] > Jun 17 04:05:52 785774 [46409940] -> Entering MASTER state > Jun 17 04:05:52 786126 [46409940] -> osm_report_notice: Reporting Generic > Notice type:3 num:66 from LID:0x0000 > GID:0x000000000000e80f,0x0002c9030000792e > Jun 17 04:05:52 786254 [46409940] -> osm_report_notice: Reporting Generic > Notice type:3 num:66 from LID:0x0000 > GID:0x000000000000e80f,0x0002c9030000792e > Jun 17 04:05:56 691253 [AAF0D060] -> Exiting SM > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Tue Jun 17 09:18:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 19:18:21 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_lid_mgr.c: Cosmetic formatting changes In-Reply-To: <1213710176.14047.740.camel@hrosenstock-ws.xsigo.com> References: <1213710176.14047.740.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080617161821.GD25554@sashak.voltaire.com> On 06:42 Tue 17 Jun , Hal Rosenstock wrote: > opensm/osm_lid_mgr.c: Cosmetic formatting changes > > Signed-off-by: Hal Rosenstock All four cosmetic patches are applied. Thanks. Sasha From sashak at voltaire.com Tue Jun 17 09:21:19 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 19:21:19 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c : TRIVIAL fixes in log handling In-Reply-To: <48579A86.5070508@dev.mellanox.co.il> References: <48579A86.5070508@dev.mellanox.co.il> Message-ID: <20080617162119.GE25554@sashak.voltaire.com> Hi Yevgeny, On 14:05 Tue 17 Jun , Yevgeny Kliteynik wrote: > > OSM_LOG_ENTER() at the beginning of the function and > OSM_LOG_EXIT() at the end of the function were missing. Why? The function is pretty isolated and we have a lot of useless logging right now - with -V initial OpenSM log files can grow to GBs. Sasha From sashak at voltaire.com Tue Jun 17 09:22:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 19:22:39 +0300 Subject: [ofa-general] Re: [PATCH] opensm/man/opensm.8.in: TRIVIAL addition in fat-tree description In-Reply-To: <48579B69.2080903@dev.mellanox.co.il> References: <48579B69.2080903@dev.mellanox.co.il> Message-ID: <20080617162239.GF25554@sashak.voltaire.com> On 14:09 Tue 17 Jun , Yevgeny Kliteynik wrote: > Hi Sasha, > > Mentioning in the man page that fat-tree can be used on > "symmetrical or almost symmetrical" trees, instead of the > previous "symmetrical" only. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. Sasha From chu11 at llnl.gov Tue Jun 17 09:25:45 2008 From: chu11 at llnl.gov (Al Chu) Date: Tue, 17 Jun 2008 09:25:45 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213639012.5369.443.camel@cardanus.llnl.gov> <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213719945.5369.454.camel@cardanus.llnl.gov> Hey Hal, > Other than the average bandwidth numbers at the top which show PL > > non PL, the ones in the table seem to be the other way around. Do > those numbers indicate bandwidth ? Sorry, the numbers for the mvapich/openmpi numbers are in units of time. I think microseconds. So lower is better. The "degradation" columns are the percent worse than LMC=0. The key thing to see is that originally, when LMC > 0 and multiple paths are not used, the numbers go up/it takes more time. When the preserving-base-lids patch is put in, the LMC > 0 numbers are about the same as the LMC=0 numbers. Al On Tue, 2008-06-17 at 07:51 -0700, Hal Rosenstock wrote: > Al, > > On Mon, 2008-06-16 at 10:56 -0700, Al Chu wrote: > > Hey Sasha, > > > > patch series seems to perform well. As a reminder mvapich 0.9.9 does > > not how to deal with multiple lids, openMPI 1.2.6 does. The "preserve > > lids" condition is labeled "PL". > > > > With LMC > 0, the preserve lids seems to do its job under mvapich 0.9.9, > > and atleast maintains (sometimes increases) performance of openmpi 1.2.6 > > with LMC > 0. Numbers are attached. > > Maybe I'm not reading this right but here's what I see: > > Other than the average bandwidth numbers at the top which show PL > non > PL, the ones in the table seem to be the other way around. Do those > numbers indicate bandwidth ? > > Also, you mentioned that mvapich doesn't know how to deal with LMC yet > the numbers seem to go up as LMC does. > > -- Hal > > > I'd give my thumbs up to the patch. Lets commit it. > > > > Al > > > > On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: > > > Basically this addresses the problem described by Al Chu in: > > > > > > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html > > > > > > When base lid paths become completely disbalanced on a fabrics with > > > lmc > 0. > > > > > > One feedback was from Yiftah Shahar: > > > > > > "I think that our requirements should be that even when you are working > > > with LMC>0 then the base LID routing should not be affected. > > > One way to achieve this goal is to first run the base-LID routing (so > > > all base LID improvement will be also in LMC>0) and then start with the > > > other LIDs as round-robbing starting from the base-lid-port + 1 > > > according current routing algorithm rules (keeping min-hop, up/down...)." > > > > > > We had some discussion with Al and Yiftah about this and considered that > > > in addition to "pure" base lid paths preservation (which is good thing by > > > itself) proposed method solves original lid disbalancing problem as well. > > > > > > This patch is implementation of the idea above. > > > > > > Signed-off-by: Sasha Khapyorsky > > > --- > > > opensm/include/opensm/osm_switch.h | 4 + > > > opensm/opensm/osm_dump.c | 3 +- > > > opensm/opensm/osm_switch.c | 8 ++- > > > opensm/opensm/osm_ucast_mgr.c | 163 ++++++++++++++++-------------------- > > > 4 files changed, 86 insertions(+), 92 deletions(-) > > > > > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > > > index 0e9c5fa..c1521a6 100644 > > > --- a/opensm/include/opensm/osm_switch.h > > > +++ b/opensm/include/opensm/osm_switch.h > > > @@ -981,6 +981,7 @@ uint8_t > > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > IN osm_port_t * p_port, > > > IN const uint16_t lid_ho, > > > + IN unsigned start_from, > > > IN const boolean_t ignore_existing, > > > IN const boolean_t dor); > > > /* > > > @@ -995,6 +996,9 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > * lid_ho > > > * [in] LID value (host order) for which to get a path advisory. > > > * > > > +* start_from > > > +* [in] Port number from where to start balance counting. > > > +* > > > * ignore_existing > > > * [in] Set to cause the switch to choose the optimal route > > > * regardless of existing paths. > > > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > > > index b96984b..60c6d25 100644 > > > --- a/opensm/opensm/osm_dump.c > > > +++ b/opensm/opensm/osm_dump.c > > > @@ -218,7 +218,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > > > else { > > > /* No LMC Optimization */ > > > best_port = osm_switch_recommend_path(p_sw, p_port, > > > - lid_ho, TRUE, dor); > > > + lid_ho, 1, TRUE, > > > + dor); > > > fprintf(file, "No %u hop path possible via port %u!", > > > best_hops, best_port); > > > } > > > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > > > index 58936e3..a9d13c8 100644 > > > --- a/opensm/opensm/osm_switch.c > > > +++ b/opensm/opensm/osm_switch.c > > > @@ -274,6 +274,7 @@ uint8_t > > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > IN osm_port_t * p_port, > > > IN const uint16_t lid_ho, > > > + IN unsigned start_from, > > > IN const boolean_t ignore_existing, > > > IN const boolean_t dor) > > > { > > > @@ -294,6 +295,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > uint8_t port_num; > > > uint8_t num_ports; > > > uint32_t least_paths = 0xFFFFFFFF; > > > + unsigned i; > > > /* > > > The follwing will track the least paths if the > > > route should go through a new system/node > > > @@ -397,8 +399,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > */ > > > > > > /* port number starts with one and num_ports is 1 + num phys ports */ > > > - for (port_num = 1; port_num < num_ports; port_num++) { > > > - if (osm_switch_get_hop_count(p_sw, base_lid, port_num) != > > > + for (i = start_from; i < start_from + num_ports; i++) { > > > + port_num = i%num_ports; > > > + if (!port_num || > > > + osm_switch_get_hop_count(p_sw, base_lid, port_num) != > > > least_hops) > > > continue; > > > > > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > > > index c073037..2aae6d5 100644 > > > --- a/opensm/opensm/osm_ucast_mgr.c > > > +++ b/opensm/opensm/osm_ucast_mgr.c > > > @@ -208,7 +208,8 @@ find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, > > > static void > > > __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > IN osm_switch_t * const p_sw, > > > - IN osm_port_t * const p_port) > > > + IN osm_port_t * const p_port, > > > + IN unsigned lid_offset) > > > { > > > uint16_t min_lid_ho; > > > uint16_t max_lid_ho; > > > @@ -217,19 +218,14 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > boolean_t is_ignored_by_port_prof; > > > ib_net64_t node_guid; > > > struct osm_routing_engine *p_routing_eng; > > > - /* > > > - The following are temporary structures that will aid > > > - in providing better routing in LMC > 0 situations > > > - */ > > > - uint16_t lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > > > - struct osm_remote_node *p_remote_guid_used = NULL; > > > + unsigned start_from = 1; > > > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > > > > > > - /* If the lids are zero - then there was some problem with the initialization. > > > - Don't handle this port. */ > > > + /* If the lids are zero - then there was some problem with > > > + * the initialization. Don't handle this port. */ > > > if (min_lid_ho == 0 || max_lid_ho == 0) { > > > OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A04: " > > > "Port 0x%" PRIx64 " has LID 0. An initialization " > > > @@ -238,16 +234,22 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > goto Exit; > > > } > > > > > > - if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { > > > + lid_ho = min_lid_ho + lid_offset; > > > + > > > + if (lid_ho > max_lid_ho) > > > + goto Exit; > > > + > > > + if (lid_offset) > > > + /* ignore potential overflow - it is handled in osm_switch.c */ > > > + start_from = osm_switch_get_port_by_lid(p_sw, lid_ho - 1) + 1; > > > + > > > + if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) > > > OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > - "Processing port 0x%" PRIx64 ", LIDs [0x%X,0x%X]\n", > > > - cl_ntoh64(osm_port_get_guid(p_port)), > > > + "Processing port 0x%" PRIx64 ", LID %u [0x%X,0x%X]\n", > > > + cl_ntoh64(osm_port_get_guid(p_port)), lid_ho, > > > min_lid_ho, max_lid_ho); > > > - } > > > > > > - /* > > > - TO DO - This should be runtime error, not a CL_ASSERT() > > > - */ > > > + /* TODO - This should be runtime error, not a CL_ASSERT() */ > > > CL_ASSERT(max_lid_ho < osm_switch_get_fwd_tbl_size(p_sw)); > > > > > > node_guid = osm_node_get_node_guid(p_sw->p_node); > > > @@ -260,80 +262,62 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > how best to distribute the LID range across the ports > > > that can reach those LIDs. > > > */ > > > - for (lid_ho = min_lid_ho; lid_ho <= max_lid_ho; lid_ho++) { > > > - /* Use the enhanced algorithm only for LMC > 0 */ > > > - if (lids_per_port > 1) { > > > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > > - p_mgr->p_subn-> > > > - ignore_existing_lfts, > > > - p_mgr->is_dor); > > > - if (port > 0 && port != OSM_NO_PATH && p_port->priv) > > > - p_remote_guid_used = > > > - find_and_add_remote_sys(p_sw, port, > > > - p_port->priv); > > > + port = osm_switch_recommend_path(p_sw, p_port, lid_ho, start_from, > > > + p_mgr->p_subn->ignore_existing_lfts, > > > + p_mgr->is_dor); > > > + > > > + if (port == OSM_NO_PATH) { > > > + /* do not try to overwrite the ppro of non existing port ... */ > > > + is_ignored_by_port_prof = TRUE; > > > + > > > + /* Up/Down routing can cause unreachable routes between some > > > + switches so we do not report that as an error in that case */ > > > + if (!p_routing_eng->build_lid_matrices) { > > > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > > > + "No path to get to LID 0x%X from switch 0x%" > > > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > > > + /* trigger a new sweep - try again ... */ > > > + p_mgr->p_subn->subnet_initialization_error = TRUE; > > > } else > > > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > > - p_mgr->p_subn-> > > > - ignore_existing_lfts, > > > - p_mgr->is_dor); > > > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > + "No path to get to LID 0x%X from switch 0x%" > > > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > > > + } else { > > > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > + "Routing LID 0x%X to port 0x%X" > > > + " for switch 0x%" PRIx64 "\n", > > > + lid_ho, port, cl_ntoh64(node_guid)); > > > > > > /* > > > - There might be no path to the target > > > + we would like to optionally ignore this port in equalization > > > + as in the case of the Mellanox Anafa Internal PCI TCA port > > > */ > > > - if (port == OSM_NO_PATH) { > > > - /* do not try to overwrite the ppro of non existing port ... */ > > > - is_ignored_by_port_prof = TRUE; > > > - > > > - /* Up/Down routing can cause unreachable routes between some > > > - switches so we do not report that as an error in that case */ > > > - if (!p_routing_eng->build_lid_matrices) { > > > - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > > > - "No path to get to LID 0x%X from switch 0x%" > > > - PRIx64 "\n", lid_ho, > > > - cl_ntoh64(node_guid)); > > > - /* trigger a new sweep - try again ... */ > > > - p_mgr->p_subn->subnet_initialization_error = > > > - TRUE; > > > - } else > > > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > - "No path to get to LID 0x%X from switch 0x%" > > > - PRIx64 "\n", lid_ho, > > > - cl_ntoh64(node_guid)); > > > - } else { > > > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > - "Routing LID 0x%X to port 0x%X" > > > - " for switch 0x%" PRIx64 "\n", > > > - lid_ho, port, cl_ntoh64(node_guid)); > > > - > > > - /* > > > - we would like to optionally ignore this port in equalization > > > - as in the case of the Mellanox Anafa Internal PCI TCA port > > > - */ > > > - is_ignored_by_port_prof = > > > - osm_port_prof_is_ignored_port(p_mgr->p_subn, > > > - node_guid, port); > > > - > > > - /* > > > - We also would ignore this route if the target lid is of a switch > > > - and the port_profile_switch_node is not TRUE > > > - */ > > > - if (!p_mgr->p_subn->opt.port_profile_switch_nodes) { > > > - is_ignored_by_port_prof |= > > > - (osm_node_get_type(p_port->p_node) == > > > - IB_NODE_TYPE_SWITCH); > > > - } > > > - } > > > + is_ignored_by_port_prof = > > > + osm_port_prof_is_ignored_port(p_mgr->p_subn, > > > + node_guid, port); > > > > > > /* > > > - We have selected the port for this LID. > > > - Write it to the forwarding tables. > > > + We also would ignore this route if the target lid is of > > > + a switch and the port_profile_switch_node is not TRUE > > > */ > > > - p_mgr->lft_buf[lid_ho] = port; > > > - if (!is_ignored_by_port_prof) { > > > - osm_switch_count_path(p_sw, port); > > > - if (p_remote_guid_used) > > > - p_remote_guid_used->forwarded_to++; > > > - } > > > + if (!p_mgr->p_subn->opt.port_profile_switch_nodes) > > > + is_ignored_by_port_prof |= > > > + (osm_node_get_type(p_port->p_node) == > > > + IB_NODE_TYPE_SWITCH); > > > + } > > > + > > > + /* > > > + We have selected the port for this LID. > > > + Write it to the forwarding tables. > > > + */ > > > + p_mgr->lft_buf[lid_ho] = port; > > > + if (!is_ignored_by_port_prof) { > > > + struct osm_remote_node *rem_node_used; > > > + osm_switch_count_path(p_sw, port); > > > + if (port > 0 && p_port->priv && > > > + (rem_node_used = find_and_add_remote_sys(p_sw, port, > > > + p_port->priv))) > > > + rem_node_used->forwarded_to++; > > > } > > > > > > Exit: > > > @@ -512,6 +496,7 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > > > osm_node_t *p_node; > > > osm_port_t *p_port; > > > const cl_qmap_t *p_port_tbl; > > > + unsigned i, lids_per_port; > > > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > > > @@ -538,12 +523,12 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > > > Iterate through every port setting LID routes for each > > > port based on base LID and LMC value. > > > */ > > > - > > > - for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > > > - p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > > > - p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) { > > > - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port); > > > - } > > > + lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > > > + for (i = 0; i < lids_per_port; i++) > > > + for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > > > + p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > > > + p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) > > > + __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port, i); > > > > > > osm_ucast_mgr_set_fwd_table(p_mgr, p_sw); > > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From chu11 at llnl.gov Tue Jun 17 09:25:45 2008 From: chu11 at llnl.gov (Al Chu) Date: Tue, 17 Jun 2008 09:25:45 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213639012.5369.443.camel@cardanus.llnl.gov> <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213719945.5369.454.camel@cardanus.llnl.gov> Hey Hal, > Other than the average bandwidth numbers at the top which show PL > > non PL, the ones in the table seem to be the other way around. Do > those numbers indicate bandwidth ? Sorry, the numbers for the mvapich/openmpi numbers are in units of time. I think microseconds. So lower is better. The "degradation" columns are the percent worse than LMC=0. The key thing to see is that originally, when LMC > 0 and multiple paths are not used, the numbers go up/it takes more time. When the preserving-base-lids patch is put in, the LMC > 0 numbers are about the same as the LMC=0 numbers. Al On Tue, 2008-06-17 at 07:51 -0700, Hal Rosenstock wrote: > Al, > > On Mon, 2008-06-16 at 10:56 -0700, Al Chu wrote: > > Hey Sasha, > > > > patch series seems to perform well. As a reminder mvapich 0.9.9 does > > not how to deal with multiple lids, openMPI 1.2.6 does. The "preserve > > lids" condition is labeled "PL". > > > > With LMC > 0, the preserve lids seems to do its job under mvapich 0.9.9, > > and atleast maintains (sometimes increases) performance of openmpi 1.2.6 > > with LMC > 0. Numbers are attached. > > Maybe I'm not reading this right but here's what I see: > > Other than the average bandwidth numbers at the top which show PL > non > PL, the ones in the table seem to be the other way around. Do those > numbers indicate bandwidth ? > > Also, you mentioned that mvapich doesn't know how to deal with LMC yet > the numbers seem to go up as LMC does. > > -- Hal > > > I'd give my thumbs up to the patch. Lets commit it. > > > > Al > > > > On Tue, 2008-06-10 at 05:59 +0300, Sasha Khapyorsky wrote: > > > Basically this addresses the problem described by Al Chu in: > > > > > > http://lists.openfabrics.org/pipermail/general/2008-April/049132.html > > > > > > When base lid paths become completely disbalanced on a fabrics with > > > lmc > 0. > > > > > > One feedback was from Yiftah Shahar: > > > > > > "I think that our requirements should be that even when you are working > > > with LMC>0 then the base LID routing should not be affected. > > > One way to achieve this goal is to first run the base-LID routing (so > > > all base LID improvement will be also in LMC>0) and then start with the > > > other LIDs as round-robbing starting from the base-lid-port + 1 > > > according current routing algorithm rules (keeping min-hop, up/down...)." > > > > > > We had some discussion with Al and Yiftah about this and considered that > > > in addition to "pure" base lid paths preservation (which is good thing by > > > itself) proposed method solves original lid disbalancing problem as well. > > > > > > This patch is implementation of the idea above. > > > > > > Signed-off-by: Sasha Khapyorsky > > > --- > > > opensm/include/opensm/osm_switch.h | 4 + > > > opensm/opensm/osm_dump.c | 3 +- > > > opensm/opensm/osm_switch.c | 8 ++- > > > opensm/opensm/osm_ucast_mgr.c | 163 ++++++++++++++++-------------------- > > > 4 files changed, 86 insertions(+), 92 deletions(-) > > > > > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > > > index 0e9c5fa..c1521a6 100644 > > > --- a/opensm/include/opensm/osm_switch.h > > > +++ b/opensm/include/opensm/osm_switch.h > > > @@ -981,6 +981,7 @@ uint8_t > > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > IN osm_port_t * p_port, > > > IN const uint16_t lid_ho, > > > + IN unsigned start_from, > > > IN const boolean_t ignore_existing, > > > IN const boolean_t dor); > > > /* > > > @@ -995,6 +996,9 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > * lid_ho > > > * [in] LID value (host order) for which to get a path advisory. > > > * > > > +* start_from > > > +* [in] Port number from where to start balance counting. > > > +* > > > * ignore_existing > > > * [in] Set to cause the switch to choose the optimal route > > > * regardless of existing paths. > > > diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c > > > index b96984b..60c6d25 100644 > > > --- a/opensm/opensm/osm_dump.c > > > +++ b/opensm/opensm/osm_dump.c > > > @@ -218,7 +218,8 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) > > > else { > > > /* No LMC Optimization */ > > > best_port = osm_switch_recommend_path(p_sw, p_port, > > > - lid_ho, TRUE, dor); > > > + lid_ho, 1, TRUE, > > > + dor); > > > fprintf(file, "No %u hop path possible via port %u!", > > > best_hops, best_port); > > > } > > > diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c > > > index 58936e3..a9d13c8 100644 > > > --- a/opensm/opensm/osm_switch.c > > > +++ b/opensm/opensm/osm_switch.c > > > @@ -274,6 +274,7 @@ uint8_t > > > osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > IN osm_port_t * p_port, > > > IN const uint16_t lid_ho, > > > + IN unsigned start_from, > > > IN const boolean_t ignore_existing, > > > IN const boolean_t dor) > > > { > > > @@ -294,6 +295,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > uint8_t port_num; > > > uint8_t num_ports; > > > uint32_t least_paths = 0xFFFFFFFF; > > > + unsigned i; > > > /* > > > The follwing will track the least paths if the > > > route should go through a new system/node > > > @@ -397,8 +399,10 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, > > > */ > > > > > > /* port number starts with one and num_ports is 1 + num phys ports */ > > > - for (port_num = 1; port_num < num_ports; port_num++) { > > > - if (osm_switch_get_hop_count(p_sw, base_lid, port_num) != > > > + for (i = start_from; i < start_from + num_ports; i++) { > > > + port_num = i%num_ports; > > > + if (!port_num || > > > + osm_switch_get_hop_count(p_sw, base_lid, port_num) != > > > least_hops) > > > continue; > > > > > > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > > > index c073037..2aae6d5 100644 > > > --- a/opensm/opensm/osm_ucast_mgr.c > > > +++ b/opensm/opensm/osm_ucast_mgr.c > > > @@ -208,7 +208,8 @@ find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, > > > static void > > > __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > IN osm_switch_t * const p_sw, > > > - IN osm_port_t * const p_port) > > > + IN osm_port_t * const p_port, > > > + IN unsigned lid_offset) > > > { > > > uint16_t min_lid_ho; > > > uint16_t max_lid_ho; > > > @@ -217,19 +218,14 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > boolean_t is_ignored_by_port_prof; > > > ib_net64_t node_guid; > > > struct osm_routing_engine *p_routing_eng; > > > - /* > > > - The following are temporary structures that will aid > > > - in providing better routing in LMC > 0 situations > > > - */ > > > - uint16_t lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > > > - struct osm_remote_node *p_remote_guid_used = NULL; > > > + unsigned start_from = 1; > > > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > > > osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); > > > > > > - /* If the lids are zero - then there was some problem with the initialization. > > > - Don't handle this port. */ > > > + /* If the lids are zero - then there was some problem with > > > + * the initialization. Don't handle this port. */ > > > if (min_lid_ho == 0 || max_lid_ho == 0) { > > > OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A04: " > > > "Port 0x%" PRIx64 " has LID 0. An initialization " > > > @@ -238,16 +234,22 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > goto Exit; > > > } > > > > > > - if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { > > > + lid_ho = min_lid_ho + lid_offset; > > > + > > > + if (lid_ho > max_lid_ho) > > > + goto Exit; > > > + > > > + if (lid_offset) > > > + /* ignore potential overflow - it is handled in osm_switch.c */ > > > + start_from = osm_switch_get_port_by_lid(p_sw, lid_ho - 1) + 1; > > > + > > > + if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) > > > OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > - "Processing port 0x%" PRIx64 ", LIDs [0x%X,0x%X]\n", > > > - cl_ntoh64(osm_port_get_guid(p_port)), > > > + "Processing port 0x%" PRIx64 ", LID %u [0x%X,0x%X]\n", > > > + cl_ntoh64(osm_port_get_guid(p_port)), lid_ho, > > > min_lid_ho, max_lid_ho); > > > - } > > > > > > - /* > > > - TO DO - This should be runtime error, not a CL_ASSERT() > > > - */ > > > + /* TODO - This should be runtime error, not a CL_ASSERT() */ > > > CL_ASSERT(max_lid_ho < osm_switch_get_fwd_tbl_size(p_sw)); > > > > > > node_guid = osm_node_get_node_guid(p_sw->p_node); > > > @@ -260,80 +262,62 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, > > > how best to distribute the LID range across the ports > > > that can reach those LIDs. > > > */ > > > - for (lid_ho = min_lid_ho; lid_ho <= max_lid_ho; lid_ho++) { > > > - /* Use the enhanced algorithm only for LMC > 0 */ > > > - if (lids_per_port > 1) { > > > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > > - p_mgr->p_subn-> > > > - ignore_existing_lfts, > > > - p_mgr->is_dor); > > > - if (port > 0 && port != OSM_NO_PATH && p_port->priv) > > > - p_remote_guid_used = > > > - find_and_add_remote_sys(p_sw, port, > > > - p_port->priv); > > > + port = osm_switch_recommend_path(p_sw, p_port, lid_ho, start_from, > > > + p_mgr->p_subn->ignore_existing_lfts, > > > + p_mgr->is_dor); > > > + > > > + if (port == OSM_NO_PATH) { > > > + /* do not try to overwrite the ppro of non existing port ... */ > > > + is_ignored_by_port_prof = TRUE; > > > + > > > + /* Up/Down routing can cause unreachable routes between some > > > + switches so we do not report that as an error in that case */ > > > + if (!p_routing_eng->build_lid_matrices) { > > > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > > > + "No path to get to LID 0x%X from switch 0x%" > > > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > > > + /* trigger a new sweep - try again ... */ > > > + p_mgr->p_subn->subnet_initialization_error = TRUE; > > > } else > > > - port = osm_switch_recommend_path(p_sw, p_port, lid_ho, > > > - p_mgr->p_subn-> > > > - ignore_existing_lfts, > > > - p_mgr->is_dor); > > > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > + "No path to get to LID 0x%X from switch 0x%" > > > + PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); > > > + } else { > > > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > + "Routing LID 0x%X to port 0x%X" > > > + " for switch 0x%" PRIx64 "\n", > > > + lid_ho, port, cl_ntoh64(node_guid)); > > > > > > /* > > > - There might be no path to the target > > > + we would like to optionally ignore this port in equalization > > > + as in the case of the Mellanox Anafa Internal PCI TCA port > > > */ > > > - if (port == OSM_NO_PATH) { > > > - /* do not try to overwrite the ppro of non existing port ... */ > > > - is_ignored_by_port_prof = TRUE; > > > - > > > - /* Up/Down routing can cause unreachable routes between some > > > - switches so we do not report that as an error in that case */ > > > - if (!p_routing_eng->build_lid_matrices) { > > > - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " > > > - "No path to get to LID 0x%X from switch 0x%" > > > - PRIx64 "\n", lid_ho, > > > - cl_ntoh64(node_guid)); > > > - /* trigger a new sweep - try again ... */ > > > - p_mgr->p_subn->subnet_initialization_error = > > > - TRUE; > > > - } else > > > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > - "No path to get to LID 0x%X from switch 0x%" > > > - PRIx64 "\n", lid_ho, > > > - cl_ntoh64(node_guid)); > > > - } else { > > > - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > > > - "Routing LID 0x%X to port 0x%X" > > > - " for switch 0x%" PRIx64 "\n", > > > - lid_ho, port, cl_ntoh64(node_guid)); > > > - > > > - /* > > > - we would like to optionally ignore this port in equalization > > > - as in the case of the Mellanox Anafa Internal PCI TCA port > > > - */ > > > - is_ignored_by_port_prof = > > > - osm_port_prof_is_ignored_port(p_mgr->p_subn, > > > - node_guid, port); > > > - > > > - /* > > > - We also would ignore this route if the target lid is of a switch > > > - and the port_profile_switch_node is not TRUE > > > - */ > > > - if (!p_mgr->p_subn->opt.port_profile_switch_nodes) { > > > - is_ignored_by_port_prof |= > > > - (osm_node_get_type(p_port->p_node) == > > > - IB_NODE_TYPE_SWITCH); > > > - } > > > - } > > > + is_ignored_by_port_prof = > > > + osm_port_prof_is_ignored_port(p_mgr->p_subn, > > > + node_guid, port); > > > > > > /* > > > - We have selected the port for this LID. > > > - Write it to the forwarding tables. > > > + We also would ignore this route if the target lid is of > > > + a switch and the port_profile_switch_node is not TRUE > > > */ > > > - p_mgr->lft_buf[lid_ho] = port; > > > - if (!is_ignored_by_port_prof) { > > > - osm_switch_count_path(p_sw, port); > > > - if (p_remote_guid_used) > > > - p_remote_guid_used->forwarded_to++; > > > - } > > > + if (!p_mgr->p_subn->opt.port_profile_switch_nodes) > > > + is_ignored_by_port_prof |= > > > + (osm_node_get_type(p_port->p_node) == > > > + IB_NODE_TYPE_SWITCH); > > > + } > > > + > > > + /* > > > + We have selected the port for this LID. > > > + Write it to the forwarding tables. > > > + */ > > > + p_mgr->lft_buf[lid_ho] = port; > > > + if (!is_ignored_by_port_prof) { > > > + struct osm_remote_node *rem_node_used; > > > + osm_switch_count_path(p_sw, port); > > > + if (port > 0 && p_port->priv && > > > + (rem_node_used = find_and_add_remote_sys(p_sw, port, > > > + p_port->priv))) > > > + rem_node_used->forwarded_to++; > > > } > > > > > > Exit: > > > @@ -512,6 +496,7 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > > > osm_node_t *p_node; > > > osm_port_t *p_port; > > > const cl_qmap_t *p_port_tbl; > > > + unsigned i, lids_per_port; > > > > > > OSM_LOG_ENTER(p_mgr->p_log); > > > > > > @@ -538,12 +523,12 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, > > > Iterate through every port setting LID routes for each > > > port based on base LID and LMC value. > > > */ > > > - > > > - for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > > > - p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > > > - p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) { > > > - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port); > > > - } > > > + lids_per_port = 1 << p_mgr->p_subn->opt.lmc; > > > + for (i = 0; i < lids_per_port; i++) > > > + for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); > > > + p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); > > > + p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) > > > + __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port, i); > > > > > > osm_ucast_mgr_set_fwd_table(p_mgr, p_sw); > > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From chu11 at llnl.gov Tue Jun 17 09:35:49 2008 From: chu11 at llnl.gov (Al Chu) Date: Tue, 17 Jun 2008 09:35:49 -0700 Subject: [ofa-general] [OPENSM PATCH 0/5]: New "guid-routing-order" option for updn routing In-Reply-To: <485798F4.40201@dev.mellanox.co.il> References: <1213397313.5369.356.camel@cardanus.llnl.gov> <4854CFFC.7040508@dev.mellanox.co.il> <1213636881.5369.438.camel@cardanus.llnl.gov> <1213646932.5369.446.camel@cardanus.llnl.gov> <485772FD.4000206@dev.mellanox.co.il> <485798F4.40201@dev.mellanox.co.il> Message-ID: <1213720549.5369.457.camel@cardanus.llnl.gov> Hey Yevgeny, On Tue, 2008-06-17 at 13:59 +0300, Yevgeny Kliteynik wrote: > Yevgeny Kliteynik wrote: > > Hi Al, > > > > Al Chu wrote: > >> On Mon, 2008-06-16 at 10:21 -0700, Al Chu wrote: > >>> Hey Yevgeny, > >>> > >>> On Sun, 2008-06-15 at 11:17 +0300, Yevgeny Kliteynik wrote: > >>>> Hi Al, > >>>> > >>>> Al Chu wrote: > >>>>> Hey Sasha, > >>>>> > >>>>> This is a conceptually simple option I've developed for updn routing. > >>>>> > >>>>> Currently in updn routing, nodes/guids are routed on switches in a > >>>>> seemingly-random order, which I believe is due to internal data > >>>>> structure organization (i.e. cl_qmap_apply_func is called on > >>>>> port_guid_tbl) as well as how the fabric is scanned (it is logically > >>>>> scanned from a port perspective, but it may not be logical from a node > >>>>> perspective). I had a hypothesis that this was leading to increased > >>>>> contention in the network for MPI. > >>>>> > >>>>> For example, suppose we have 12 uplinks from a leaf switch to a spine > >>>>> switch. If we want to send data from this leaf switch to node[13-24], > >>>>> the up links we will send on are pretty random. It's because: > >>>>> > >>>>> A) node[13-24] are individually routed at seemingly-random points > >>>>> based > >>>>> on when they are called by cl_qmap_apply_func(). > >>>>> > >>>>> B) the ports chosen for routing are based on least used port usage. > >>>>> > >>>>> C) least used port usage is based on whatever was routed earlier on. > >>>>> > >>>>> So I developed this patch series, which supports an option called > >>>>> "guid_routing_order_file" which allows the user to input a file with a > >>>>> list of port_guids which will indicate the order in which guids are > >>>>> routed instead (naturally, those guids not listed are routed last). > >>>> Great idea! > >>> Thanks. > >>> > >>>> I understand that this guid_routing_order_file is synchronized with > >>>> an MPI rank file, right? If not, then synchronizing them might give > >>>> even better results. > >>> Not quite sure what you mean by a MPI rank file. At LLNL, slurm is > >>> responsible for MPI ranks, so I order the guids in my file according to > >>> how slurm is configured for chosing MPI ranks. I will admit to being a > >>> novice to MPI's configuration (blindly accepting slurm MPI rankings). > >>> Is there an underlying file that MPI libs use for ranking knowledge? > >> > >> I spoke to one of our MPI guys. I wasn't aware that in some MPIs you > >> can input a file to tell it how ranks should be assigned to nodes for > >> MPI. I assume that's what you're talking about? > > > > Yes, that is what I was talking about. > > There is a host file, where you list all the hosts that MPI should use, > > and in some MPIs there is also a way to specify the order of MPI ranks > > that would be assigned to processes (I'm not an MPI expert, so I'm not > > sure about the terminology that I use). > > I know that MVAPICH is using the host order when assigning ranks, so > > the order of the cluster nodes listed in host file is important. > > Not sure about OpenMPI. > > > >>>> Another idea: OpenSM can create such file (list, doesn't have to be > >>>> actual file) automatically, just by checking topologically-adjacent > >>>> leaf switches and their HCAs. > >>> Definitely a good idea. This patch set was just a "step one" kind of > >>> thing. > >>> > >>>>> I list the port guids of the nodes of the cluster from node0 to > >>>>> nodeN, one > >>>>> per line in the file. By listing the nodes in this order, I > >>>>> believe we > >>>>> could get less contention in the network. In the example above, > >>>>> sending > >>>>> to node[13-24] should use all of the 12 uplinks, b/c the ports will be > >>>>> equally used b/c nodes[1-12] were routed beforehand in order. > >>>>> > >>>>> The results from some tests are pretty impressive when I do this. > >>>>> LMC=0 > >>>>> average bandwidth in mpiGraph goes from 391.374 MB/s to 573.678 MB/s > >>>>> when I use guid_routing_order. > >>>> Can you compare this to the fat-tree routing? Conceptually, fat-tree > >>>> is doing the same - it routes LIDs on nodes in a topological order, so > >>>> it would be interesting to see the comparison. > >>> Actually I already did :-). w/ LMC=0. > >>> > >>> updn default - 391.374 MB/s > >>> updn w/ guid_routing_order - 573.678 MB/s > >>> ftree - 579.603 MB/s > >>> > >>> I later discovered that one of the internal ports of the cluster I'm > >>> testing on was broken (sLB of a 288 port), and think that is the cause > >>> of some of the slowdown w/ updn w/ guid_routing_order. So ftree (as > >>> designed) seemed to be able to work around it properly, while updn (as > >>> currently implemented) couldn't. > >>> > >>> When we turn on LMC > 0, mpi libraries that are LMC > 0 aware were able > >>> to do better on some tests than ftree. One example (I think these > >>> numbers are in microseconds. Lower is better): > >>> > >>> Alltoall 16K packets > >>> ftree - 415490.6919 > >>> updn normal (LMC=0) - 495460.5526 > >>> updn w/ ordered routing (LMC=0) - 416562.7417 > >>> updn w/ ordered routing (LMC=1) - 453153.7289 > >>> - this ^^^ result is quite odd. Not sure why. > >>> updn w/ ordered routing (LMC=2) - 3660132.1530 > >>> > >>> We are regularly debating what will be better overall at the end of the > >>> day. > >>> > >>>> Also, fat-tree produces the guid order file automatically, but nobody > >>>> used it yet as an input to produce MPI rank file. > >>> I didn't know about this option. How do you do this (just skimmed the > >>> manpage, didn't see anything)? > > > > Right, it's missing there. I'll add this info. > > Nope, it's there: > > "The algorithm also dumps compute node ordering file (opensm-ftree-ca-order.dump) > in the same directory where the OpenSM log resides. This ordering file provides > the CN order that may be used to create efficient communication pattern, that > will match the routing tables." Thanks. I guess I just missed it. The manpage is getting big :-) Al > -- Yevgeny > > > > The file is /var/log/opensm-ftree-ca-order.dump. > > Small correction though - the file contains ordered list of HCA LIDs > > and their host names. It's not a problem to change it to have guids > > as well, but MPI doesn't need guids anyway. > > Note that the optimal order might be different depending on the current > > topology state and the location of the management node that runs OpenSM. > > > >>> I know about the --cn_guid_file. But > >>> since that file doesn't have to be ordered, that's why I created a > >>> different option (rather than have the cn_guid_file for both ftree and > >>> updn). > > > > Right, the cn file doesn't have to be ordered - ftree will order it > > by itself. The ordering is by topology-adjacent leaf switches. > > > > -- Yevgeny > > > >>> > >>> Al > >>> > >>>> -- Yevgeny > >>>> > >>>>> A variety of other positive performance > >>>>> increases were found when doing other tests, other MPIs, and other > >>>>> LMCs > >>>>> if anyone is interested. > >>>>> > >>>>> BTW, I developed this patch series before your preserve-base-lid patch > >>>>> series. It will 100% conflict with the preserve-base-lid patch > >>>>> series. > >>>>> I will fix this patch series once the preserve-base-lids patch > >>>>> series is > >>>>> committed to git. I'm just looking for comments right now. > >>>>> > >>>>> Al > >>>>> > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From hrosenstock at xsigo.com Tue Jun 17 09:43:20 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 09:43:20 -0700 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213719945.5369.454.camel@cardanus.llnl.gov> References: <20080610025621.GJ10301@sashak.voltaire.com> <20080610025939.GK10301@sashak.voltaire.com> <1213639012.5369.443.camel@cardanus.llnl.gov> <1213714314.13056.35.camel@hrosenstock-ws.xsigo.com> <1213719945.5369.454.camel@cardanus.llnl.gov> Message-ID: <1213721000.13056.38.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-17 at 09:25 -0700, Al Chu wrote: > Sorry, the numbers for the mvapich/openmpi numbers are in units of > time. > I think microseconds. So lower is better. That's what wasn't clear. -- Hal From hrosenstock at xsigo.com Tue Jun 17 09:51:18 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 09:51:18 -0700 Subject: [ofa-general] IB network problem - "Unknown remote side for node" In-Reply-To: References: Message-ID: <1213721478.13056.44.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-17 at 11:07 -0400, Michael Di Domenico wrote: > Can anyone tell me what i likely causing this? The SM seems to be in > a loop entering/exiting, Are you saying the SM exits all on its own ? If so, is there some script that restarts it ? > and the "Unknown remote side" comes up with a different list of port > each time it cycles. That's weird. How different are the ports ? Unknown remote side is due to a port which is not physically DOWN but perhaps unresponsive (or slow response). > I though it was bad cables, but since it keeps changing, that seems > unlikely. > > Thanks > - Michael > > > Jun 17 04:05:51 681669 [AAF0D060] -> OpenSM Rev:openib-2.0.5 OpenIB > svn 9905 This looks close to OFED 1.1. Any chance of updating to a more recent OpenSM version ? > Jun 17 04:05:51 681723 [AAF0D060] -> OpenSM Rev:openib-2.0.5 OpenIB > svn 9905 > Jun 17 04:05:51 686837 [AAF0D060] -> osm_vendor_bind: Binding to port > 0x2c9030000792e > Jun 17 04:05:51 689709 [AAF0D060] -> osm_vendor_bind: Binding to port > 0x2c9030000792e > Jun 17 04:05:52 785232 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff00510d port 4. Adding to > light sweep sampling list > Jun 17 04:05:52 785273 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785281 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff00510d port 6. Adding to > light sweep sampling list Can you check the ports indicated and the switch to see if they are responsive ? -- Hal > Jun 17 04:05:52 785290 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785296 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff00510d port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785305 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785312 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff00510d port 10. Adding to > light sweep sampling list > Jun 17 04:05:52 785319 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785325 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff00510d port 12. Adding to > light sweep sampling list > Jun 17 04:05:52 785334 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][13] > Jun 17 04:05:52 785352 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff005118 port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785359 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][4] > Jun 17 04:05:52 785424 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff00507a port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785431 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][6] > Jun 17 04:05:52 785444 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff005094 port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785454 [46409940] -> Directed Path Dump of 2 hop path: > Path = [0][2][3] > Jun 17 04:05:52 785467 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff005095 port 10. Adding to > light sweep sampling list > Jun 17 04:05:52 785475 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][8][2] > Jun 17 04:05:52 785501 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050a0 port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785509 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][8] > Jun 17 04:05:52 785536 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050af port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785545 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][C] > Jun 17 04:05:52 785553 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050b0 port 8. Adding to > light sweep sampling list > Jun 17 04:05:52 785561 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][13][A] > Jun 17 04:05:52 785602 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050c1 port 4. Adding to > light sweep sampling list > Jun 17 04:05:52 785631 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][D] > Jun 17 04:05:52 785645 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050c4 port 4. Adding to > light sweep sampling list > Jun 17 04:05:52 785651 [46409940] -> Directed Path Dump of 3 hop path: > Path = [0][2][3][8] > Jun 17 04:05:52 785662 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050c6 port 4. Adding to > light sweep sampling list > Jun 17 04:05:52 785671 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][D][4] > Jun 17 04:05:52 785686 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050d2 port 2. Adding to > light sweep sampling list > Jun 17 04:05:52 785694 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][8] > Jun 17 04:05:52 785702 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050d2 port 12. Adding to > light sweep sampling list > Jun 17 04:05:52 785710 [46409940] -> Directed Path Dump of 4 hop path: > Path = [0][2][3][11][8] > Jun 17 04:05:52 785741 [46409940] -> osm_drop_mgr_process: ERR 0108: > Unknown remote side for node 0x000b8cffff0050fa port 10. Adding to > light sweep sampling list > Jun 17 04:05:52 785748 [46409940] -> Directed Path Dump of 5 hop path: > Path = [0][2][3][11][8][C] > Jun 17 04:05:52 785774 [46409940] -> Entering MASTER state > Jun 17 04:05:52 786126 [46409940] -> osm_report_notice: Reporting > Generic Notice type:3 num:66 from LID:0x0000 > GID:0x000000000000e80f,0x0002c9030000792e > Jun 17 04:05:52 786254 [46409940] -> osm_report_notice: Reporting > Generic Notice type:3 num:66 from LID:0x0000 > GID:0x000000000000e80f,0x0002c9030000792e > Jun 17 04:05:56 691253 [AAF0D060] -> Exiting SM > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sean.hefty at intel.com Tue Jun 17 09:58:22 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 17 Jun 2008 09:58:22 -0700 Subject: [ofa-general] RE: local-sa diff between ofed 1.2 and 1.3 In-Reply-To: References: Message-ID: <000001c8d09b$599a93b0$bb37170a@amr.corp.intel.com> >I took a look on the local-sa and it turns that its been through quite a >big change between the last two ofed releases (below). Does this diff >relates to the structural change of the local-sa being now embedded in the >ib-sa module? is it documented somewhere (eg by change-log of patch/es >in a git tree)? looking at your rdma-dev git tree I didn't evidence to >this change... There were several changes. Once was integrating it into the ib_sa module. But it also replaced module parameters with user-level controls that could be set using sysfs, along with SA event support. - Sean From hrosenstock at xsigo.com Tue Jun 17 10:03:21 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 10:03:21 -0700 Subject: [ofa-general] Re: [PATCHv3][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements In-Reply-To: <20080616151603.GG13363@sashak.voltaire.com> References: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> <20080615063727.GK24365@sashak.voltaire.com> <1213621882.14047.611.camel@hrosenstock-ws.xsigo.com> <20080616151603.GG13363@sashak.voltaire.com> Message-ID: <1213722201.13056.55.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 18:16 +0300, Sasha Khapyorsky wrote: > On 06:11 Mon 16 Jun , Hal Rosenstock wrote: > > > > @@ -1278,8 +1281,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, > > > > } > > > > } else { > > > > CL_PLOCK_RELEASE(sa->p_lock); > > > > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > > > - "Failed since multicast group not present\n"); > > > > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " > > > > + "Failed since multicast group 0x%16" > > > > > > Why to change log level? It is not OpenSM error, but invalid request. > > > > Are errors really restricted to OpenSM (internal) errors ? > > I guess it should be. Why ? ERROR level shows up always and that is useful for things beyond internal errors (and that is used in OpenSM in a variety of places). -- Hal > > I thought > > they were more inclusive than that including network events that were > > "interesting events". > > We have VERBOSE level for "interesting" events and INFO for "very > interesting" ones. > > In this specific case, I agree it is a bad idea as I think restarting > > IPoIB does causes this code path to be used. Modified patch already > > issued. > > Cool. Thanks. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Tue Jun 17 10:07:05 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 17 Jun 2008 10:07:05 -0700 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c : TRIVIAL fixes in log handling In-Reply-To: <20080617162119.GE25554@sashak.voltaire.com> References: <48579A86.5070508@dev.mellanox.co.il> <20080617162119.GE25554@sashak.voltaire.com> Message-ID: <1213722425.13056.60.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-17 at 19:21 +0300, Sasha Khapyorsky wrote: > we have a lot of useless logging right now - with -V initial OpenSM > log files can grow to GBs. Is the problem only during initialization ? If so, would an option to disable "unimportant" logging during initialization be sufficient for this ? Or is the problem beyond initialization and all logging needs to be inspected for whether it is appropriate and what level it occurs at ? -- Hal From sashak at voltaire.com Tue Jun 17 10:40:25 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 20:40:25 +0300 Subject: [ofa-general] Re: [PATCHv3][TRIVIAL] opensm/osm_sa_mcmember_record.c: Some error message improvements In-Reply-To: <1213722201.13056.55.camel@hrosenstock-ws.xsigo.com> References: <1213371041.14047.556.camel@hrosenstock-ws.xsigo.com> <20080615063727.GK24365@sashak.voltaire.com> <1213621882.14047.611.camel@hrosenstock-ws.xsigo.com> <20080616151603.GG13363@sashak.voltaire.com> <1213722201.13056.55.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080617174025.GG25554@sashak.voltaire.com> On 10:03 Tue 17 Jun , Hal Rosenstock wrote: > On Mon, 2008-06-16 at 18:16 +0300, Sasha Khapyorsky wrote: > > On 06:11 Mon 16 Jun , Hal Rosenstock wrote: > > > > > @@ -1278,8 +1281,11 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, > > > > > } > > > > > } else { > > > > > CL_PLOCK_RELEASE(sa->p_lock); > > > > > - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > > > > - "Failed since multicast group not present\n"); > > > > > + OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B31: " > > > > > + "Failed since multicast group 0x%16" > > > > > > > > Why to change log level? It is not OpenSM error, but invalid request. > > > > > > Are errors really restricted to OpenSM (internal) errors ? > > > > I guess it should be. > > Why ? ERROR level shows up always Yes, internal errors or subnet errors - IOW events which require some action. > and that is useful for things beyond > internal errors (and that is used in OpenSM in a variety of places). Ok I said this badly - not only OpenSM internal error, but also subnet related, but when it is errors, not just for verbose or debug tracking. Sasha From sashak at voltaire.com Tue Jun 17 11:41:03 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Jun 2008 21:41:03 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c : TRIVIAL fixes in log handling In-Reply-To: <1213722425.13056.60.camel@hrosenstock-ws.xsigo.com> References: <48579A86.5070508@dev.mellanox.co.il> <20080617162119.GE25554@sashak.voltaire.com> <1213722425.13056.60.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080617184103.GA31935@sashak.voltaire.com> On 10:07 Tue 17 Jun , Hal Rosenstock wrote: > > Is the problem only during initialization ? No, not only there. Just saying that huge log are generated just few seconds after OpenSM starts. > If so, would an option to > disable "unimportant" logging during initialization be sufficient for > this ? Or is the problem beyond initialization and all logging needs to > be inspected for whether it is appropriate and what level it occurs at ? Yes, I think it is general problem and all logging needs review. Sasha From kliteyn at dev.mellanox.co.il Tue Jun 17 13:16:46 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 23:16:46 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c : TRIVIAL fixes in log handling In-Reply-To: <20080617162119.GE25554@sashak.voltaire.com> References: <48579A86.5070508@dev.mellanox.co.il> <20080617162119.GE25554@sashak.voltaire.com> Message-ID: <48581BAE.7030108@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Hi Yevgeny, > > On 14:05 Tue 17 Jun , Yevgeny Kliteynik wrote: >> OSM_LOG_ENTER() at the beginning of the function and >> OSM_LOG_EXIT() at the end of the function were missing. > > Why? The function is pretty isolated and we have a lot of useless > logging right now - with -V initial OpenSM log files can grow to GBs. There's an OSM_LOG_EXIT() in this function w/o OSM_LOG_ENTER(). I can just remove the OSM_LOG_EXIT() instead - fine by me. -- Yevgeny > Sasha > From kliteyn at dev.mellanox.co.il Tue Jun 17 13:19:18 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Tue, 17 Jun 2008 23:19:18 +0300 Subject: [ofa-general] [PATCH] ibutils: fixing seg. fault in ibis_gsi_mad_ctrl.c In-Reply-To: <1213713241.13056.31.camel@hrosenstock-ws.xsigo.com> References: <48356525.9010700@dev.mellanox.co.il> <1213712793.13056.29.camel@hrosenstock-ws.xsigo.com> <1213713241.13056.31.camel@hrosenstock-ws.xsigo.com> Message-ID: <48581C46.3070208@dev.mellanox.co.il> Hi Hal, Hal Rosenstock wrote: > On Tue, 2008-06-17 at 07:26 -0700, Hal Rosenstock wrote: >> On Thu, 2008-05-22 at 15:20 +0300, Yevgeny Kliteynik wrote: >>> Hi Oren, >>> >>> Fixing seg fault in allocation of gsi management class vector. >> Was this ever committed ? Should it be applied for both trunk and >> ofed_1_3 ? > > Looks like it's there for trunk but not ofed_1_3. It's not needed for ofed_1_3 - the seg fault happens only when adding a new feature to ibutils (support for CC MADs), which doesn't exist in ofed_1_3. -- Yevgeny > -- Hal > >> -- Hal >> >>> Signed-off-by: Yevgeny Kliteynik >>> --- >>> ibis/src/ibis_gsi_mad_ctrl.c | 2 +- >>> 1 files changed, 1 insertions(+), 1 deletions(-) >>> >>> diff --git a/ibis/src/ibis_gsi_mad_ctrl.c b/ibis/src/ibis_gsi_mad_ctrl.c >>> index 356d33d..bfb5fe6 100644 >>> --- a/ibis/src/ibis_gsi_mad_ctrl.c >>> +++ b/ibis/src/ibis_gsi_mad_ctrl.c >>> @@ -731,7 +731,7 @@ ibis_gsi_mad_ctrl_set_class_attr_cb( >>> size = cl_vector_get_size(&p_ctrl->class_vector); >>> if (size <= mad_class) >>> { >>> - cl_status = cl_vector_set_size(&p_ctrl->class_vector,mad_class); >>> + cl_status = cl_vector_set_size(&p_ctrl->class_vector,mad_class+1); >>> >>> if( cl_status != CL_SUCCESS) >>> { >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From okir at lst.de Tue Jun 17 13:59:05 2008 From: okir at lst.de (Olaf Kirch) Date: Tue, 17 Jun 2008 22:59:05 +0200 Subject: [ofa-general] Several RDS patches In-Reply-To: <200806171722.02780.olaf.kirch@oracle.com> References: <200806171722.02780.olaf.kirch@oracle.com> Message-ID: <200806172259.05882.okir@lst.de> On Tuesday 17 June 2008 17:22:02 Olaf Kirch wrote: > Hi Vlad, > > during testing of RDS, a number of issues came up that I had to > address. Some of them were bugs, others were usability issues. > > I submitted the following patches to > git://git.openfabrics.org/~okir/ofed_1_3/linux-2.6.git code-drop-20080617 I just added another patch to that branch; please pull this one as well. Thanks! Olaf ---------------------------- commit 1bc8f17379a43106aeb604fbfe9a78687df1e7ab Author: Olaf Kirch Date: Tue Jun 17 22:57:30 2008 +0200 RDS: Remove spurious variable name from stats A previous patch introduced a spurious name in rds_stat_names[], causing some of the values reported by rds-info to no longer match the names they were reported for. Signed-off-by: Olaf Kirch diff --git a/net/rds/stats.c b/net/rds/stats.c index d1c40d7..06f9971 100644 --- a/net/rds/stats.c +++ b/net/rds/stats.c @@ -68,7 +68,6 @@ static char *rds_stat_names[] = { "send_pong", "page_remainder_hit", "page_remainder_miss", - "page_copy_nonatomic", "copy_to_user", "copy_from_user", "cong_update_queued", -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play okir at lst.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax From swise at opengridcomputing.com Tue Jun 17 14:38:56 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 17 Jun 2008 16:38:56 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. Message-ID: <20080617213856.8122.96180.stgit@dell3.ogc.int> This patch adds a sysfs attribute group called "proto_stats" under /sys/class/infiniband/$device/ and populates this group with protocol statistics if they exist for a given device. Currently, only iWARP stats are defined, but the code is designed to allow Infiniband protocol stats if they become available. These stats are per-device and more importantly -not- per port. Details: - Add union rdma_protocol_stats in ib_verbs.h. This union allows defining transport-specific stats. Currently only iwarp stats are defined. - Add struct iw_protocol_stats to define the current set of iwarp protocol stats. - Add new ib_device method called get_proto_stats() to return protocol statistics. - Add logic in core/sysfs.c to create iwarp protocol stats attributes if the device is an RNIC and has a get_proto_stats() method. Signed-off-by: Steve Wise --- drivers/infiniband/core/sysfs.c | 102 +++++++++++++++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 39 +++++++++++++++ 2 files changed, 141 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 9575655..97bdbe7 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -665,6 +665,103 @@ static struct class ib_class = { .dev_uevent = ib_device_uevent, }; +/* Show a given an attribute in the statistics group */ +static ssize_t show_protocol_stat(const struct device *device, + struct device_attribute *attr, char *buf, + int width, unsigned offset) +{ + struct ib_device *dev = container_of(device, struct ib_device, dev); + union rdma_protocol_stats stats; + ssize_t ret = -EINVAL; + + WARN_ON(offset > sizeof(union rdma_protocol_stats) || + offset % (width == 32 ? 4 : 8) != 0); + + if (!dev->get_protocol_stats) + goto out; + ret = (*dev->get_protocol_stats)(dev, &stats); + if (ret) + goto out; + if (width == 32) + ret = sprintf(buf, "%u\n", *(u32 *)(((u8 *)&stats) + offset)); + else if (width == 64) + ret = sprintf(buf, "%llu\n", *(u64 *)(((u8 *)&stats) + offset)); + else + goto out; +out: + return ret; +} + +/* generate a read-only iwarp statistics attribute */ +#define IW_STATS_ENTRY(name, width) \ +static ssize_t show_##name(struct device *device, \ + struct device_attribute *attr, char *buf) \ +{ \ + return show_protocol_stat(device, attr, buf, width, \ + offsetof(struct iw_protocol_stats, name)); \ +} \ +static DEVICE_ATTR(name, S_IRUGO, show_##name, NULL) + +IW_STATS_ENTRY(tcpInSegs, 64); +IW_STATS_ENTRY(tcpOutSegs, 64); +IW_STATS_ENTRY(tcpRetransSeg, 64); +IW_STATS_ENTRY(tcpInErrs, 64); +IW_STATS_ENTRY(tcpActiveOpens, 32); +IW_STATS_ENTRY(tcpPassiveOpens, 32); +IW_STATS_ENTRY(tcpAttemptFails, 32); +IW_STATS_ENTRY(tcpEstabResets, 32); +IW_STATS_ENTRY(tcpOutRsts, 32); +IW_STATS_ENTRY(tcpCurrEstab, 32); +IW_STATS_ENTRY(tcpRtoMin, 32); +IW_STATS_ENTRY(tcpRtoMax, 32); +IW_STATS_ENTRY(ipInReceive, 64); +IW_STATS_ENTRY(ipInHdrErrors, 64); +IW_STATS_ENTRY(ipInAddrErrors, 64); +IW_STATS_ENTRY(ipInUnknownProtos, 64); +IW_STATS_ENTRY(ipInDiscards, 64); +IW_STATS_ENTRY(ipInDelivers, 64); +IW_STATS_ENTRY(ipOutRequests, 64); +IW_STATS_ENTRY(ipOutDiscards, 64); +IW_STATS_ENTRY(ipOutNoRoutes, 64); +IW_STATS_ENTRY(ipReasmTimeout, 32); +IW_STATS_ENTRY(ipReasmReqds, 32); +IW_STATS_ENTRY(ipReasmOKs, 32); +IW_STATS_ENTRY(ipReasmFails, 32); + +static struct attribute *iw_proto_stats_attrs[] = { + &dev_attr_tcpInSegs.attr, + &dev_attr_tcpOutSegs.attr, + &dev_attr_tcpRetransSeg.attr, + &dev_attr_tcpInErrs.attr, + &dev_attr_tcpActiveOpens.attr, + &dev_attr_tcpPassiveOpens.attr, + &dev_attr_tcpAttemptFails.attr, + &dev_attr_tcpEstabResets.attr, + &dev_attr_tcpOutRsts.attr, + &dev_attr_tcpCurrEstab.attr, + &dev_attr_tcpRtoMin.attr, + &dev_attr_tcpRtoMax.attr, + &dev_attr_ipInReceive.attr, + &dev_attr_ipInHdrErrors.attr, + &dev_attr_ipInAddrErrors.attr, + &dev_attr_ipInUnknownProtos.attr, + &dev_attr_ipInDiscards.attr, + &dev_attr_ipInDelivers.attr, + &dev_attr_ipOutRequests.attr, + &dev_attr_ipOutDiscards.attr, + &dev_attr_ipOutNoRoutes.attr, + &dev_attr_ipReasmTimeout.attr, + &dev_attr_ipReasmReqds.attr, + &dev_attr_ipReasmOKs.attr, + &dev_attr_ipReasmFails.attr, + NULL +}; + +static struct attribute_group iw_stats_group = { + .name = "proto_stats", + .attrs = iw_proto_stats_attrs, +}; + int ib_device_register_sysfs(struct ib_device *device) { struct device *class_dev = &device->dev; @@ -707,6 +804,11 @@ int ib_device_register_sysfs(struct ib_device *device) } } + if (device->node_type == RDMA_NODE_RNIC && device->get_protocol_stats) { + ret = sysfs_create_group(&class_dev->kobj, &iw_stats_group); + if (ret) + goto err_put; + } return 0; err_put: diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 62e5bcd..0260645 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -53,6 +53,43 @@ #include #include +struct ib_protocol_stats { + /* TBD... */ +}; + +struct iw_protocol_stats { + u64 tcpInSegs; + u64 tcpOutSegs; + u64 tcpRetransSeg; + u64 tcpInErrs; + u32 tcpActiveOpens; + u32 tcpPassiveOpens; + u32 tcpAttemptFails; + u32 tcpEstabResets; + u32 tcpOutRsts; + u32 tcpCurrEstab; + u32 tcpRtoMin; + u32 tcpRtoMax; + u64 ipInReceive; + u64 ipInHdrErrors; + u64 ipInAddrErrors; + u64 ipInUnknownProtos; + u64 ipInDiscards; + u64 ipInDelivers; + u64 ipOutRequests; + u64 ipOutDiscards; + u64 ipOutNoRoutes; + u32 ipReasmTimeout; + u32 ipReasmReqds; + u32 ipReasmOKs; + u32 ipReasmFails; +}; + +union rdma_protocol_stats { + struct ib_protocol_stats ib; + struct iw_protocol_stats iw; +}; + union ib_gid { u8 raw[16]; struct { @@ -947,6 +984,8 @@ struct ib_device { struct iw_cm_verbs *iwcm; + int (*get_protocol_stats)(struct ib_device *device, + union rdma_protocol_stats *stats); int (*query_device)(struct ib_device *device, struct ib_device_attr *device_attr); int (*query_port)(struct ib_device *device, From swise at opengridcomputing.com Tue Jun 17 14:38:59 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 17 Jun 2008 16:38:59 -0500 Subject: [ofa-general] [PATCH RFCv2 2/2] RDMA/cxgb3: Add support for protocol statistics. In-Reply-To: <20080617213856.8122.96180.stgit@dell3.ogc.int> References: <20080617213856.8122.96180.stgit@dell3.ogc.int> Message-ID: <20080617213858.8122.64051.stgit@dell3.ogc.int> - Add a new rdma ctl command called RDMA_GET_MIB to the cxgb3 low level driver to obtain the protocol mib from the rnic hardware. - Add new iw_cxgb3 provider method to get the MIB from the low level driver. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 50 +++++++++++++++++++++++++-- drivers/net/cxgb3/cxgb3_ctl_defs.h | 1 + drivers/net/cxgb3/cxgb3_offload.c | 7 ++++ 3 files changed, 54 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 193dfe7..89af953 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -56,6 +56,7 @@ #include "iwch_provider.h" #include "iwch_cm.h" #include "iwch_user.h" +#include "common.h" static int iwch_modify_port(struct ib_device *ibdev, u8 port, int port_modify_mask, @@ -1210,6 +1211,49 @@ static ssize_t show_board(struct device *dev, struct device_attribute *attr, iwch_dev->rdev.rnic_info.pdev->device); } +static int iwch_get_mib(struct ib_device *ibdev, + union rdma_protocol_stats *stats) +{ + struct iwch_dev *dev; + struct tp_mib_stats m; + int ret; + + PDBG("%s ibdev %p\n", __func__, ibdev); + dev = to_iwch_dev(ibdev); + ret = dev->rdev.t3cdev_p->ctl(dev->rdev.t3cdev_p, RDMA_GET_MIB, &m); + if (ret) + return -ENOSYS; + + memset(stats, 0, sizeof *stats); + stats->iw.ipInReceive = m.ipInReceive_hi + m.ipInReceive_lo; + stats->iw.ipInHdrErrors = m.ipInHdrErrors_hi + m.ipInHdrErrors_lo; + stats->iw.ipInAddrErrors = m.ipInAddrErrors_hi + m.ipInAddrErrors_lo; + stats->iw.ipInUnknownProtos = m.ipInUnknownProtos_hi + + m.ipInUnknownProtos_lo; + stats->iw.ipInDiscards = m.ipInDiscards_hi + m.ipInDiscards_lo; + stats->iw.ipInDelivers = m.ipInDelivers_hi + m.ipInDelivers_lo; + stats->iw.ipOutRequests = m.ipOutRequests_hi + m.ipOutRequests_lo; + stats->iw.ipOutDiscards = m.ipOutDiscards_hi + m.ipOutDiscards_lo; + stats->iw.ipOutNoRoutes = m.ipOutNoRoutes_hi + m.ipOutNoRoutes_lo; + stats->iw.ipReasmTimeout = m.ipReasmTimeout; + stats->iw.ipReasmReqds = m.ipReasmReqds; + stats->iw.ipReasmOKs = m.ipReasmOKs; + stats->iw.ipReasmFails = m.ipReasmFails; + stats->iw.tcpActiveOpens = m.tcpActiveOpens; + stats->iw.tcpPassiveOpens = m.tcpPassiveOpens; + stats->iw.tcpAttemptFails = m.tcpAttemptFails; + stats->iw.tcpEstabResets = m.tcpEstabResets; + stats->iw.tcpOutRsts = m.tcpOutRsts; + stats->iw.tcpCurrEstab = m.tcpCurrEstab; + stats->iw.tcpInSegs = m.tcpInSegs_hi + m.tcpInSegs_lo; + stats->iw.tcpOutSegs = m.tcpOutSegs_hi + m.tcpOutSegs_lo; + stats->iw.tcpRetransSeg = m.tcpRetransSeg_hi + m.tcpRetransSeg_lo; + stats->iw.tcpInErrs = m.tcpInErrs_hi + m.tcpInErrs_lo; + stats->iw.tcpRtoMin = m.tcpRtoMin; + stats->iw.tcpRtoMax = m.tcpRtoMax; + return 0; +} + static DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL); static DEVICE_ATTR(fw_ver, S_IRUGO, show_fw_ver, NULL); static DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL); @@ -1219,7 +1263,7 @@ static struct device_attribute *iwch_class_attributes[] = { &dev_attr_hw_rev, &dev_attr_fw_ver, &dev_attr_hca_type, - &dev_attr_board_id + &dev_attr_board_id, }; int iwch_register_device(struct iwch_dev *dev) @@ -1290,15 +1334,13 @@ int iwch_register_device(struct iwch_dev *dev) dev->ibdev.alloc_fast_reg_mr = iwch_alloc_fast_reg_mr; dev->ibdev.alloc_fast_reg_page_list = iwch_alloc_fastreg_pbl; dev->ibdev.free_fast_reg_page_list = iwch_free_fastreg_pbl; - dev->ibdev.attach_mcast = iwch_multicast_attach; dev->ibdev.detach_mcast = iwch_multicast_detach; dev->ibdev.process_mad = iwch_process_mad; - dev->ibdev.req_notify_cq = iwch_arm_cq; dev->ibdev.post_send = iwch_post_send; dev->ibdev.post_recv = iwch_post_receive; - + dev->ibdev.get_protocol_stats = iwch_get_mib; dev->ibdev.iwcm = kmalloc(sizeof(struct iw_cm_verbs), GFP_KERNEL); if (!dev->ibdev.iwcm) diff --git a/drivers/net/cxgb3/cxgb3_ctl_defs.h b/drivers/net/cxgb3/cxgb3_ctl_defs.h index 6c4f320..ed0ecd9 100644 --- a/drivers/net/cxgb3/cxgb3_ctl_defs.h +++ b/drivers/net/cxgb3/cxgb3_ctl_defs.h @@ -54,6 +54,7 @@ enum { RDMA_CQ_DISABLE = 16, RDMA_CTRL_QP_SETUP = 17, RDMA_GET_MEM = 18, + RDMA_GET_MIB = 19, GET_RX_PAGE_INFO = 50, }; diff --git a/drivers/net/cxgb3/cxgb3_offload.c b/drivers/net/cxgb3/cxgb3_offload.c index ff9c013..cf26968 100644 --- a/drivers/net/cxgb3/cxgb3_offload.c +++ b/drivers/net/cxgb3/cxgb3_offload.c @@ -303,6 +303,12 @@ static int cxgb_rdma_ctl(struct adapter *adapter, unsigned int req, void *data) spin_unlock_irq(&adapter->sge.reg_lock); break; } + case RDMA_GET_MIB: { + spin_lock(&adapter->stats_lock); + t3_tp_get_mib_stats(adapter, (struct tp_mib_stats *)data); + spin_unlock(&adapter->stats_lock); + break; + } default: ret = -EOPNOTSUPP; } @@ -381,6 +387,7 @@ static int cxgb_offload_ctl(struct t3cdev *tdev, unsigned int req, void *data) case RDMA_CQ_DISABLE: case RDMA_CTRL_QP_SETUP: case RDMA_GET_MEM: + case RDMA_GET_MIB: if (!offload_running(adapter)) return -EAGAIN; return cxgb_rdma_ctl(adapter, req, data); From underpaysr6 at loechgauer-bank.de Tue Jun 17 15:00:26 2008 From: underpaysr6 at loechgauer-bank.de (Jarred Villalobos) Date: Wed, 18 Jun 2008 00:00:26 +0200 Subject: [ofa-general] Earn more money. fpuour qurype ucbjfs Message-ID: <01c8d0d6$4fe5fb30$0ffc8b3e@underpaysr6> De ilg gre gb e - ad rl van zpp cem wnh ent Ac bm ade mwv mic Qu pbd alifi yk cati sr ons available from prestigious Non-Ac mf cre qs dit glb ed Un ooj ive glf rsi oaz ties Do you have the kn pp owl lc edge and the experience but lack the qua jj lifica twh tio zhk ns? Are you getting turned down time and time again for the job of your dreams because you just don't have the right letters after your name? Get the pre nl sti rb ge that you deserve today! Move ahead in your ca ql re vjk er today! Ba kee che hun lor xeo s, M py ast fhc ers and D tt oct kie ora pf te D fce egr gk ees available in y our field! No exa yv minat tta ions! No cl sxg ass kgd es! No te dm xtbo aq oks! Call to register and receive your qu za alifi qq catio qc ns within days! Leave a Mes ylx sage 24 ho dxt urs a day 7 da jbs ys a we of ek! 1 tt -5 tvc 2 rd 0-2 wez 0 ac 3-0 ll 21 pp 3 Co fw nfid fg enti ai ali epg ty as mbf su vsb red! -------------- next part -------------- An HTML attachment was scrubbed... URL: From 55hassan.shirjang at avanegar.com Tue Jun 17 19:49:50 2008 From: 55hassan.shirjang at avanegar.com (carsten kathi) Date: Wed, 18 Jun 2008 02:49:50 +0000 Subject: [ofa-general] Best of Bally, Boss, Versace Message-ID: <000501c8d0fc$05312182$1cec8ea3@qauke> The worldwide great de luxe store for shoes and bags is just one click away. Recommended by hundred thousands of grateful customers worldwide, we carry many of populars brands including: Prada Adidas Burberry Dsquared Louis Vuitton Here you willc find millions of best designs for shoes, and leather products, at at cheapest priceE. Sale close this week, so visit us now and start pampering yourself and your loved second half! - Visit our site: www.milajeat[DOT]com (copy this link then replace "[DOT]" to ".") From keshetti85-student at yahoo.co.in Wed Jun 18 01:23:28 2008 From: keshetti85-student at yahoo.co.in (Keshetti Mahesh) Date: Wed, 18 Jun 2008 13:53:28 +0530 Subject: [ofa-general] [OpenSM] How to find active routing algorithm ? Message-ID: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> Hi all, What is the best way to qeury which routing algorithm of OpenSM is active at the moment ? AFAIK, the routing algorithm with which the OpenSM is started may not be the active algorithm after some time. So looking at the 'opensm' command line may not help. And OpenSM log file contains lot of details. Searching through log file for finding the routing algorithm doesn't look like a good way to accomplish it. -Mahesh From kliteyn at dev.mellanox.co.il Wed Jun 18 01:43:04 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 18 Jun 2008 11:43:04 +0300 Subject: [ofa-general] [OpenSM] How to find active routing algorithm ? In-Reply-To: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> Message-ID: <4858CA98.3010604@dev.mellanox.co.il> Hi Mahesh, Keshetti Mahesh wrote: > Hi all, > > What is the best way to qeury which routing algorithm of OpenSM > is active at the moment ? > > AFAIK, the routing algorithm with which the OpenSM is started may not be > the active algorithm after some time. So looking at the 'opensm' command line > may not help. > > And OpenSM log file contains lot of details. Searching through log > file for finding the routing algorithm doesn't look like a good way to > accomplish it. grep "tables configured on all switches" /var/log/opensm.log You will see the routing name in the full message. -- Yevgeny > -Mahesh > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From vlad at lists.openfabrics.org Wed Jun 18 02:25:35 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 18 Jun 2008 02:25:35 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080618-0200 daily build status Message-ID: <20080618092537.51F76E285D1@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.21.1 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080618-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From jackm at dev.mellanox.co.il Wed Jun 18 02:32:35 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Wed, 18 Jun 2008 12:32:35 +0300 Subject: [ofa-general] [PATCH v2] mlx4: make firmware diagnostic counters available via sysfs In-Reply-To: References: <200804021615.44982.jackm@dev.mellanox.co.il> Message-ID: <200806181232.35634.jackm@dev.mellanox.co.il> mlx4: make firmware diagnostic counters available via sysfs. version 2: 1. Extensive cleanup based upon Roland's comments. 2. Eliminated use of class_device. 3. Eliminated intermediate macros and functions for diag clear. Developed by: Gabi Liron of Mellanox. Signed-off-by: Jack Morgenstein --- Roland, I cleaned up the patch along the lines you indicated. I also eliminated all use of class_device. I left in the "in_mod" parameter (renamed "op_mod") in macro DEVICE_DIAG_RPRT_ATTR, in case we later implement further diagnostic counter groups. I did leave in the array arguments of mlx4_query_diag_counters(), so that the API is ready if we decide to implement query functions which retrieve an entire vector of counters at once (the FW command gets all the counters of a given diagnostic group at once). Finally, I eliminated the CLASS_DEVICE_DIAG_CLR_RPRT_ATTR macro, because it does not add anything at this point -- I just declare "clear_diag" directly, and define its procedure directly (the "clear" resets an entire counter group). If we add more counter groups, I will add more "clear" functions. Index: ofa_1_4_dev_kernel/drivers/net/mlx4/fw.c =================================================================== --- ofa_1_4_dev_kernel.orig/drivers/net/mlx4/fw.c 2008-06-18 10:31:50.000000000 +0300 +++ ofa_1_4_dev_kernel/drivers/net/mlx4/fw.c 2008-06-18 12:00:34.000000000 +0300 @@ -860,3 +860,37 @@ int mlx4_NOP(struct mlx4_dev *dev) /* Input modifier of 0x1f means "finish as soon as possible." */ return mlx4_cmd(dev, 0, 0x1f, 0, MLX4_CMD_NOP, 100); } + +int mlx4_query_diag_counters(struct mlx4_dev *dev, int array_length, + u8 op_modifier, u32 in_offset[], u32 counter_out[]) +{ + struct mlx4_cmd_mailbox *mailbox; + u32 *outbox; + int ret; + int i; + + mailbox = mlx4_alloc_cmd_mailbox(dev); + if (IS_ERR(mailbox)) + return PTR_ERR(mailbox); + outbox = mailbox->buf; + + ret = mlx4_cmd_box(dev, 0, mailbox->dma, 0, op_modifier, + MLX4_CMD_DIAG_RPRT, MLX4_CMD_TIME_CLASS_A); + if (ret) + goto out; + + for (i=0; i < array_length; i++) { + if (in_offset[i] > MLX4_MAILBOX_SIZE) { + ret = -EINVAL; + goto out; + } + + MLX4_GET(counter_out[i], outbox, in_offset[i]); + } + +out: + mlx4_free_cmd_mailbox(dev, mailbox); + return ret; +} +EXPORT_SYMBOL_GPL(mlx4_query_diag_counters); + Index: ofa_1_4_dev_kernel/include/linux/mlx4/device.h =================================================================== --- ofa_1_4_dev_kernel.orig/include/linux/mlx4/device.h 2008-06-18 10:31:50.000000000 +0300 +++ ofa_1_4_dev_kernel/include/linux/mlx4/device.h 2008-06-18 10:46:02.000000000 +0300 @@ -417,5 +417,7 @@ void mlx4_fmr_unmap(struct mlx4_dev *dev u32 *lkey, u32 *rkey); int mlx4_fmr_free(struct mlx4_dev *dev, struct mlx4_fmr *fmr); int mlx4_SYNC_TPT(struct mlx4_dev *dev); +int mlx4_query_diag_counters(struct mlx4_dev *mlx4_dev, int array_length, + u8 op_modifier, u32 in_offset[], u32 counter_out[]); #endif /* MLX4_DEVICE_H */ Index: ofa_1_4_dev_kernel/drivers/infiniband/hw/mlx4/main.c =================================================================== --- ofa_1_4_dev_kernel.orig/drivers/infiniband/hw/mlx4/main.c 2008-06-18 10:31:50.000000000 +0300 +++ ofa_1_4_dev_kernel/drivers/infiniband/hw/mlx4/main.c 2008-06-18 12:08:23.000000000 +0300 @@ -614,6 +614,136 @@ static struct device_attribute *mlx4_cla &dev_attr_board_id }; +/* + * create show function and a device_attribute struct pointing to + * the function for _name + */ +#define DEVICE_DIAG_RPRT_ATTR(_name, _offset, _op_mod) \ +static ssize_t show_rprt_##_name(struct device *dev, \ + struct device_attribute *attr, \ + char *buf){ \ + return show_diag_rprt(dev, buf, _offset, _op_mod); \ +} \ +static DEVICE_ATTR(_name, S_IRUGO, show_rprt_##_name, NULL); + +#define MLX4_DIAG_RPRT_CLEAR_DIAGS 3 + +static size_t show_diag_rprt(struct device *device, char *buf, + u32 offset, u8 op_modifier) +{ + size_t ret; + u32 counter_offset = offset; + u32 diag_counter = 0; + struct mlx4_ib_dev *dev = container_of(device, struct mlx4_ib_dev, + ib_dev.dev); + + ret = mlx4_query_diag_counters(dev->dev, 1, op_modifier, + &counter_offset, &diag_counter); + if (ret) + return ret; + + return sprintf(buf,"%d\n", diag_counter); +} + +static ssize_t clear_diag_counters(struct device *device, + struct device_attribute *attr, + const char *buf, size_t length) +{ + size_t ret; + struct mlx4_ib_dev *dev = container_of(device, struct mlx4_ib_dev, + ib_dev.dev); + + ret = mlx4_query_diag_counters(dev->dev, 0, MLX4_DIAG_RPRT_CLEAR_DIAGS, + NULL, NULL); + if (ret) + return ret; + + return length; +} + +DEVICE_DIAG_RPRT_ATTR(rq_num_lle , 0x00, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_lle , 0x04, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_lqpoe , 0x08, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_lqpoe , 0x0C, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_leeoe , 0x10, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_leeoe , 0x14, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_lpe , 0x18, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_lpe , 0x1C, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_wrfe , 0x20, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_wrfe , 0x24, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_mwbe , 0x2C, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_bre , 0x34, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_lae , 0x38, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_rire , 0x44, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_rire , 0x48, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_rae , 0x4C, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_rae , 0x50, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_roe , 0x54, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_tree , 0x5C, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_rree , 0x64, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_rnr , 0x68, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_rnr , 0x6C, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_rabrte , 0x7C, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_ieecne , 0x84, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_ieecse , 0x8C, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_oos , 0x100, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_oos , 0x104, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_mce , 0x108, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_rsync , 0x110, 2); +DEVICE_DIAG_RPRT_ATTR(sq_num_rsync , 0x114, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_udsdprd , 0x118, 2); +DEVICE_DIAG_RPRT_ATTR(rq_num_ucsdprd , 0x120, 2); +DEVICE_DIAG_RPRT_ATTR(num_cqovf , 0x1A0, 2); +DEVICE_DIAG_RPRT_ATTR(num_eqovf , 0x1A4, 2); +DEVICE_DIAG_RPRT_ATTR(num_baddb , 0x1A8, 2); + +static DEVICE_ATTR(clear_diag, S_IWUGO, NULL, clear_diag_counters); + +static struct attribute *diag_rprt_attrs[] = { + &dev_attr_rq_num_lle.attr, + &dev_attr_sq_num_lle.attr, + &dev_attr_rq_num_lqpoe.attr, + &dev_attr_sq_num_lqpoe.attr, + &dev_attr_rq_num_leeoe.attr, + &dev_attr_sq_num_leeoe.attr, + &dev_attr_rq_num_lpe.attr, + &dev_attr_sq_num_lpe.attr, + &dev_attr_rq_num_wrfe.attr, + &dev_attr_sq_num_wrfe.attr, + &dev_attr_sq_num_mwbe.attr, + &dev_attr_sq_num_bre.attr, + &dev_attr_rq_num_lae.attr, + &dev_attr_sq_num_rire.attr, + &dev_attr_rq_num_rire.attr, + &dev_attr_sq_num_rae.attr, + &dev_attr_rq_num_rae.attr, + &dev_attr_sq_num_roe.attr, + &dev_attr_sq_num_tree.attr, + &dev_attr_sq_num_rree.attr, + &dev_attr_rq_num_rnr.attr, + &dev_attr_sq_num_rnr.attr, + &dev_attr_sq_num_rabrte.attr, + &dev_attr_sq_num_ieecne.attr, + &dev_attr_sq_num_ieecse.attr, + &dev_attr_rq_num_oos.attr, + &dev_attr_sq_num_oos.attr, + &dev_attr_rq_num_mce.attr, + &dev_attr_rq_num_rsync.attr, + &dev_attr_sq_num_rsync.attr, + &dev_attr_rq_num_udsdprd.attr, + &dev_attr_rq_num_ucsdprd.attr, + &dev_attr_num_cqovf.attr, + &dev_attr_num_eqovf.attr, + &dev_attr_num_baddb.attr, + &dev_attr_clear_diag.attr, + NULL +}; + +static struct attribute_group diag_counters_group = { + .name = "diag_counters", + .attrs = diag_rprt_attrs +}; + static void *mlx4_ib_add(struct mlx4_dev *dev) { static int mlx4_ib_version_printed; @@ -757,6 +887,9 @@ static void *mlx4_ib_add(struct mlx4_dev goto err_reg; } + if(sysfs_create_group(&ibdev->ib_dev.dev.kobj, &diag_counters_group)) + goto err_reg; + return ibdev; err_reg: @@ -782,6 +915,8 @@ static void mlx4_ib_remove(struct mlx4_d struct mlx4_ib_dev *ibdev = ibdev_ptr; int p; + sysfs_remove_group(&ibdev->ib_dev.dev.kobj, &diag_counters_group); + for (p = 1; p <= dev->caps.num_ports; ++p) mlx4_CLOSE_PORT(dev, p); From vlad at dev.mellanox.co.il Wed Jun 18 02:27:35 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Wed, 18 Jun 2008 12:27:35 +0300 Subject: [ofa-general] Several RDS patches In-Reply-To: <200806172259.05882.okir@lst.de> References: <200806171722.02780.olaf.kirch@oracle.com> <200806172259.05882.okir@lst.de> Message-ID: <4858D507.5080503@dev.mellanox.co.il> Olaf Kirch wrote: > On Tuesday 17 June 2008 17:22:02 Olaf Kirch wrote: >> Hi Vlad, >> >> during testing of RDS, a number of issues came up that I had to >> address. Some of them were bugs, others were usability issues. >> >> I submitted the following patches to >> git://git.openfabrics.org/~okir/ofed_1_3/linux-2.6.git code-drop-20080617 > > I just added another patch to that branch; please pull this one as well. > > Thanks! Olaf Done, http://www.openfabrics.org/downloads/OFED/ofed-1.3.2-daily/OFED-1.3.2-20080618-0225.tgz Regards, Vladimir From olaf.kirch at oracle.com Wed Jun 18 02:41:40 2008 From: olaf.kirch at oracle.com (Olaf Kirch) Date: Wed, 18 Jun 2008 11:41:40 +0200 Subject: [ofa-general] Several RDS patches In-Reply-To: <4858D507.5080503@dev.mellanox.co.il> References: <200806171722.02780.olaf.kirch@oracle.com> <200806172259.05882.okir@lst.de> <4858D507.5080503@dev.mellanox.co.il> Message-ID: <200806181141.41888.olaf.kirch@oracle.com> On Wednesday 18 June 2008 11:27:35 Vladimir Sokolovsky wrote: > Done, > http://www.openfabrics.org/downloads/OFED/ofed-1.3.2-daily/OFED-1.3.2-20080618-0225.tgz Thanks a lot Vlad, very much appreciated! Olaf -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play okir at lst.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax From tziporet at dev.mellanox.co.il Wed Jun 18 02:48:40 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Wed, 18 Jun 2008 12:48:40 +0300 Subject: [ofa-general] Several RDS patches In-Reply-To: <4858D507.5080503@dev.mellanox.co.il> References: <200806171722.02780.olaf.kirch@oracle.com> <200806172259.05882.okir@lst.de> <4858D507.5080503@dev.mellanox.co.il> Message-ID: <4858D9F8.4030300@mellanox.co.il> Vladimir Sokolovsky wrote: > Olaf Kirch wrote: >> On Tuesday 17 June 2008 17:22:02 Olaf Kirch wrote: >>> Hi Vlad, >>> >>> during testing of RDS, a number of issues came up that I had to >>> address. Some of them were bugs, others were usability issues. >>> >>> I submitted the following patches to >>> git://git.openfabrics.org/~okir/ofed_1_3/linux-2.6.git >>> code-drop-20080617 >> >> I just added another patch to that branch; please pull this one as well. >> >> Thanks! Olaf > > Done, > http://www.openfabrics.org/downloads/OFED/ofed-1.3.2-daily/OFED-1.3.2-20080618-0225.tgz > > I guess we need this for 1.4 tree too Tziporet From okir at lst.de Wed Jun 18 02:56:04 2008 From: okir at lst.de (Olaf Kirch) Date: Wed, 18 Jun 2008 11:56:04 +0200 Subject: [ofa-general] Several RDS patches In-Reply-To: <4858D9F8.4030300@mellanox.co.il> References: <200806171722.02780.olaf.kirch@oracle.com> <4858D507.5080503@dev.mellanox.co.il> <4858D9F8.4030300@mellanox.co.il> Message-ID: <200806181156.05160.okir@lst.de> On Wednesday 18 June 2008 11:48:40 Tziporet Koren wrote: > I guess we need this for 1.4 tree too Yes. I'm currently re-working my patch set for 1.4 and will submit everything shortly. Olaf -- Olaf Kirch | --- o --- Nous sommes du soleil we love when we play okir at lst.de | / | \ sol.dhoop.naytheet.ah kin.ir.samse.qurax From keshetti.mahesh at gmail.com Wed Jun 18 03:37:03 2008 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Wed, 18 Jun 2008 16:07:03 +0530 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> Message-ID: <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> >grep "tables configured on all switches" /var/log/opensm.log >You will see the routing name in the full message. I have mentioned this in my query also. But what if someone (working on some node[X] which is part of same subnet) who doesn't have access to the node/switch where OpenSM is running want to know the same ? It will be lot helpful if some one can make changes to 'sminfo' utility so that active routing algorithm information is also reported along with the existing output. -Mahesh From ogerlitz at voltaire.com Wed Jun 18 03:54:12 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Jun 2008 13:54:12 +0300 (IDT) Subject: [ofa-general] [PATCH 0/4 for-2.6.27] rdma: add ADDR_CHANGE rdma-cm notifications Message-ID: Roland, The four patches that follow implement the RDMA stack part of the ability to notify ulps which use the rdma-cm on networking changes that make them "non aligned" with the network stack wrt what links (eg ) are being used for communication. They passed review on the list. The patch sequence is dependent on the below three commits to the davem-next branch at Jeff Garzik netdev-2.6 tree, so my thought was that you can pull/merge this tree/branch into your for-2.6.27 branch, such that it compiles after accepting the rdma patches. Or. commit 01f3109de49a889db8adf9116449727547ee497e Author: Or Gerlitz Date: Fri Jun 13 18:12:02 2008 -0700 bonding: deliver netdev event for fail-over under the active-backup mode commit 709f8a45e8521f2f4229e5fdf3ded1fb77e2ca4e Author: Or Gerlitz Date: Fri Jun 13 18:12:01 2008 -0700 bonding: bond_change_active_slave() cleanup under active-backup commit c1da4ac752b8b0411791d26c678fcf23d2eed242 Author: Or Gerlitz Date: Fri Jun 13 18:12:00 2008 -0700 net/core: add NETDEV_BONDING_FAILOVER event From ogerlitz at voltaire.com Wed Jun 18 03:54:40 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Jun 2008 13:54:40 +0300 (IDT) Subject: [ofa-general] [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: References: Message-ID: Keep also the local (src) netdevice name in struct rdma_dev_addr. In a High-Availability scheme this can be used by the rdma-cm to align RDMA sessions to use the same links as the IP stack does under fail-over and route change cases. Signed-off-by: Or Gerlitz drivers/infiniband/core/addr.c | 1 + drivers/infiniband/core/cma.c | 8 +++++--- include/rdma/ib_addr.h | 1 + 3 files changed, 7 insertions(+), 3 deletions(-) Index: infiniband/drivers/infiniband/core/addr.c =================================================================== --- infiniband.orig/drivers/infiniband/core/addr.c +++ infiniband/drivers/infiniband/core/addr.c @@ -105,6 +105,7 @@ int rdma_copy_addr(struct rdma_dev_addr memcpy(dev_addr->broadcast, dev->broadcast, MAX_ADDR_LEN); if (dst_dev_addr) memcpy(dev_addr->dst_dev_addr, dst_dev_addr, MAX_ADDR_LEN); + memcpy(dev_addr->src_dev_name, dev->name, IFNAMSIZ); return 0; } EXPORT_SYMBOL(rdma_copy_addr); Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -1002,6 +1002,7 @@ static struct rdma_id_private *cma_new_c union cma_ip_addr *src, *dst; __be16 port; u8 ip_ver; + int ret; if (cma_get_net_info(ib_event->private_data, listen_id->ps, &ip_ver, &port, &src, &dst)) @@ -1026,10 +1027,11 @@ static struct rdma_id_private *cma_new_c if (rt->num_paths == 2) rt->path_rec[1] = *ib_event->param.req_rcvd.alternate_path; - ib_addr_set_sgid(&rt->addr.dev_addr, &rt->path_rec[0].sgid); ib_addr_set_dgid(&rt->addr.dev_addr, &rt->path_rec[0].dgid); - ib_addr_set_pkey(&rt->addr.dev_addr, be16_to_cpu(rt->path_rec[0].pkey)); - rt->addr.dev_addr.dev_type = RDMA_NODE_IB_CA; + ret = rdma_translate_ip(&id->route.addr.src_addr, + &id->route.addr.dev_addr); + if (ret) + goto destroy_id; id_priv = container_of(id, struct rdma_id_private, id); id_priv->state = CMA_CONNECT; Index: infiniband/include/rdma/ib_addr.h =================================================================== --- infiniband.orig/include/rdma/ib_addr.h +++ infiniband/include/rdma/ib_addr.h @@ -61,6 +61,7 @@ struct rdma_dev_addr { unsigned char dst_dev_addr[MAX_ADDR_LEN]; unsigned char broadcast[MAX_ADDR_LEN]; enum rdma_node_type dev_type; + char src_dev_name[IFNAMSIZ]; }; /** From ogerlitz at voltaire.com Wed Jun 18 03:55:10 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Jun 2008 13:55:10 +0300 (IDT) Subject: [ofa-general] [PATCH 2/4] rdma/cma: simplify locking needed for serialization of callbacks In-Reply-To: References: Message-ID: The rdma-cm has some logic in place to make sure that callbacks on an ID are delivered to the consumer in serialized manner, specifically it has code to protect against a device removal racing with a callback now being delivered to the user. This patch simplifies this logic by using a mutex per ID instead of the wait queue and atomic variable, such that cma_disable_remove is now called cma_disable_callback and cma_enable_remove is now just mutex_unlock. Signed-off-by: Or Gerlitz drivers/infiniband/core/cma.c | 106 +++++++++++++++++++----------------------- 1 files changed, 50 insertions(+), 56 deletions(-) Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -130,8 +130,7 @@ struct rdma_id_private { struct completion comp; atomic_t refcount; - wait_queue_head_t wait_remove; - atomic_t dev_remove; + struct mutex handler_mutex; int backlog; int timeout_ms; @@ -355,26 +354,15 @@ static void cma_deref_id(struct rdma_id_ complete(&id_priv->comp); } -static int cma_disable_remove(struct rdma_id_private *id_priv, +static int cma_disable_callback(struct rdma_id_private *id_priv, enum cma_state state) { - unsigned long flags; - int ret; - - spin_lock_irqsave(&id_priv->lock, flags); - if (id_priv->state == state) { - atomic_inc(&id_priv->dev_remove); - ret = 0; - } else - ret = -EINVAL; - spin_unlock_irqrestore(&id_priv->lock, flags); - return ret; -} - -static void cma_enable_remove(struct rdma_id_private *id_priv) -{ - if (atomic_dec_and_test(&id_priv->dev_remove)) - wake_up(&id_priv->wait_remove); + mutex_lock(&id_priv->handler_mutex); + if (id_priv->state != state) { + mutex_unlock(&id_priv->handler_mutex); + return -EINVAL; + } + return 0; } static int cma_has_cm_dev(struct rdma_id_private *id_priv) @@ -399,8 +387,7 @@ struct rdma_cm_id *rdma_create_id(rdma_c mutex_init(&id_priv->qp_mutex); init_completion(&id_priv->comp); atomic_set(&id_priv->refcount, 1); - init_waitqueue_head(&id_priv->wait_remove); - atomic_set(&id_priv->dev_remove, 0); + mutex_init(&id_priv->handler_mutex); INIT_LIST_HEAD(&id_priv->listen_list); INIT_LIST_HEAD(&id_priv->mc_list); get_random_bytes(&id_priv->seq_num, sizeof id_priv->seq_num); @@ -927,7 +914,7 @@ static int cma_ib_handler(struct ib_cm_i struct rdma_cm_event event; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -984,12 +971,12 @@ static int cma_ib_handler(struct ib_cm_i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.ib = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -1101,7 +1088,7 @@ static int cma_req_handler(struct ib_cm_ int offset, ret; listen_id = cm_id->context; - if (cma_disable_remove(listen_id, CMA_LISTEN)) + if (cma_disable_callback(listen_id, CMA_LISTEN)) return -ECONNABORTED; memset(&event, 0, sizeof event); @@ -1122,7 +1109,7 @@ static int cma_req_handler(struct ib_cm_ goto out; } - atomic_inc(&conn_id->dev_remove); + mutex_lock(&conn_id->handler_mutex); mutex_lock(&lock); ret = cma_acquire_dev(conn_id); mutex_unlock(&lock); @@ -1144,7 +1131,7 @@ static int cma_req_handler(struct ib_cm_ !cma_is_ud_ps(conn_id->id.ps)) ib_send_cm_mra(cm_id, CMA_CM_MRA_SETTING, NULL, 0); mutex_unlock(&lock); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); goto out; } @@ -1153,11 +1140,11 @@ static int cma_req_handler(struct ib_cm_ release_conn_id: cma_exch(conn_id, CMA_DESTROYING); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(&conn_id->id); out: - cma_enable_remove(listen_id); + mutex_unlock(&listen_id->handler_mutex); return ret; } @@ -1223,7 +1210,7 @@ static int cma_iw_handler(struct iw_cm_i struct sockaddr_in *sin; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -1267,12 +1254,12 @@ static int cma_iw_handler(struct iw_cm_i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.iw = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -1288,7 +1275,7 @@ static int iw_conn_req_handler(struct iw struct ib_device_attr attr; listen_id = cm_id->context; - if (cma_disable_remove(listen_id, CMA_LISTEN)) + if (cma_disable_callback(listen_id, CMA_LISTEN)) return -ECONNABORTED; /* Create a new RDMA id for the new IW CM ID */ @@ -1300,19 +1287,19 @@ static int iw_conn_req_handler(struct iw goto out; } conn_id = container_of(new_cm_id, struct rdma_id_private, id); - atomic_inc(&conn_id->dev_remove); + mutex_lock(&conn_id->handler_mutex); conn_id->state = CMA_CONNECT; dev = ip_dev_find(&init_net, iw_event->local_addr.sin_addr.s_addr); if (!dev) { ret = -EADDRNOTAVAIL; - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } ret = rdma_copy_addr(&conn_id->id.route.addr.dev_addr, dev, NULL); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1321,7 +1308,7 @@ static int iw_conn_req_handler(struct iw ret = cma_acquire_dev(conn_id); mutex_unlock(&lock); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1337,7 +1324,7 @@ static int iw_conn_req_handler(struct iw ret = ib_query_device(conn_id->id.device, &attr); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1353,14 +1340,14 @@ static int iw_conn_req_handler(struct iw /* User wants to destroy the CM ID */ conn_id->cm_id.iw = NULL; cma_exch(conn_id, CMA_DESTROYING); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(&conn_id->id); } out: if (dev) dev_put(dev); - cma_enable_remove(listen_id); + mutex_unlock(&listen_id->handler_mutex); return ret; } @@ -1592,7 +1579,7 @@ static void cma_work_handler(struct work struct rdma_id_private *id_priv = work->id; int destroy = 0; - atomic_inc(&id_priv->dev_remove); + mutex_lock(&id_priv->handler_mutex); if (!cma_comp_exch(id_priv, work->old_state, work->new_state)) goto out; @@ -1601,7 +1588,7 @@ static void cma_work_handler(struct work destroy = 1; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); if (destroy) rdma_destroy_id(&id_priv->id); @@ -1764,7 +1751,7 @@ static void addr_handler(int status, str struct rdma_cm_event event; memset(&event, 0, sizeof event); - atomic_inc(&id_priv->dev_remove); + mutex_lock(&id_priv->handler_mutex); /* * Grab mutex to block rdma_destroy_id() from removing the device while @@ -1793,13 +1780,13 @@ static void addr_handler(int status, str if (id_priv->id.event_handler(&id_priv->id, &event)) { cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); rdma_destroy_id(&id_priv->id); return; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); } @@ -2126,7 +2113,7 @@ static int cma_sidr_rep_handler(struct i struct ib_cm_sidr_rep_event_param *rep = &ib_event->param.sidr_rep_rcvd; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -2167,12 +2154,12 @@ static int cma_sidr_rep_handler(struct i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.ib = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -2570,9 +2557,12 @@ static int cma_ib_mc_handler(int status, int ret; id_priv = mc->id_priv; - if (cma_disable_remove(id_priv, CMA_ADDR_BOUND) && - cma_disable_remove(id_priv, CMA_ADDR_RESOLVED)) + mutex_lock(&id_priv->handler_mutex); + if ((id_priv->state != CMA_ADDR_BOUND) && + (id_priv->state != CMA_ADDR_RESOLVED)) { + mutex_unlock(&id_priv->handler_mutex); return 0; + } mutex_lock(&id_priv->qp_mutex); if (!status && id_priv->id.qp) @@ -2596,12 +2586,12 @@ static int cma_ib_mc_handler(int status, ret = id_priv->id.event_handler(&id_priv->id, &event); if (ret) { cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return 0; } - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return 0; } @@ -2760,6 +2750,7 @@ static int cma_remove_id_dev(struct rdma { struct rdma_cm_event event; enum cma_state state; + int ret = 0; /* Record that we want to remove the device */ state = cma_exch(id_priv, CMA_DEVICE_REMOVAL); @@ -2767,15 +2758,18 @@ static int cma_remove_id_dev(struct rdma return 0; cma_cancel_operation(id_priv, state); - wait_event(id_priv->wait_remove, !atomic_read(&id_priv->dev_remove)); + mutex_lock(&id_priv->handler_mutex); /* Check for destruction from another callback. */ if (!cma_comp(id_priv, CMA_DEVICE_REMOVAL)) - return 0; + goto out; memset(&event, 0, sizeof event); event.event = RDMA_CM_EVENT_DEVICE_REMOVAL; - return id_priv->id.event_handler(&id_priv->id, &event); + ret = id_priv->id.event_handler(&id_priv->id, &event); +out: + mutex_unlock(&id_priv->handler_mutex); + return ret; } static void cma_process_remove(struct cma_device *cma_dev) From ogerlitz at voltaire.com Wed Jun 18 03:55:38 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Jun 2008 13:55:38 +0300 (IDT) Subject: [ofa-general] [PATCH 3/4] rdma/cma: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: RDMA_CM_EVENT_ADDR_CHANGE event can be used by rdma-cm consumers that wish to have their RDMA sessions always use the same links (eg ) as the IP stack does. In the current code, this does not happen when bonding is used and fail-over happened, but the IB link used by an already existing session is operating fine. Use netevent notification for sensing that a change has happened in the IP stack, then scan the rdma-cm IDs list to see if there is an ID that is "misaligned" in that respect with the IP stack, and deliver RDMA_CM_EVENT_ADDR_CHANGE for this ID. The user can act on the event or just ignore it Signed-off-by: Or Gerlitz drivers/infiniband/core/cma.c | 90 ++++++++++++++++++++++++++++++++++++++++++ include/rdma/rdma_cm.h | 3 - 2 files changed, 92 insertions(+), 1 deletion(-) Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -168,6 +168,12 @@ struct cma_work { struct rdma_cm_event event; }; +struct cma_ndev_work { + struct work_struct work; + struct rdma_id_private *id; + struct rdma_cm_event event; +}; + union cma_ip_addr { struct in6_addr ip6; struct { @@ -1595,6 +1601,29 @@ out: kfree(work); } +static void cma_ndev_work_handler(struct work_struct *_work) +{ + struct cma_ndev_work *work = container_of(_work, struct cma_ndev_work, work); + struct rdma_id_private *id_priv = work->id; + int destroy = 0; + + mutex_lock(&id_priv->handler_mutex); + if (id_priv->state == CMA_DESTROYING || + id_priv->state == CMA_DEVICE_REMOVAL) + goto out; + + if (id_priv->id.event_handler(&id_priv->id, &work->event)) { + cma_exch(id_priv, CMA_DESTROYING); + destroy = 1; + } +out: + mutex_unlock(&id_priv->handler_mutex); + cma_deref_id(id_priv); + if (destroy) + rdma_destroy_id(&id_priv->id); + kfree(work); +} + static int cma_resolve_ib_route(struct rdma_id_private *id_priv, int timeout_ms) { struct rdma_route *route = &id_priv->id.route; @@ -2723,6 +2752,64 @@ void rdma_leave_multicast(struct rdma_cm } EXPORT_SYMBOL(rdma_leave_multicast); +static int cma_netdev_change(struct net_device *ndev, struct rdma_id_private *id_priv) +{ + struct rdma_dev_addr *dev_addr; + struct cma_ndev_work *work; + + dev_addr = &id_priv->id.route.addr.dev_addr; + + if (!memcmp(dev_addr->src_dev_name, ndev->name, IFNAMSIZ) && + memcmp(dev_addr->src_dev_addr, ndev->dev_addr, ndev->addr_len)) { + printk(KERN_INFO "RDMA CM addr change for ndev %s used by id %p\n", + ndev->name, &id_priv->id); + work = kzalloc(sizeof *work, GFP_KERNEL); + if (!work) + return -ENOMEM; + INIT_WORK(&work->work, cma_ndev_work_handler); + work->id = id_priv; + work->event.event = RDMA_CM_EVENT_ADDR_CHANGE; + atomic_inc(&id_priv->refcount); + queue_work(cma_wq, &work->work); + } + + return 0; +} + +static int cma_netdev_callback(struct notifier_block *self, unsigned long event, + void *ctx) +{ + struct net_device *ndev = (struct net_device *)ctx; + struct cma_device *cma_dev; + struct rdma_id_private *id_priv; + int ret = NOTIFY_DONE; + + if (dev_net(ndev) != &init_net) + return NOTIFY_DONE; + + if (event != NETDEV_BONDING_FAILOVER) + return NOTIFY_DONE; + + if (!(ndev->flags & IFF_MASTER) || !(ndev->priv_flags & IFF_BONDING)) + return NOTIFY_DONE; + + mutex_lock(&lock); + list_for_each_entry(cma_dev, &dev_list, list) + list_for_each_entry(id_priv, &cma_dev->id_list, list) { + ret = cma_netdev_change(ndev, id_priv); + if (ret) + goto out; + } + +out: + mutex_unlock(&lock); + return ret; +} + +static struct notifier_block cma_nb = { + .notifier_call = cma_netdev_callback +}; + static void cma_add_one(struct ib_device *device) { struct cma_device *cma_dev; @@ -2831,6 +2918,7 @@ static int cma_init(void) ib_sa_register_client(&sa_client); rdma_addr_register_client(&addr_client); + register_netdevice_notifier(&cma_nb); ret = ib_register_client(&cma_client); if (ret) @@ -2838,6 +2926,7 @@ static int cma_init(void) return 0; err: + unregister_netdevice_notifier(&cma_nb); rdma_addr_unregister_client(&addr_client); ib_sa_unregister_client(&sa_client); destroy_workqueue(cma_wq); @@ -2847,6 +2936,7 @@ err: static void cma_cleanup(void) { ib_unregister_client(&cma_client); + unregister_netdevice_notifier(&cma_nb); rdma_addr_unregister_client(&addr_client); ib_sa_unregister_client(&sa_client); destroy_workqueue(cma_wq); Index: infiniband/include/rdma/rdma_cm.h =================================================================== --- infiniband.orig/include/rdma/rdma_cm.h +++ infiniband/include/rdma/rdma_cm.h @@ -57,7 +57,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_ADDR_CHANGE }; enum rdma_port_space { From ogerlitz at voltaire.com Wed Jun 18 03:56:15 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Jun 2008 13:56:15 +0300 (IDT) Subject: [ofa-general] [PATCH 3/4] ib/iser: add support for the ADDR_CHANGE rdma-cm event In-Reply-To: References: Message-ID: enhance iser to act upon notification on network stack changes which makes its rdma connection unaligned with the link used by the stack for the IPs used to establish the connection. When RDMA_CM_EVENT_ADDR_CHANGE arrives, just disconnect the connection, following that the user space iscsid daemon would reconnect, and the new connection would be aligned with the IP stack. Signed-off-by: Or Gerlitz drivers/infiniband/ulp/iser/iser_verbs.c | 1 + 1 files changed, 1 insertion(+) Index: infiniband/drivers/infiniband/ulp/iser/iser_verbs.c =================================================================== --- infiniband.orig/drivers/infiniband/ulp/iser/iser_verbs.c +++ infiniband/drivers/infiniband/ulp/iser/iser_verbs.c @@ -472,6 +472,7 @@ static int iser_cma_handler(struct rdma_ break; case RDMA_CM_EVENT_DISCONNECTED: case RDMA_CM_EVENT_DEVICE_REMOVAL: + case RDMA_CM_EVENT_ADDR_CHANGE: iser_disconnected_handler(cma_id); break; default: From sashak at voltaire.com Wed Jun 18 03:59:24 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 18 Jun 2008 13:59:24 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c : TRIVIAL fixes in log handling In-Reply-To: <48581BAE.7030108@dev.mellanox.co.il> References: <48579A86.5070508@dev.mellanox.co.il> <20080617162119.GE25554@sashak.voltaire.com> <48581BAE.7030108@dev.mellanox.co.il> Message-ID: <20080618105924.GO13363@sashak.voltaire.com> On 23:16 Tue 17 Jun , Yevgeny Kliteynik wrote: > Sasha Khapyorsky wrote: >> Hi Yevgeny, >> On 14:05 Tue 17 Jun , Yevgeny Kliteynik wrote: >>> OSM_LOG_ENTER() at the beginning of the function and >>> OSM_LOG_EXIT() at the end of the function were missing. >> Why? The function is pretty isolated and we have a lot of useless >> logging right now - with -V initial OpenSM log files can grow to GBs. > > There's an OSM_LOG_EXIT() in this function w/o OSM_LOG_ENTER(). > I can just remove the OSM_LOG_EXIT() instead - fine by me. Cool. I did already. Thanks. Sasha From sashak at voltaire.com Wed Jun 18 04:04:16 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 18 Jun 2008 14:04:16 +0300 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> Message-ID: <20080618110416.GP13363@sashak.voltaire.com> On 16:07 Wed 18 Jun , Keshetti Mahesh wrote: > > It will be lot helpful if some one can make changes to 'sminfo' utility so > that active routing algorithm information is also reported along with the > existing output. It is not related information to sminfo. If you want to do this remotely you can use OpenSM console (need to be enabled during ./configure). Sasha From hrosenstock at xsigo.com Wed Jun 18 04:17:14 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 04:17:14 -0700 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> Message-ID: <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-18 at 16:07 +0530, Keshetti Mahesh wrote: > >grep "tables configured on all switches" /var/log/opensm.log > >You will see the routing name in the full message. > > I have mentioned this in my query also. But what if someone > (working on some node[X] which is part of same subnet) who doesn't have > access to the node/switch where OpenSM is running want to know the same ? > > It will be lot helpful if some one can make changes to 'sminfo' utility so > that active routing algorithm information is also reported along with the > existing output. If that were to be done, it would need to be done via a vendor specific SM query as there is no standard for IB routing algorithms or reporting this to other nodes. It would not work with other SMs or prior OpenSMs. Is this really needed over IB ? Isn't some remote access to the SM node sufficient for this and then use one of the methods previously described on the list ? -- Hal > -Mahesh > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From keshetti.mahesh at gmail.com Wed Jun 18 04:27:45 2008 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Wed, 18 Jun 2008 16:57:45 +0530 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> Message-ID: <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> Thanks for your replies. > If that were to be done, it would need to be done via a vendor specific > SM query as there is no standard for IB routing algorithms or reporting > this to other nodes. It would not work with other SMs or prior OpenSMs. > Is this really needed over IB ? Isn't some remote access to the SM node > sufficient for this and then use one of the methods previously described > on the list ? Yes, all the above methods are sufficient to find out the active routing algorithm if the user has the remote access to the SM node. But in our subnet, SM node is also a compute node and when some user reserves that node for running jobs no one else except 'root' is allowed to enter that node. There I am facing the problem. -Mahesh From hrosenstock at xsigo.com Wed Jun 18 05:29:51 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 05:29:51 -0700 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> Message-ID: <1213792191.13056.98.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-18 at 16:57 +0530, Keshetti Mahesh wrote: > Thanks for your replies. > > > If that were to be done, it would need to be done via a vendor specific > > SM query as there is no standard for IB routing algorithms or reporting > > this to other nodes. It would not work with other SMs or prior OpenSMs. > > Is this really needed over IB ? Isn't some remote access to the SM node > > sufficient for this and then use one of the methods previously described > > on the list ? > > Yes, all the above methods are sufficient to find out the active > routing algorithm > if the user has the remote access to the SM node. But in our subnet, SM node > is also a compute node and when some user reserves that node for running jobs > no one else except 'root' is allowed to enter that node. There I am > facing the problem. Stepping back for a moment, why do other nodes need to know the routing algorithm being used ? How would this information be used ? -- Hal > -Mahesh From kaber at trash.net Wed Jun 18 05:38:00 2008 From: kaber at trash.net (Patrick McHardy) Date: Wed, 18 Jun 2008 14:38:00 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <71d336490806180532g4ea7067akc157581cb1fa946d@mail.gmail.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <20080614190358.GU22807@obsidianresearch.com> <48562A41.7000304@trash.net> <71d336490806180532g4ea7067akc157581cb1fa946d@mail.gmail.com> Message-ID: <485901A8.8060705@trash.net> Ramachandra K wrote: > Is your suggestion that the netdevice needs to be registered as soon > as we get parameters from userspace even if that netdevice stays > inactive until we finish all of our communication with the QLogic EVIC > gateway device over the IB network ? Yes, it should be created as soon as the administrator issues the command to do so. > Currently we register netdevice only after we have made sure that we > can connect to the EVIC and that there are enough resources on the > EVIC to support this virtual ethernet interface. But if we register > the netdevice > without making sure that we can reach the EVIC and that it has resources, > there is a possibility that we register a netdevice that can never > become active. Thats similar to a ethernet device that never has a cable plugged in. You should register the device with its carrier turned off, then change the carrier state once the connection has been established. > Isn't our current behavior analogous to an ethernet driver not > registering a netdev if it > can't properly initialize its hardware ? No, hardware probing is done automatically, while with your device the administrator explicitly asks to create the device. The expected behaviour is that the device exists once the command finishes without returning an error. From ramachandra.kuchimanchi at qlogic.com Wed Jun 18 05:32:28 2008 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Wed, 18 Jun 2008 18:02:28 +0530 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <48562A41.7000304@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <20080614190358.GU22807@obsidianresearch.com> <48562A41.7000304@trash.net> Message-ID: <71d336490806180532g4ea7067akc157581cb1fa946d@mail.gmail.com> On Mon, Jun 16, 2008 at 2:24 PM, Patrick McHardy wrote: > Jason Gunthorpe wrote: >> >> On Sat, Jun 14, 2008 at 11:03:08AM -0700, Roland Dreier wrote: >>>> >>>> We have linkstate/operstate for this. How is a user supposed >>>> to configure the network device when it appears at a more or >>>> less random time from his perspective? >> >> >>> >>> Well, keep in mind that this driver is for an ethernet virtual NIC >>> that is actually >>> remote across another network (an InfiniBand fabric) -- so these devices >>> can actually appear or disappear at random times by their nature. It's a >>> similar issue to, say, a USB ethernet adapter presents -- there's no sane >>> way >>> to have operstate set for a device that hasn't been attached yet. >> >> I think that is only a fair analogy if the VNIC used some kind of >> hot-plug auto discovery procedure like USB does. As soon as you >> require the administrator to run a command to bind a device to a VNIC >> GID you are much more like a tunnel device and the ethernet device >> should exist from the moment the administator creates it up until the >> administator destroys it. The state of the connection to the VNIC >> should be reflected in some way other than device presence. > > Fully agreed. Is your suggestion that the netdevice needs to be registered as soon as we get parameters from userspace even if that netdevice stays inactive until we finish all of our communication with the QLogic EVIC gateway device over the IB network ? Currently we register netdevice only after we have made sure that we can connect to the EVIC and that there are enough resources on the EVIC to support this virtual ethernet interface. But if we register the netdevice without making sure that we can reach the EVIC and that it has resources, there is a possibility that we register a netdevice that can never become active. Isn't our current behavior analogous to an ethernet driver not registering a netdev if it can't properly initialize its hardware ? Regards, Ram From twpzkeqy at beyondmoseying.net Wed Jun 18 06:58:44 2008 From: twpzkeqy at beyondmoseying.net (Jorge) Date: Wed, 18 Jun 2008 05:58:44 -0800 Subject: [ofa-general] is it you? Jorge here Message-ID: <1-9122426-8JXtOjSRZFDSFpPg1EG12yS@atr.interaxel2.com> Hi, i am here sitting in the internet caffe. Found your email and decided to write. I am 25 y.o.girl. I have a picture if you want. No need to reply here as this is not may email. Write me at aJorge92 at easyglo.info From keshetti85-student at yahoo.co.in Wed Jun 18 03:06:03 2008 From: keshetti85-student at yahoo.co.in (keshetti85-student at yahoo.co.in) Date: Wed, 18 Jun 2008 15:36:03 +0530 (IST) Subject: ***SPAM*** Re: [ofa-general] [OpenSM] How to find active routing algorithm ? In-Reply-To: <4858CA98.3010604@dev.mellanox.co.il> Message-ID: <665932.86280.qm@web8314.mail.in.yahoo.com> >grep "tables configured on all switches" /var/log/opensm.log >You will see the routing name in the full message. I have mentioned this technique in my query also. But what if someone (working on some node[X] which is part of same subnet) who doesn't have access to the node/switch where OpenSM is running want to know the same ? It will be lot helpful if some one can make changes to 'sminfo' utility so that active routing algorithm information is also reported along with the existing output. -Mahesh Meet people who discuss and share your passions. Go to http://in.promos.yahoo.com/groups/bestofyahoo/ From amirv at mellanox.co.il Wed Jun 18 06:02:38 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Wed, 18 Jun 2008 16:02:38 +0300 Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event Message-ID: <4859076E.1050704@mellanox.co.il> Sean Hi, SDP needs to be notified when CM exits the TIMEWAIT state. CM does notify the CMA but CMA doesn't pass it to the upper layer. Below is a patch for the CMA code. I wasn't sure if it is ok to set CMA_DISCONNECT in the IB_CM_TIMEWAIT_EXIT instead of doing it in the IB_CM_DREQ_RECEIVED/IB_CM_DREP_RECEIVED as it is done today. Instead I changed the check at the beginning of cma_ib_handler() as you can see in the patch. Signed-off-by: Amir Vadai --- include/rdma/rdma_cm.h.orig 2008-06-18 15:04:22.560904000 +0300 +++ include/rdma/rdma_cm.h 2008-06-11 11:55:26.758053000 +0300 @@ -53,7 +53,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_TIMWAIT_EXIT }; enum rdma_port_space { --- drivers/infiniband/core/cma.c.orig 2008-06-11 11:24:38.021543000 +0300 +++ drivers/infiniband/core/cma.c 2008-06-18 13:27:08.098747000 +0300 @@ -931,7 +931,8 @@ static int cma_ib_handler(struct ib_cm_i struct rdma_cm_event event; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if ((ib_event->event != IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_CONNECT)) || + (ib_event->event == IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_DISCONNECT))) return 0; memset(&event, 0, sizeof event); @@ -967,6 +968,8 @@ static int cma_ib_handler(struct ib_cm_i event.event = RDMA_CM_EVENT_DISCONNECTED; break; case IB_CM_TIMEWAIT_EXIT: + event.event = RDMA_CM_EVENT_TIMWAIT_EXIT; + break; case IB_CM_MRA_RECEIVED: /* ignore event */ goto out; From amirv Wed Jun 18 15:39:39 2008 Date: Wed, 18 Jun 2008 15:39:39 +0300 From: amirv at mellanox.co.il To: cat Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event Message-ID: <4859020b.dbZUBXSQXSVeYrVX%amirv at mellanox.co.il> User-Agent: nail 11.25 7/29/05 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sean Hi, SDP needs to be notified when CM exits the TIMEWAIT state. CM does notify the CMA but CMA doesn't pass it to the upper layer. Below is a patch for the CMA code. I wasn't sure if it is ok to set CMA_DISCONNECT in the IB_CM_TIMEWAIT_EXIT instead of doing it in the IB_CM_DREQ_RECEIVED/IB_CM_DREP_RECEIVED as it is done today. Instead I changed the check at the beginning of cma_ib_handler() as you can see in the patch. Signed-off-by: Amir Vadai --- include/rdma/rdma_cm.h.orig 2008-06-18 15:04:22.560904000 +0300 +++ include/rdma/rdma_cm.h 2008-06-11 11:55:26.758053000 +0300 @@ -53,7 +53,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_TIMWAIT_EXIT }; enum rdma_port_space { --- drivers/infiniband/core/cma.c.orig 2008-06-11 11:24:38.021543000 +0300 +++ drivers/infiniband/core/cma.c 2008-06-18 13:27:08.098747000 +0300 @@ -931,7 +931,8 @@ static int cma_ib_handler(struct ib_cm_i struct rdma_cm_event event; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if ((ib_event->event != IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_CONNECT)) || + (ib_event->event == IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_DISCONNECT))) return 0; memset(&event, 0, sizeof event); @@ -967,6 +968,8 @@ static int cma_ib_handler(struct ib_cm_i event.event = RDMA_CM_EVENT_DISCONNECTED; break; case IB_CM_TIMEWAIT_EXIT: + event.event = RDMA_CM_EVENT_TIMWAIT_EXIT; + break; case IB_CM_MRA_RECEIVED: /* ignore event */ goto out; -- Amir Vadai Mellanox Technologies LTD. SW Phone: +972 (3) 625 9500 (ext.539) Mobile: +972 (52) 632 4714 E-mail: amirv at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From crackpoto49 at klassarch.uni-kiel.de Wed Jun 18 06:06:39 2008 From: crackpoto49 at klassarch.uni-kiel.de (Rita Farmer) Date: Wed, 18 Jun 2008 14:06:39 +0100 Subject: [ofa-general] =?iso-8859-1?q?Need_a_University_Degree_to_obtain_t?= =?iso-8859-1?q?he_career_you=92ve_always_wanted=3F_p_yegf_yxqlms?= Message-ID: <418050129.27247870368163@klassarch.uni-kiel.de> A Gen ie uine Un teu ivers xei ity Deg mwx ree in 4-6 we kob eks! Have you ever thought that the only thing stopping you from a great j uib ob and better p qy ay was a few let wxi ters behind your name? Well now you can get them! BA___B sn Sc___MA___MSc___MBA___PhD Within 4-6 weeks! No St uh udy Re jcc qui sf red! 100% Ver uf ifia nwf ble! These are real, ge gmo nuine degrees that include Bac tyd helors, Ma sd sters, MBA and Do mse ctorate Deg tmt rees. They are fully ve ls rifia lx ble and ce qu rtif jr ied transcripts are available. Just ca tgf ll the number below. You will thank me later 1-5 ly 20-2 fki 03-0 cld 21 eew 3Leave a message 24 hours a day, 7 days a week including Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Wed Jun 18 06:44:50 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 06:44:50 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/osm_port.h: Eliminate some unneeded includes Message-ID: <1213796690.13056.110.camel@hrosenstock-ws.xsigo.com> opensm/include/osm_port.h: Eliminate some unneeded includes Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index c1cbd39..ce9d68c 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -43,8 +43,6 @@ #ifndef _OSM_PORT_H_ #define _OSM_PORT_H_ -#include -#include #include #include #include @@ -511,8 +509,9 @@ static inline const osm_pkey_tbl_t *osm_physp_get_pkey_tbl(IN const osm_physp_t { CL_ASSERT(osm_physp_is_valid(p_physp)); /* - (14.2.5.7) - the block number valid values are 0-2047, and are further - limited by the size of the P_Key table specified by the PartitionCap on the node. + (14.2.5.7) - the block number valid values are 0-2047, and are + further limited by the size of the P_Key table specified by the + PartitionCap on the node. */ return (&p_physp->pkeys); }; @@ -558,7 +557,8 @@ osm_physp_set_slvl_tbl(IN osm_physp_t * const p_physp, * [in] Pointer to an osm_physp_t object. * * p_slvl_tbl -* [in] Pointer to the IBA defined SLtoVL map table for this port number. +* [in] Pointer to the IBA defined SLtoVL map table for this +* port number. * * in_port_num * [in] Input Port Number for this SLtoVL. @@ -634,7 +634,8 @@ osm_physp_set_vla_tbl(IN osm_physp_t * const p_physp, * [in] Pointer to an osm_physp_t object. * * p_vla_tbl -* [in] Pointer to the IBA defined VL Arbitration table for this port number. +* [in] Pointer to the IBA defined VL Arbitration table for this +* port number. * * block_num * [in] The part of the VL arbitration as defined in the IBA From hrosenstock at xsigo.com Wed Jun 18 06:44:54 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 06:44:54 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/osm_db.h: Fix some typos Message-ID: <1213796694.13056.112.camel@hrosenstock-ws.xsigo.com> opensm/include/osm_db.h: Fix some typos Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_db.h b/opensm/include/opensm/osm_db.h index a35bae8..2ea7483 100644 --- a/opensm/include/opensm/osm_db.h +++ b/opensm/include/opensm/osm_db.h @@ -59,11 +59,11 @@ BEGIN_C_DECLS * Database * * DESCRIPTION -* The OpenSM database interface provide the means to restore persistat -* data, query, modify, delete and evemtually commit it back to the +* The OpenSM database interface provide the means to restore persistent +* data, query, modify, delete and eventually commit it back to the * persistent media. * -* The interface is defined such that it can is not "data dependant": +* The interface is defined such that it can is not "data dependent": * All keys and data items are texts. * * The DB implementation should be thread safe, thus callers do not need to @@ -148,7 +148,7 @@ void osm_db_construct(IN osm_db_t * const p_db); /* * PARAMETERS * p_db -* [in] Pointer to the database object to custruct +* [in] Pointer to the database object to construct * * RETURN VALUES * NONE @@ -244,7 +244,8 @@ int osm_db_restore(IN osm_db_domain_t * p_domain); * PARAMETERS * * p_domain -* [in] Pointer to the database domain object to restore from persistent db +* [in] Pointer to the database domain object to restore +* from persistent db * * RETURN VALUES * 0 if successful 1 otherwize @@ -292,7 +293,8 @@ int osm_db_store(IN osm_db_domain_t * p_domain); * PARAMETERS * * p_domain -* [in] Pointer to the database domain object to restore from persistent db +* [in] Pointer to the database domain object to restore from +* persistent db * * RETURN VALUES * 0 if successful 1 otherwize From amirv at mellanox.co.il Wed Jun 18 06:38:58 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Wed, 18 Jun 2008 16:38:58 +0300 Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event (second try) Message-ID: <48590FF2.8090507@mellanox.co.il> Sean Hi, Sorry for the corrupted mail I sent before - had some problems with the mail client. SDP needs to be notified when CM exits the TIMEWAIT state. CM does notify the CMA but CMA doesn't pass it to the upper layer. Below is a patch for the CMA code. I wasn't sure if it is ok to set CMA_DISCONNECT in the IB_CM_TIMEWAIT_EXIT instead of doing it in the IB_CM_DREQ_RECEIVED/IB_CM_DREP_RECEIVED as it is done today. Instead I changed the check at the beginning of cma_ib_handler() as you can see in the patch. Signed-off-by: Amir Vadai --- include/rdma/rdma_cm.h.orig 2008-06-18 15:04:22.560904000 +0300 +++ include/rdma/rdma_cm.h 2008-06-11 11:55:26.758053000 +0300 @@ -53,7 +53,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_TIMWAIT_EXIT }; enum rdma_port_space { --- drivers/infiniband/core/cma.c.orig 2008-06-11 11:24:38.021543000 +0300 +++ drivers/infiniband/core/cma.c 2008-06-18 13:27:08.098747000 +0300 @@ -931,7 +931,8 @@ static int cma_ib_handler(struct ib_cm_i struct rdma_cm_event event; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if ((ib_event->event != IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_CONNECT)) || + (ib_event->event == IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_DISCONNECT))) return 0; memset(&event, 0, sizeof event); @@ -967,6 +968,8 @@ static int cma_ib_handler(struct ib_cm_i event.event = RDMA_CM_EVENT_DISCONNECTED; break; case IB_CM_TIMEWAIT_EXIT: + event.event = RDMA_CM_EVENT_TIMWAIT_EXIT; + break; case IB_CM_MRA_RECEIVED: /* ignore event */ goto out; -- Amir Vadai Mellanox Technologies LTD. SW Phone: +972 (3) 625 9500 (ext.539) Mobile: +972 (52) 632 4714 E-mail: amirv at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From hrosenstock at xsigo.com Wed Jun 18 07:04:20 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 07:04:20 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/osm_prefix_route.h: Support C++ inclusion Message-ID: <1213797860.13056.117.camel@hrosenstock-ws.xsigo.com> opensm/include/osm_prefix_route.h: Support C++ inclusion Also, make define name consistent with file name Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_prefix_route.h b/opensm/include/opensm/osm_prefix_route.h index 10b9aa4..aa3ad62 100644 --- a/opensm/include/opensm/osm_prefix_route.h +++ b/opensm/include/opensm/osm_prefix_route.h @@ -33,12 +33,22 @@ * */ -#ifndef _OSM_PREFIX_ROUTES_H_ -#define _OSM_PREFIX_ROUTES_H_ +#ifndef _OSM_PREFIX_ROUTE_H_ +#define _OSM_PREFIX_ROUTE_H_ #include #include +#ifdef __cplusplus +# define BEGIN_C_DECLS extern "C" { +# define END_C_DECLS } +#else /* !__cplusplus */ +# define BEGIN_C_DECLS +# define END_C_DECLS +#endif /* __cplusplus */ + +BEGIN_C_DECLS + typedef struct { cl_list_item_t list_item; /* must be first */ uint64_t prefix; /* network order, zero means "any" */ @@ -49,4 +59,5 @@ typedef struct { #error ROUTER_EXP is deprecated, specify prefix routes at runtime instead (see opensm man page for details) #endif -#endif /* _OSM_PREFIX_ROUTES_H_ */ +END_C_DECLS +#endif /* _OSM_PREFIX_ROUTE_H_ */ From hrosenstock at xsigo.com Wed Jun 18 07:09:40 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 07:09:40 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/osm_mtree.h: Eliminate unneeded include Message-ID: <1213798180.13056.121.camel@hrosenstock-ws.xsigo.com> opensm/include/osm_mtree.h: Eliminate unneeded include Also, other cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_mtree.h b/opensm/include/opensm/osm_mtree.h index e9d663c..0ea6755 100644 --- a/opensm/include/opensm/osm_mtree.h +++ b/opensm/include/opensm/osm_mtree.h @@ -43,7 +43,6 @@ #ifndef _OSM_MTREE_H_ #define _OSM_MTREE_H_ -#include #include #include #include @@ -192,7 +191,6 @@ osm_mtree_node_get_max_children(IN const osm_mtree_node_t * const p_mtn) { return (p_mtn->max_children); } - /* * PARAMETERS * p_mtn @@ -223,7 +221,6 @@ static inline osm_mtree_node_t *osm_mtree_node_get_child(IN const CL_ASSERT(child < p_mtn->max_children); return (p_mtn->child_array[child]); } - /* * PARAMETERS * p_mtn @@ -256,7 +253,6 @@ static inline osm_switch_t *osm_mtree_node_get_switch_ptr(IN const { return (p_mtn->p_sw); } - /* * PARAMETERS * p_mtn From weiny2 at llnl.gov Wed Jun 18 07:55:11 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 18 Jun 2008 07:55:11 -0700 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> Message-ID: <20080618075511.1aa48a10.weiny2@llnl.gov> I think the "easiest" method at this time is to use the console via an expect script on the SM node. Get the routing algo out of the console output (see "status" command). Then have that script "push" this information to all the nodes to be cached. Compute nodes then simply look at this file. You could set up a cron job to do this periodically although once OpenSM is run it should not change. But this would separate this script from having to know when OpenSM is restarted... That said, I really don't know why user jobs need to know this information. But right now the best way to get it is via the console. Ira On Wed, 18 Jun 2008 16:57:45 +0530 "Keshetti Mahesh" wrote: > Thanks for your replies. > > > If that were to be done, it would need to be done via a vendor specific > > SM query as there is no standard for IB routing algorithms or reporting > > this to other nodes. It would not work with other SMs or prior OpenSMs. > > Is this really needed over IB ? Isn't some remote access to the SM node > > sufficient for this and then use one of the methods previously described > > on the list ? > > Yes, all the above methods are sufficient to find out the active > routing algorithm > if the user has the remote access to the SM node. But in our subnet, SM node > is also a compute node and when some user reserves that node for running jobs > no one else except 'root' is allowed to enter that node. There I am > facing the problem. > > -Mahesh > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rrgwrd at boutiquehotelsrome.com Wed Jun 18 08:02:29 2008 From: rrgwrd at boutiquehotelsrome.com (Alvaro Jennings) Date: Wed, 18 Jun 2008 16:02:29 +0100 Subject: [ofa-general] Luxury Message-ID: <01c8d15c$b55c9880$59603252@rrgwrd> We are now bringing forward the replicas of the watches, Accessories, boxsets, Cufflinks, Keychains, Lighters, Pens, Tiffany & Co Jewelry. A replica presupposes the exact copy of the original, that is the quality and all the otherfeatures the original possesses. For instance,you might get a $18000 Rolex replica for a $229.00 price only! Our replica is of such an amazing quality that it can be destinguished from the originalonly in the jewellery workshop.You're anxiously wanting to be glamorous enough for demanding high soiciety standarts, but your earnings are not that sky high? You are willing to reward your lover with an outrageous gift but you don't want that to empty your wallet? Come, visit our shop!P.S When applying for a well-paid job, when flirting with a girl, whenasking for a credit in the bank you get your chances doubled with A Rolex President replica on your hand!PSS: The only missing thing here is a luxury suit!Have a look at our site -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Wed Jun 18 08:18:05 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 08:18:05 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions Message-ID: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> pensm/osm_port_profile: No need to inline osm_port_prof_is_ignored_port and osm_port_prof_set_ignored_port functions Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index 9997d0b..6777e15 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -114,7 +114,6 @@ static inline void osm_port_prof_construct(IN osm_port_profile_t * const p_prof) CL_ASSERT(p_prof); memset(p_prof, 0, sizeof(*p_prof)); } - /* * PARAMETERS * p_prof @@ -144,7 +143,6 @@ osm_port_prof_path_count_inc(IN osm_port_profile_t * const p_prof) CL_ASSERT(p_prof); p_prof->num_paths++; } - /* * PARAMETERS * p_prof @@ -172,7 +170,6 @@ osm_port_prof_path_count_get(IN const osm_port_profile_t * const p_prof) { return (p_prof->num_paths); } - /* * PARAMETERS * p_prof @@ -196,22 +193,9 @@ osm_port_prof_path_count_get(IN const osm_port_profile_t * const p_prof) * * SYNOPSIS */ -static inline boolean_t +boolean_t osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, - IN ib_net64_t port_guid, IN uint8_t port_num) -{ - const cl_map_t *p_map = &p_subn->port_prof_ignore_guids; - const void *p_obj = cl_map_get(p_map, port_guid); - size_t res; - - // HACK: we currently support ignoring ports 0 - 31 - if (p_obj != NULL) { - res = (size_t) p_obj & (size_t) (1 << port_num); - return (res != 0); - } - return FALSE; -} - + IN ib_net64_t port_guid, IN uint8_t port_num); /* * PARAMETERS * p_subn @@ -237,26 +221,9 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, * * SYNOPSIS */ -static inline void +void osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, - IN ib_net64_t port_guid, IN uint8_t port_num) -{ - cl_map_t *p_map = &p_subn->port_prof_ignore_guids; - const void *p_obj = cl_map_get(p_map, port_guid); - size_t value = 0; - - // HACK: we currently support ignoring ports 0 - 31 - CL_ASSERT(port_num < 32); - - if (p_obj != NULL) { - value = (size_t) p_obj; - cl_map_remove(p_map, port_guid); - } - - value = value | (1 << port_num); - cl_map_insert(p_map, port_guid, (void *)value); -} - + IN ib_net64_t port_guid, IN uint8_t port_num); /* * PARAMETERS * p_subn diff --git a/opensm/opensm/osm_port.c b/opensm/opensm/osm_port.c index 18f1ef2..f8a411e 100644 --- a/opensm/opensm/osm_port.c +++ b/opensm/opensm/osm_port.c @@ -756,14 +756,14 @@ osm_physp_set_pkey_tbl(IN osm_log_t * p_log, CL_ASSERT(p_pkey_tbl); /* - (14.2.5.7) - the block number valid values are 0-2047, and are further - limited by the size of the P_Key table specified by the PartitionCap on the - node. + (14.2.5.7) - the block number valid values are 0-2047, and are + further limited by the size of the P_Key table specified by the + the PartitionCap on the node. */ if (!p_physp->p_node->sw || p_physp->port_num == 0) /* - The maximum blocks is defined in the node info: partition cap for CA, - routers and switch management ports. + The maximum blocks is defined in the node info: partition cap + for CA, router, and switch management ports. */ max_blocks = (cl_ntoh16(p_physp->p_node->node_info.partition_cap) + @@ -771,8 +771,9 @@ osm_physp_set_pkey_tbl(IN osm_log_t * p_log, / IB_NUM_PKEY_ELEMENTS_IN_BLOCK; else /* - This is a switch, and not a management port. The maximum blocks is defined - in the switch info: partition enforcement cap. + This is a switch, and not a management port. The maximum + blocks is defined in the switch info: partition enforcement + cap. */ max_blocks = (cl_ntoh16(p_physp->p_node->sw->switch_info.enforce_cap) + @@ -791,3 +792,43 @@ osm_physp_set_pkey_tbl(IN osm_log_t * p_log, osm_pkey_tbl_set(&p_physp->pkeys, block_num, p_pkey_tbl); } + +/********************************************************************** + **********************************************************************/ +boolean_t +osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, + IN ib_net64_t port_guid, IN uint8_t port_num) +{ + const cl_map_t *p_map = &p_subn->port_prof_ignore_guids; + const void *p_obj = cl_map_get(p_map, port_guid); + size_t res; + + // HACK: we currently support ignoring ports 0 - 31 + if (p_obj != NULL) { + res = (size_t) p_obj & (size_t) (1 << port_num); + return (res != 0); + } + return FALSE; +} + +/********************************************************************** + **********************************************************************/ +void +osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, + IN ib_net64_t port_guid, IN uint8_t port_num) +{ + cl_map_t *p_map = &p_subn->port_prof_ignore_guids; + const void *p_obj = cl_map_get(p_map, port_guid); + size_t value = 0; + + // HACK: we currently support ignoring ports 0 - 31 + CL_ASSERT(port_num < 32); + + if (p_obj != NULL) { + value = (size_t) p_obj; + cl_map_remove(p_map, port_guid); + } + + value = value | (1 << port_num); + cl_map_insert(p_map, port_guid, (void *)value); +} From ogerlitz at voltaire.com Wed Jun 18 08:28:41 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Jun 2008 18:28:41 +0300 Subject: [ofa-general] [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <1213628310.5447.91.camel@mtls03> References: <1213628310.5447.91.camel@mtls03> Message-ID: <485929A9.6030306@voltaire.com> Eli Cohen wrote: > I plan to implement RCA for IPoIB and would like your comments. Using RCA allows better utilization of available CPU cores power by evenly distributing received SKBs between a few receive queues. Since each such queue is connected to a different CQ, which in turn is connected to an EQ which triggers a distinct IRQ vector, we can have more power to > process received buffers. Eli, Roland Can you point me to Ethernet driver in the mainline kernel that implements multiple receive queues? > In IPoIB support will be added only for UD mode. For sending we will use the same QP used today. For receive we will create a an RCA QP (by using IB_QP_CREATE_IPOIB_RCA in the create flags) and a group of QPs. The base QPN, number of QPs and a default QPN have to be passed to the RCA QP -- I am not sure what is the best way to do that but I am thinking of passing these values with modify QP to init using new fields in struct ib_qp_attr. The default QPN will be the current QP used for UD. We will have multiple receive rings (as the number of QPs) so we don't need to add spinlocks. I am trying to think if this design will allow to use later also multiple send queues, eg in the spirit of Documentation/networking/multiqueue.txt, what's your thinking? grepping on NETIF_F_MULTI_QUEUE under drivers/net does not retrieve many hits, so I wasn't sure if this is the current practice for multiple send queues nowadays. Or. From hrosenstock at xsigo.com Wed Jun 18 08:34:18 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 08:34:18 -0700 Subject: [ofa-general] [PATCH][MINOR] opensm/osm_switch: Don't inline osm_switch_sp0_is_lmc_capable function Message-ID: <1213803258.13056.134.camel@hrosenstock-ws.xsigo.com> opensm/osm_switch: Don't inline osm_switch_sp0_is_lmc_capable function Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index c1521a6..cb6ac53 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -251,7 +251,6 @@ osm_switch_get_hop_count(IN const osm_switch_t * const p_sw, return (lid_ho > p_sw->max_lid_ho || !p_sw->hops[lid_ho]) ? OSM_NO_PATH : p_sw->hops[lid_ho][port_num]; } - /* * PARAMETERS * p_sw @@ -285,7 +284,6 @@ static inline osm_fwd_tbl_t *osm_switch_get_fwd_tbl_ptr(IN const osm_switch_t * { return ((osm_fwd_tbl_t *) & p_sw->fwd_tbl); } - /* * PARAMETERS * p_sw @@ -371,7 +369,6 @@ osm_switch_get_least_hops(IN const osm_switch_t * const p_sw, return (lid_ho > p_sw->max_lid_ho || !p_sw->hops[lid_ho]) ? OSM_NO_PATH : p_sw->hops[lid_ho][0]; } - /* * PARAMETERS * p_sw @@ -437,7 +434,6 @@ osm_switch_get_port_by_lid(IN const osm_switch_t * const p_sw, { return (osm_fwd_tbl_get(&p_sw->fwd_tbl, lid_ho)); } - /* * PARAMETERS * p_sw @@ -517,7 +513,6 @@ static inline osm_physp_t *osm_switch_get_route_by_lid(IN const osm_switch_t * else return NULL; } - /* * PARAMETERS * p_sw @@ -547,14 +542,9 @@ static inline osm_physp_t *osm_switch_get_route_by_lid(IN const osm_switch_t * * Returns whether switch port 0 (SP0) can support LMC * */ -static inline unsigned +unsigned osm_switch_sp0_is_lmc_capable(IN const osm_switch_t * const p_sw, - IN osm_subn_t * p_subn) -{ - return (p_subn->opt.lmc_esp0 && - ib_switch_info_is_enhanced_port0(&p_sw->switch_info)) ? 1 : 0; -} - + IN osm_subn_t * p_subn); /* * PARAMETERS * p_sw @@ -588,7 +578,6 @@ osm_switch_get_max_block_id(IN const osm_switch_t * const p_sw) return ((uint32_t) (osm_fwd_tbl_get_size(&p_sw->fwd_tbl) / osm_fwd_tbl_get_lids_per_block(&p_sw->fwd_tbl))); } - /* * PARAMETERS * p_sw @@ -620,7 +609,6 @@ osm_switch_get_max_block_id_in_use(IN const osm_switch_t * const p_sw) cl_ntoh16(p_sw->switch_info. lin_top))); } - /* * PARAMETERS * p_sw @@ -684,7 +672,6 @@ osm_switch_supports_mcast(IN const osm_switch_t * const p_sw) { return (p_sw->switch_info.mcast_cap != 0); } - /* * PARAMETERS * p_sw @@ -716,7 +703,6 @@ osm_switch_set_switch_info(IN osm_switch_t * const p_sw, CL_ASSERT(p_si); p_sw->switch_info = *p_si; } - /* * PARAMETERS * p_sw @@ -747,7 +733,6 @@ osm_switch_count_path(IN osm_switch_t * const p_sw, IN const uint8_t port) { osm_port_prof_path_count_inc(&p_sw->p_prof[port]); } - /* * PARAMETERS * p_sw @@ -781,7 +766,6 @@ osm_switch_set_ft_block(IN osm_switch_t * const p_sw, CL_ASSERT(p_sw); return (osm_fwd_tbl_set_block(&p_sw->fwd_tbl, p_block, block_num)); } - /* * PARAMETERS * p_sw @@ -819,7 +803,6 @@ osm_switch_set_mft_block(IN osm_switch_t * const p_sw, return (osm_mcast_tbl_set_block(&p_sw->mcast_tbl, p_block, block_num, position)); } - /* * PARAMETERS * p_sw @@ -861,7 +844,6 @@ osm_switch_get_mft_block(IN osm_switch_t * const p_sw, return (osm_mcast_tbl_get_block(&p_sw->mcast_tbl, block_num, position, p_block)); } - /* * PARAMETERS * p_sw @@ -901,7 +883,6 @@ osm_switch_get_mft_max_block(IN osm_switch_t * const p_sw) CL_ASSERT(p_sw); return (osm_mcast_tbl_get_max_block(&p_sw->mcast_tbl)); } - /* * PARAMETERS * p_sw @@ -925,7 +906,6 @@ osm_switch_get_mft_max_block_in_use(IN osm_switch_t * const p_sw) CL_ASSERT(p_sw); return (osm_mcast_tbl_get_max_block_in_use(&p_sw->mcast_tbl)); } - /* * PARAMETERS * p_sw @@ -955,7 +935,6 @@ osm_switch_get_mft_max_position(IN osm_switch_t * const p_sw) CL_ASSERT(p_sw); return (osm_mcast_tbl_get_max_position(&p_sw->mcast_tbl)); } - /* * PARAMETERS * p_sw @@ -1070,7 +1049,6 @@ osm_switch_get_fwd_tbl_size(IN const osm_switch_t * const p_sw) { return (osm_fwd_tbl_get_size(&p_sw->fwd_tbl)); } - /* * PARAMETERS * p_sw @@ -1098,7 +1076,6 @@ osm_switch_get_mcast_fwd_tbl_size(IN const osm_switch_t * const p_sw) { return (cl_ntoh16(p_sw->switch_info.mcast_cap)); } - /* * PARAMETERS * p_sw @@ -1127,7 +1104,6 @@ osm_switch_path_count_get(IN const osm_switch_t * const p_sw, { return (osm_port_prof_path_count_get(&p_sw->p_prof[port_num])); } - /* * PARAMETERS * p_sw @@ -1186,7 +1162,6 @@ static inline osm_mcast_tbl_t *osm_switch_get_mcast_tbl_ptr(IN const { return ((osm_mcast_tbl_t *) & p_sw->mcast_tbl); } - /* * PARAMETERS * p_sw @@ -1222,7 +1197,6 @@ osm_switch_is_in_mcast_tree(IN const osm_switch_t * const p_sw, else return (FALSE); } - /* * PARAMETERS * p_sw diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index a9d13c8..554bbc4 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -370,13 +370,15 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, osm_switch_get_hop_count(p_sw, base_lid, port_num); /* - If we aren't using pre-defined user routes function, then - we need to make sure that the current path is the minimum one. - In case of having such a user function - this check will not - be done, and the old routing will be used. - Note: This means that it is the user's job to clean all data - in the forwarding tables that he wants to be overridden by the - minimum hop function. + If we aren't using pre-defined user routes + function, then we need to make sure that the + current path is the minimum one. In case of + having such a user function - this check will + not be done, and the old routing will be used. + Note: This means that it is the user's job to + clean all data in the forwarding tables that + he wants to be overridden by the minimum + hop function. */ if (hops == least_hops) return (port_num); @@ -671,3 +673,13 @@ osm_switch_recommend_mcast_path(IN osm_switch_t * const p_sw, CL_ASSERT(port_num < num_ports); return (port_num); } + +/********************************************************************** + **********************************************************************/ +unsigned +osm_switch_sp0_is_lmc_capable(IN const osm_switch_t * const p_sw, + IN osm_subn_t * p_subn) +{ + return (p_subn->opt.lmc_esp0 && + ib_switch_info_is_enhanced_port0(&p_sw->switch_info)) ? 1 : 0; +} From chu11 at llnl.gov Wed Jun 18 08:56:05 2008 From: chu11 at llnl.gov (Al Chu) Date: Wed, 18 Jun 2008 08:56:05 -0700 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <20080618075511.1aa48a10.weiny2@llnl.gov> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> <20080618075511.1aa48a10.weiny2@llnl.gov> Message-ID: <1213804565.5369.461.camel@cardanus.llnl.gov> Another idea would be to write an script that is setuid root, that can rsh/ssh/etc. to the remote node an get the information via the console or grepping /var/log. You just have to install the script on every compute node, make sure permissions are right, etc. Al On Wed, 2008-06-18 at 07:55 -0700, Ira Weiny wrote: > I think the "easiest" method at this time is to use the console via an expect > script on the SM node. Get the routing algo out of the console output (see > "status" command). Then have that script "push" this information to all the > nodes to be cached. Compute nodes then simply look at this file. > > You could set up a cron job to do this periodically although once OpenSM is run > it should not change. But this would separate this script from having to know > when OpenSM is restarted... > > That said, I really don't know why user jobs need to know this information. > But right now the best way to get it is via the console. > > Ira > > On Wed, 18 Jun 2008 16:57:45 +0530 > "Keshetti Mahesh" wrote: > > > Thanks for your replies. > > > > > If that were to be done, it would need to be done via a vendor specific > > > SM query as there is no standard for IB routing algorithms or reporting > > > this to other nodes. It would not work with other SMs or prior OpenSMs. > > > Is this really needed over IB ? Isn't some remote access to the SM node > > > sufficient for this and then use one of the methods previously described > > > on the list ? > > > > Yes, all the above methods are sufficient to find out the active > > routing algorithm > > if the user has the remote access to the SM node. But in our subnet, SM node > > is also a compute node and when some user reserves that node for running jobs > > no one else except 'root' is allowed to enter that node. There I am > > facing the problem. > > > > -Mahesh > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From eli at dev.mellanox.co.il Wed Jun 18 08:51:00 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 18 Jun 2008 18:51:00 +0300 Subject: [ofa-general] [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <485929A9.6030306@voltaire.com> References: <1213628310.5447.91.camel@mtls03> <485929A9.6030306@voltaire.com> Message-ID: <20080618155100.GC11489@mtls03> On Wed, Jun 18, 2008 at 06:28:41PM +0300, Or Gerlitz wrote: > Can you point me to Ethernet driver in the mainline kernel that implements > multiple receive queues? I don't know of any. > I am trying to think if this design will allow to use later also multiple > send queues, eg in the spirit of Documentation/networking/multiqueue.txt, > what's your thinking? grepping on NETIF_F_MULTI_QUEUE under drivers/net > does not retrieve many hits, so I wasn't sure if this is the current > practice for multiple send queues nowadays. > This RFC refers only to multiple queues in the receive flow and should not have any impact on future multiple send queue implementation. From swise at opengridcomputing.com Wed Jun 18 09:10:06 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Jun 2008 11:10:06 -0500 Subject: [ofa-general] [ANNOUNCE] chelsio libcxgb3 version 1.2.1 released In-Reply-To: <485718E3.3010903@opengridcomputing.com> References: <4856A93A.8010400@opengridcomputing.com> <485718E3.3010903@opengridcomputing.com> Message-ID: <4859335E.3030907@opengridcomputing.com> Steve Wise wrote: > As it stands now, it requires 6.0 firmware. > > > > Roland Dreier wrote: >>> Chelsio iWARP fans, >>> >>> Version 1.2.1 of libcxgb3 is available at: >>> >>> http://www.openfabrics.org/downloads/cxgb3/libcxgb3-1.2.1.tar.gz >>> >>> This library is packaged along with ofed-1.3.1 and is required for >>> 2.6.26 >>> kernels and beyond. >> >> I didn't look at this yet, but can it work with pre-6.0 firmware too? >> Would be nice to be able to upgrade the library once and then work >> with both old and new kernels... also makes it much easier for distro >> packaging, since then the libcxgb3 package doesn't have to worry >> about exactly what kernel is present. >> >> - R. Roland, after reviewing libcxgb, I believe the changes done in 1.2.x for peer2peer support (and 6.0 fw) are actually backwards compatible to 5.0. So I can create a 1.2.2 which will support 5.x and 6.x firmware. But ofed-1.3.1 has shipped with 1.2.1 and will require 6.0. Stay tuned. Steve. From sean.hefty at intel.com Wed Jun 18 09:11:39 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 18 Jun 2008 09:11:39 -0700 Subject: [ofa-general] RE: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: References: Message-ID: <000101c8d15d$fd873740$affc070a@amr.corp.intel.com> >Keep also the local (src) netdevice name in struct rdma_dev_addr. In a High- >Availability >scheme this can be used by the rdma-cm to align RDMA sessions to use the same >links as >the IP stack does under fail-over and route change cases. > >Signed-off-by: Or Gerlitz Reviewed-by: Sean Hefty From rdreier at cisco.com Wed Jun 18 09:10:55 2008 From: rdreier at cisco.com (Roland Dreier (rdreier)) Date: Wed, 18 Jun 2008 09:10:55 -0700 Subject: [ofa-general] [ANNOUNCE] chelsio libcxgb3 version 1.2.1 released References: <4856A93A.8010400@opengridcomputing.com> <485718E3.3010903@opengridcomputing.com> <4859335E.3030907@opengridcomputing.com> Message-ID: <20B709A530FCD04F865986FC15B54DE2535CEB@xmb-sjc-22d.amer.cisco.com> > Roland, after reviewing libcxgb, I believe the changes done in 1.2.x for > peer2peer support (and 6.0 fw) are actually backwards compatible to > 5.0. So I can create a 1.2.2 which will support 5.x and 6.x firmware. > But ofed-1.3.1 has shipped with 1.2.1 and will require 6.0. That would be great. I'm not that concerned about OFED 1.3.x but I don't feel good about updating my Debian/Ubuntu packages for libcxgb3 until there is a version that can work with both 5.x and 6.x firmware, since I can't really guess which kernel the package will be used with. - R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Wed Jun 18 09:12:56 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 18 Jun 2008 09:12:56 -0700 Subject: [ofa-general] RE: [PATCH 2/4] rdma/cma: simplify locking needed for serialization of callbacks In-Reply-To: References: Message-ID: <000201c8d15e$2b63f040$affc070a@amr.corp.intel.com> >The rdma-cm has some logic in place to make sure that callbacks on an ID are >delivered to the consumer in serialized manner, specifically it has code to >protect >against a device removal racing with a callback now being delivered to the >user. > >This patch simplifies this logic by using a mutex per ID instead of the wait >queue >and atomic variable, such that cma_disable_remove is now called >cma_disable_callback >and cma_enable_remove is now just mutex_unlock. > >Signed-off-by: Or Gerlitz Reviewed-by: Sean Hefty From sean.hefty at intel.com Wed Jun 18 09:16:15 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 18 Jun 2008 09:16:15 -0700 Subject: [ofa-general] RE: [PATCH 3/4] rdma/cma: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: <000301c8d15e$a26fdc30$affc070a@amr.corp.intel.com> >RDMA_CM_EVENT_ADDR_CHANGE event can be used by rdma-cm consumers that wish >to have their RDMA sessions always use the same links (eg ) as the >IP stack does. In the current code, this does not happen when bonding is used >and fail-over happened, but the IB link used by an already existing session is >operating fine. > >Use netevent notification for sensing that a change has happened in the IP >stack, >then scan the rdma-cm IDs list to see if there is an ID that is "misaligned" in >that respect with the IP stack, and deliver RDMA_CM_EVENT_ADDR_CHANGE for this >ID. >The user can act on the event or just ignore it > >Signed-off-by: Or Gerlitz Acked-by: Sean Hefty From marcel.heinz at informatik.tu-chemnitz.de Wed Jun 18 09:20:37 2008 From: marcel.heinz at informatik.tu-chemnitz.de (Marcel Heinz) Date: Wed, 18 Jun 2008 18:20:37 +0200 Subject: [ofa-general] Multicast Performance In-Reply-To: <4836E231.4000601@informatik.tu-chemnitz.de> References: <4836E231.4000601@informatik.tu-chemnitz.de> Message-ID: <485935D5.1040900@informatik.tu-chemnitz.de> Hi, Marcel Heinz wrote: > [Multicast bandwidth of only ~250MByte/s] I'm still fighting with this issue. I've played around with my code to track down possible causes of the effects, but it looks like I know even less than before. These are the old results (client is always running on host A, the hosts running the servers are listed right of the '->'): A -> [none] 258.867MB/s A -> A: 951.288MB/s A -> B: 258.863MB/s A -> A,B: 952.234MB/s A -> B,B: 143.878MB/s I've tested with the client attaching it's QP to the mcast group (but not posting any receive WRs to it). And this already changed something: A -> [none] 303.440MB/s A -> A: 99.121MB/s A -> B: 303.426MB/s A -> A,B: 99.207MB/s A -> B,B: 143.866MB/s The same effect can also be reproduced by starting another instance on Host A which just attaches a new QP to the mc group, and then doing nothing else, so it is not related to that particular QP I use for sending. After that, I checked what happens if I post just one WR to the client's receive queue (still attached to the mc group) and don't care about it any more. I'd expect that for that first WR, the behavior would be the same as in the scenario with having another server instance running on that host, and after that, it should behave like in the "attached but no WRs" scenario above. But this is not the case: A -> [none]: 383.125MB/s A -> A: 104.039MB/s A -> B: 383.130MB/s A -> A,B: 104.038MB/s A -> B,B: 143.920MB/s I don't know why, but the overall rate stays at 383MB/s instead of going back to the 300MB/s from the last test. I've tried to confirm these numbers by posting more recv WRs (this time also with polling the recv CQ), so that I really could benchmark the 2 different phases. I chose 1000000 and got: Phase I Phase II A -> [none]: 975.614MB/s 382.953MB/s A -> B: 975.614MB/s 382.953MB/s A -> A,B: 144.615MB/s 144.615MB/s A -> B,B: 143.911MB/s 143.852MB/s A -> A: 144.701MB/s 103.944MB/s At least my expectations for phase I were met. But I have no idea what could cause such effects. The fact that performance increases when the there is a "local" QP attached to the group indicates the this is a problem at the local side, not involving the switch at all, but I can't be sure. Has anyone an explanation for these numbers? Or any ideas what else I should check? Regards, Marcel From rdreier at cisco.com Wed Jun 18 10:48:01 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 10:48:01 -0700 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: <48512C94.1040306@voltaire.com> (Or Gerlitz's message of "Thu, 12 Jun 2008 17:03:00 +0300") References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> <48511089.8050609@voltaire.com> <48512C94.1040306@voltaire.com> Message-ID: > > ?? Nothing ties the CQ event vector number to a CPU number that I know of. > I am not sure to follow you. Do you want to say that the suggested > implementation in this patch is such that the EQ is not tied to a > specific CPU or that there is no proper mapping from the CPU this > process is shielded on to the vector (=EQ) number this CQ would be > associated with? Well, both of your statements seem to be true: nothing sets the affinity for the interrupts created for multiple EQs, and there is not any simply way to guarantee that CQ vector 5 is sent to CPU 5 that I see. From rdreier at cisco.com Wed Jun 18 10:48:18 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 10:48:18 -0700 Subject: [ofa-general] Re: [PATCH 1/3] mcast loopback block In-Reply-To: <3b5e77ad0806120350r7dff2239h60a44a8bc50ec943@mail.gmail.com> (Ron Livne's message of "Thu, 12 Jun 2008 13:50:00 +0300") References: <3b5e77ad0806120350r7dff2239h60a44a8bc50ec943@mail.gmail.com> Message-ID: > looking in the for-2.6.27 branch of your tree, > IB_DEVICE_SEND_W_INV (21) was removed and IB_DEVICE_MEM_MGT_EXTENSIONS (21) > was added instead. > So it seems that the value of 22 is free for adding a cap bit, if we decide > we go this way. Yes, you are right. From rdreier at cisco.com Wed Jun 18 10:57:57 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 10:57:57 -0700 Subject: [ofa-general] FW: QLogic vNIC Kernel Submission In-Reply-To: <99863D2ED484D449811D97A4C44C9CBD7C50F7@EPEXCH2.qlogic.org> (John Russo's message of "Tue, 17 Jun 2008 08:06:03 -0500") References: <99863D2ED484D449811D97A4C44C9CBD7C50F7@EPEXCH2.qlogic.org> Message-ID: John> It looks as if my original email was "scrubbed" before it made John> the mailing list so I am resending it... QLogic has been John> attempting to submit our virtual NIC (vNIC) driver to the John> Linux kernel for several months.  We have made changes to the John> code based on the feedback we have received over four rounds John> of submissions. Among the feedback we received during this John> process was a request to alter our code to use a single value John> per file for configuration of our driver through sysfs John> interface.  After spending much time and effort to complete John> this change to our design we re-submitted the driver only to John> receive a response suggesting that we change once again from John> this interface to a different API interface called rtnl_link.  John> Needless to say I am very frustrated with this process. This John> new API interface would require substantial changes to our John> code. John> QLogic has met the initial request to move to a single valued John> sysfs interface and I would hope that this new request will be John> waived and will not be a roadblock to inclusion of our driver John> to the kernel. No, we are not going to "waive" this request. The one thing that we really really want to get right when merging code is the userspace-visible interface it presents, because changing it after the fact is very painful. I can understand why this might be frustrating for you, but we're simply not going to do something that is technically wrong just because some code reviewers were busy when you first posted your code for review. From rdreier at cisco.com Wed Jun 18 11:13:38 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 11:13:38 -0700 Subject: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: (Or Gerlitz's message of "Wed, 18 Jun 2008 13:54:40 +0300 (IDT)") References: Message-ID: > Keep also the local (src) netdevice name in struct rdma_dev_addr. In a High-Availability > scheme this can be used by the rdma-cm to align RDMA sessions to use the same links as > the IP stack does under fail-over and route change cases. Is the netdevice name the best we can do? It seems pretty fragile in the face of renaming etc. From jgunthorpe at obsidianresearch.com Wed Jun 18 11:21:48 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Wed, 18 Jun 2008 12:21:48 -0600 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <485901A8.8060705@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <20080614190358.GU22807@obsidianresearch.com> <48562A41.7000304@trash.net> <71d336490806180532g4ea7067akc157581cb1fa946d@mail.gmail.com> <485901A8.8060705@trash.net> Message-ID: <20080618182148.GD28285@obsidianresearch.com> On Wed, Jun 18, 2008 at 02:38:00PM +0200, Patrick McHardy wrote: >> Currently we register netdevice only after we have made sure that >> we can connect to the EVIC and that there are enough resources on >> the EVIC to support this virtual ethernet interface. But if we >> register the netdevice without making sure that we can reach the >> EVIC and that it has resources, there is a possibility that we >> register a netdevice that can never become active. > > Thats similar to a ethernet device that never has a cable > plugged in. You should register the device with its carrier > turned off, then change the carrier state once the connection > has been established. I would also encourage you to embrace this view of the VNIC as a tunnel fully and let the administrator re-target an active netdevice to a different VNIC without tearing down the netdevice.. The user space interface for that pretty much come for free with the netlink implementation, one of the advantages I suppose.. Jason From or.gerlitz at gmail.com Wed Jun 18 11:37:24 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Wed, 18 Jun 2008 21:37:24 +0300 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: References: Message-ID: <15ddcffd0806181137k4f2c9b75y42f7a17ac195baa5@mail.gmail.com> On 6/18/08, Roland Dreier wrote: >> Keep also the local (src) netdevice name in struct rdma_dev_addr. In a High-Availability >> scheme this can be used by the rdma-cm to align RDMA sessions to use the same links >> as the IP stack does under fail-over and route change cases. > Is the netdevice name the best we can do? It seems pretty fragile in the face of renaming etc. Taking into account that under this scheme the hw address of the netdevice changes, I didn't see something really better than the name, were you thinking on some specific alternative? just saving pointer to the netdevice seemed quite not elegant... Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Wed Jun 18 11:53:17 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 18 Jun 2008 21:53:17 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: forward congestion class MADs to the HCA Message-ID: <20080618185317.GA8525@mtls03> >From e24d6092ee43d2bb632380ac1885348dc9b06113 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Wed, 18 Jun 2008 18:59:33 +0300 Subject: [PATCH] IB/mlx4: forward congestion class MADs to the HCA mlx4 supports IB_MGMT_CLASS_CONG_MGMT management class so make MADs bearing this management class processed by the HCA through MAD_IFC. Signed-off-by: Eli Cohen --- drivers/infiniband/hw/mlx4/mad.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c index 4c1e72f..cdca3a5 100644 --- a/drivers/infiniband/hw/mlx4/mad.c +++ b/drivers/infiniband/hw/mlx4/mad.c @@ -255,7 +255,8 @@ int mlx4_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num, return IB_MAD_RESULT_SUCCESS; } else if (in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_PERF_MGMT || in_mad->mad_hdr.mgmt_class == MLX4_IB_VENDOR_CLASS1 || - in_mad->mad_hdr.mgmt_class == MLX4_IB_VENDOR_CLASS2) { + in_mad->mad_hdr.mgmt_class == MLX4_IB_VENDOR_CLASS2 || + in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_CONG_MGMT) { if (in_mad->mad_hdr.method != IB_MGMT_METHOD_GET && in_mad->mad_hdr.method != IB_MGMT_METHOD_SET) return IB_MAD_RESULT_SUCCESS; -- 1.5.5.4 From swise at opengridcomputing.com Wed Jun 18 12:18:23 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Jun 2008 14:18:23 -0500 Subject: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: References: Message-ID: <48595F7F.3070806@opengridcomputing.com> Roland Dreier wrote: > > Keep also the local (src) netdevice name in struct rdma_dev_addr. In a High-Availability > > scheme this can be used by the rdma-cm to align RDMA sessions to use the same links as > > the IP stack does under fail-over and route change cases. > > Is the netdevice name the best we can do? It seems pretty fragile in > the face of renaming etc. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > shouldn't it keep the netdev ptr? Like a neigh entry or a dst entry. From swise at opengridcomputing.com Wed Jun 18 12:20:50 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Jun 2008 14:20:50 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: <15ddcffd0806181137k4f2c9b75y42f7a17ac195baa5@mail.gmail.com> References: <15ddcffd0806181137k4f2c9b75y42f7a17ac195baa5@mail.gmail.com> Message-ID: <48596012.3080601@opengridcomputing.com> Or Gerlitz wrote: > On 6/18/08, Roland Dreier > wrote: > > >> Keep also the local (src) netdevice name in struct rdma_dev_addr. > In a High-Availability > >> scheme this can be used by the rdma-cm to align RDMA sessions to > use the same links > >> as the IP stack does under fail-over and route change cases. > > > Is the netdevice name the best we can do? It seems pretty fragile > in the face of renaming etc. > > Taking into account that under this scheme the hw address of the > netdevice changes, I didn't see something really better than the name, > were you thinking on some specific alternative? just saving pointer to > the netdevice seemed quite not elegant... > > Or. The netdevice ptr is the one thing that won't change. Its used in other routing structs like dst_entry and neighbour. Steve. From yiftahs at voltaire.com Wed Jun 18 12:35:39 2008 From: yiftahs at voltaire.com (Yiftah Shahar) Date: Wed, 18 Jun 2008 22:35:39 +0300 Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes In-Reply-To: <1213721000.13056.38.camel@hrosenstock-ws.xsigo.com> Message-ID: <39C75744D164D948A170E9792AF8E7CA01450E04@exil.voltaire.com> Hi Al, Thanks for doing those tests, I think they clearly demonstrate the importance of doing the right routing when running with LMC>0, actually it is important to do correct routing anytime ;-) We can also see that Open MPI manage to get significant performance improvement (faster by up to 38%) when utilizing 4 routs (with PL patch) and mange to maintain scalability of this feature for almost 1000 ranks. We are thinking on additional ways to improve Open MPI utilization of multi routs so we can even get better performance in the future. Yiftah > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general- > bounces at lists.openfabrics.org] On Behalf Of Hal Rosenstock > Sent: Tuesday, June 17, 2008 19:43 > To: chu11 at llnl.gov > Cc: Yevgeny Kliteynik; moody20 at llnl.gov; OpenIB > Subject: [ofa-general] Re: [PATCH] opensm: preserve base lid routes > > On Tue, 2008-06-17 at 09:25 -0700, Al Chu wrote: > > Sorry, the numbers for the mvapich/openmpi numbers are in units of > > time. > > > I think microseconds. So lower is better. > > That's what wasn't clear. > > -- Hal > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general From rdreier at cisco.com Wed Jun 18 13:13:46 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 13:13:46 -0700 Subject: [ofa-general] Re: RDMA_CM_EVENT_ADDR_CHANGE notification merge In-Reply-To: <48577654.1010109@voltaire.com> (Or Gerlitz's message of "Tue, 17 Jun 2008 11:31:16 +0300") References: <000001c8cfcd$7ebe5fb0$9d37170a@amr.corp.intel.com> <48577654.1010109@voltaire.com> Message-ID: > Does the method I suggested of pulling some patches from Jeff's tree > over which my patches are dependent upon into the infiniband tree make > sense to you? no, I think the easiest thing to do is for me to ask Jeff to merge these patches through his tree. We don't have anything else pending in my tree that's going to create a conflict if we do that, do we? - R. From rdreier at cisco.com Wed Jun 18 13:15:43 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 13:15:43 -0700 Subject: [ofa-general] [PATCH RFCv2 2/2] RDMA/cxgb3: Add support for protocol statistics. In-Reply-To: <20080617213858.8122.64051.stgit@dell3.ogc.int> (Steve Wise's message of "Tue, 17 Jun 2008 16:38:59 -0500") References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <20080617213858.8122.64051.stgit@dell3.ogc.int> Message-ID: > + stats->iw.ipInReceive = m.ipInReceive_hi + m.ipInReceive_lo; Surely all the code like this is wrong? Shouldn't it be more like stats->iw.ipInReceive = ((u64) m.ipInReceive_hi << 32) + m.ipInReceive_lo; - R. From rdreier at cisco.com Wed Jun 18 13:22:57 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 13:22:57 -0700 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <20080617213856.8122.96180.stgit@dell3.ogc.int> (Steve Wise's message of "Tue, 17 Jun 2008 16:38:56 -0500") References: <20080617213856.8122.96180.stgit@dell3.ogc.int> Message-ID: > This patch adds a sysfs attribute group called "proto_stats" under > /sys/class/infiniband/$device/ and populates this group with protocol > statistics if they exist for a given device. Currently, only iWARP > stats are defined, but the code is designed to allow Infiniband protocol > stats if they become available. These stats are per-device and more > importantly -not- per port. Making this per-device makes sense I guess given that protocol processing is not bound to a port -- ie in IB we have alternate path migration that can make things switch ports, and for iWARP RNICs presumably an offloaded connection could migrate from one port to another if routing changes (although I doubt we have in-kernel support for this). And I would guess in most HW implementations, the transport-handling logic has no idea what physical port the packets it's working on is attached to. However I see that iw_nes has an ethtool stat for "Retransmission count", which is per-netdev and hence attached to a physical port. Is this a compromise to work with ethool, or is this really per-port for nes devices? How does Steve's approach work for you? - R. From rdreier at cisco.com Wed Jun 18 13:26:42 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 13:26:42 -0700 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <20080617213856.8122.96180.stgit@dell3.ogc.int> (Steve Wise's message of "Tue, 17 Jun 2008 16:38:56 -0500") References: <20080617213856.8122.96180.stgit@dell3.ogc.int> Message-ID: > +struct iw_protocol_stats { > + u64 tcpInSegs; > + u64 tcpOutSegs; > + u64 tcpRetransSeg; > + u64 tcpInErrs; > + u32 tcpActiveOpens; > + u32 tcpPassiveOpens; > + u32 tcpAttemptFails; > + u32 tcpEstabResets; > + u32 tcpOutRsts; > + u32 tcpCurrEstab; > + u32 tcpRtoMin; > + u32 tcpRtoMax; > + u64 ipInReceive; > + u64 ipInHdrErrors; > + u64 ipInAddrErrors; > + u64 ipInUnknownProtos; > + u64 ipInDiscards; > + u64 ipInDelivers; > + u64 ipOutRequests; > + u64 ipOutDiscards; > + u64 ipOutNoRoutes; > + u32 ipReasmTimeout; > + u32 ipReasmReqds; > + u32 ipReasmOKs; > + u32 ipReasmFails; > +}; Two comments about this: - how about if we make every field u64 and avoid having to worry about the field width? I guess cxgb3 made some implementation choices but for example I don't see any good reason that ipInHdrErrors should be 64 bits while tcpPassiveOpens is only 32 bits. - similarly let's include all the standard IPv4 and TCP MIB fields, and cxgb3 can just leave the ones it doesn't support as 0. it looks like this requires zero changes to the cxgb3 implementation anyway. - R. From rdreier at cisco.com Wed Jun 18 13:44:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 13:44:21 -0700 Subject: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: <15ddcffd0806181137k4f2c9b75y42f7a17ac195baa5@mail.gmail.com> (Or Gerlitz's message of "Wed, 18 Jun 2008 21:37:24 +0300") References: <15ddcffd0806181137k4f2c9b75y42f7a17ac195baa5@mail.gmail.com> Message-ID: > Taking into account that under this scheme the hw address of the netdevice > changes, I didn't see something really better than the name, were you > thinking on some specific alternative? just saving pointer to the netdevice > seemed quite not elegant... To me using the pointer to the netdev seems much cleaner than doing a string comparison on the name, but I'm probably missing something. Why didn't you like that idea? From swise at opengridcomputing.com Wed Jun 18 13:59:02 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Jun 2008 15:59:02 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int> Message-ID: <48597716.5070803@opengridcomputing.com> Roland Dreier wrote: > > +struct iw_protocol_stats { > > + u64 tcpInSegs; > > + u64 tcpOutSegs; > > + u64 tcpRetransSeg; > > + u64 tcpInErrs; > > + u32 tcpActiveOpens; > > + u32 tcpPassiveOpens; > > + u32 tcpAttemptFails; > > + u32 tcpEstabResets; > > + u32 tcpOutRsts; > > + u32 tcpCurrEstab; > > + u32 tcpRtoMin; > > + u32 tcpRtoMax; > > + u64 ipInReceive; > > + u64 ipInHdrErrors; > > + u64 ipInAddrErrors; > > + u64 ipInUnknownProtos; > > + u64 ipInDiscards; > > + u64 ipInDelivers; > > + u64 ipOutRequests; > > + u64 ipOutDiscards; > > + u64 ipOutNoRoutes; > > + u32 ipReasmTimeout; > > + u32 ipReasmReqds; > > + u32 ipReasmOKs; > > + u32 ipReasmFails; > > +}; > > Two comments about this: > > - how about if we make every field u64 and avoid having to worry about > the field width? I guess cxgb3 made some implementation choices but > for example I don't see any good reason that ipInHdrErrors should be > 64 bits while tcpPassiveOpens is only 32 bits. > This struct was taken from the chelsio MIB, which is based on the standard TCP and IP MIBs. But the this struct could certainly use u64. And that would simplify the code. Will do. > - similarly let's include all the standard IPv4 and TCP MIB fields, and > cxgb3 can just leave the ones it doesn't support as 0. it looks like > this requires zero changes to the cxgb3 implementation anyway. > Ok, but I'm fuzzy on which MIB values we should carry. For instance, some objects in the MIB dump the table of active TCP connections. I don't want to deal with that at this point. Stevo. From or.gerlitz at gmail.com Wed Jun 18 13:59:32 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Wed, 18 Jun 2008 23:59:32 +0300 Subject: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep the netdevice name in struct rdma_dev_addr In-Reply-To: References: <15ddcffd0806181137k4f2c9b75y42f7a17ac195baa5@mail.gmail.com> Message-ID: <15ddcffd0806181359r21f81f05p21709bb6c9957db4@mail.gmail.com> On 6/18/08, Roland Dreier wrote: >> just saving pointer to the netdevice seemed quite not elegant... > To me using the pointer to the netdev seems much cleaner than doing a > string comparison on the name, but I'm probably missing something. Why > didn't you like that idea? I was thinking that I would need to go into reference counting / book keeping on the netdevice, but as Steve noted, it seems doable, and I guess I don't need to issue a dev_get() just for keeping the copy of neigh->dev, correct? since its holds the rtnl lock, the netevent notififier should provide a safe context for comparision event->dev to id->addr->dev. If we agree on this approach and you prefer it, I can change the patch. Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Wed Jun 18 14:05:14 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Jun 2008 16:05:14 -0500 Subject: [ofa-general] [PATCH RFCv2 2/2] RDMA/cxgb3: Add support for protocol statistics. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <20080617213858.8122.64051.stgit@dell3.ogc.int> Message-ID: <4859788A.8080101@opengridcomputing.com> Roland Dreier wrote: > > + stats->iw.ipInReceive = m.ipInReceive_hi + m.ipInReceive_lo; > > Surely all the code like this is wrong? Shouldn't it be more like > > stats->iw.ipInReceive = ((u64) m.ipInReceive_hi << 32) + > m.ipInReceive_lo; > > - R. > Hrrrm, I would think so! Steve. From or.gerlitz at gmail.com Wed Jun 18 14:09:59 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Thu, 19 Jun 2008 00:09:59 +0300 Subject: [ofa-general] Re: RDMA_CM_EVENT_ADDR_CHANGE notification merge In-Reply-To: References: <000001c8cfcd$7ebe5fb0$9d37170a@amr.corp.intel.com> <48577654.1010109@voltaire.com> Message-ID: <15ddcffd0806181409ta4f03e2jcf5836a330f3019@mail.gmail.com> On 6/18/08, Roland Dreier wrote: > > > Does the method I suggested of pulling some patches from Jeff's tree > > over which my patches are dependent upon into the infiniband tree make > sense to you? no, I think the easiest thing to do is for me to ask Jeff to merge these > patches through his > tree. We don't have anything else pending in my tree that's going to > create a conflict if we do that, do we? > Generally speaking, yes, you can ask Jeff to pull them into his tree and merge them from there. But, we might have few more patches to the rdma-cm, eg something was sent today (see "cma: add support for TIMEWAIT_EXIT event") and other things may pop as well. So, can more patches for the rdma-cm be pushed from your tree if we go this way or a conflict would be introduced? my preference is that if its not too complicated for you to do this sync, you would merge the rdma pieces. Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Wed Jun 18 14:10:12 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 18 Jun 2008 14:10:12 -0700 Subject: [ofa-general] [PATCH] IB/mlx4: forward congestion class MADs to the HCA In-Reply-To: <20080618185317.GA8525@mtls03> References: <20080618185317.GA8525@mtls03> Message-ID: <1213823412.13056.156.camel@hrosenstock-ws.xsigo.com> Eli, On Wed, 2008-06-18 at 21:53 +0300, Eli Cohen wrote: > >From e24d6092ee43d2bb632380ac1885348dc9b06113 Mon Sep 17 00:00:00 2001 > From: Eli Cohen > Date: Wed, 18 Jun 2008 18:59:33 +0300 > Subject: [PATCH] IB/mlx4: forward congestion class MADs to the HCA > > mlx4 supports IB_MGMT_CLASS_CONG_MGMT management class so make MADs > bearing this management class processed by the HCA through MAD_IFC. Does the current MLX4 firmware support CC ? Also, just wondering if the intention is to head towards cc agent support as well ? -- Hal > Signed-off-by: Eli Cohen > --- > drivers/infiniband/hw/mlx4/mad.c | 3 ++- > 1 files changed, 2 insertions(+), 1 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c > index 4c1e72f..cdca3a5 100644 > --- a/drivers/infiniband/hw/mlx4/mad.c > +++ b/drivers/infiniband/hw/mlx4/mad.c > @@ -255,7 +255,8 @@ int mlx4_ib_process_mad(struct ib_device *ibdev, int mad_flags, u8 port_num, > return IB_MAD_RESULT_SUCCESS; > } else if (in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_PERF_MGMT || > in_mad->mad_hdr.mgmt_class == MLX4_IB_VENDOR_CLASS1 || > - in_mad->mad_hdr.mgmt_class == MLX4_IB_VENDOR_CLASS2) { > + in_mad->mad_hdr.mgmt_class == MLX4_IB_VENDOR_CLASS2 || > + in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_CONG_MGMT) { > if (in_mad->mad_hdr.method != IB_MGMT_METHOD_GET && > in_mad->mad_hdr.method != IB_MGMT_METHOD_SET) > return IB_MAD_RESULT_SUCCESS; From rdreier at cisco.com Wed Jun 18 14:24:00 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 14:24:00 -0700 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <48597716.5070803@opengridcomputing.com> (Steve Wise's message of "Wed, 18 Jun 2008 15:59:02 -0500") References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <48597716.5070803@opengridcomputing.com> Message-ID: > Ok, but I'm fuzzy on which MIB values we should carry. For instance, > some objects in the MIB dump the table of active TCP connections. I > don't want to deal with that at this point. mmm, good point, I didn't look closely at everything in the RFCs. I guess the relevant RFC is 2011. I was just thinking of what's in for IP and TCP: IPSTATS_MIB_INRECEIVES, /* InReceives */ IPSTATS_MIB_INHDRERRORS, /* InHdrErrors */ IPSTATS_MIB_INTOOBIGERRORS, /* InTooBigErrors */ IPSTATS_MIB_INNOROUTES, /* InNoRoutes */ IPSTATS_MIB_INADDRERRORS, /* InAddrErrors */ IPSTATS_MIB_INUNKNOWNPROTOS, /* InUnknownProtos */ IPSTATS_MIB_INTRUNCATEDPKTS, /* InTruncatedPkts */ IPSTATS_MIB_INDISCARDS, /* InDiscards */ IPSTATS_MIB_INDELIVERS, /* InDelivers */ IPSTATS_MIB_OUTFORWDATAGRAMS, /* OutForwDatagrams */ IPSTATS_MIB_OUTREQUESTS, /* OutRequests */ IPSTATS_MIB_OUTDISCARDS, /* OutDiscards */ IPSTATS_MIB_OUTNOROUTES, /* OutNoRoutes */ IPSTATS_MIB_REASMTIMEOUT, /* ReasmTimeout */ IPSTATS_MIB_REASMREQDS, /* ReasmReqds */ IPSTATS_MIB_REASMOKS, /* ReasmOKs */ IPSTATS_MIB_REASMFAILS, /* ReasmFails */ IPSTATS_MIB_FRAGOKS, /* FragOKs */ IPSTATS_MIB_FRAGFAILS, /* FragFails */ IPSTATS_MIB_FRAGCREATES, /* FragCreates */ IPSTATS_MIB_INMCASTPKTS, /* InMcastPkts */ IPSTATS_MIB_OUTMCASTPKTS, /* OutMcastPkts */ IPSTATS_MIB_INBCASTPKTS, /* InBcastPkts */ IPSTATS_MIB_OUTBCASTPKTS, /* OutBcastPkts */ TCP_MIB_RTOALGORITHM, /* RtoAlgorithm */ TCP_MIB_RTOMIN, /* RtoMin */ TCP_MIB_RTOMAX, /* RtoMax */ TCP_MIB_MAXCONN, /* MaxConn */ TCP_MIB_ACTIVEOPENS, /* ActiveOpens */ TCP_MIB_PASSIVEOPENS, /* PassiveOpens */ TCP_MIB_ATTEMPTFAILS, /* AttemptFails */ TCP_MIB_ESTABRESETS, /* EstabResets */ TCP_MIB_CURRESTAB, /* CurrEstab */ TCP_MIB_INSEGS, /* InSegs */ TCP_MIB_OUTSEGS, /* OutSegs */ TCP_MIB_RETRANSSEGS, /* RetransSegs */ TCP_MIB_INERRS, /* InErrs */ TCP_MIB_OUTRSTS, /* OutRsts */ From swise at opengridcomputing.com Wed Jun 18 15:04:43 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Jun 2008 17:04:43 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <48597716.5070803@opengridcomputing.com> Message-ID: <4859867B.5040704@opengridcomputing.com> Roland Dreier wrote: > > Ok, but I'm fuzzy on which MIB values we should carry. For instance, > > some objects in the MIB dump the table of active TCP connections. I > > don't want to deal with that at this point. > > mmm, good point, I didn't look closely at everything in the RFCs. I > guess the relevant RFC is 2011. > > I was just thinking of what's in for IP and TCP: > > IPSTATS_MIB_INRECEIVES, /* InReceives */ > IPSTATS_MIB_INHDRERRORS, /* InHdrErrors */ > IPSTATS_MIB_INTOOBIGERRORS, /* InTooBigErrors */ > IPSTATS_MIB_INNOROUTES, /* InNoRoutes */ > IPSTATS_MIB_INADDRERRORS, /* InAddrErrors */ > IPSTATS_MIB_INUNKNOWNPROTOS, /* InUnknownProtos */ > IPSTATS_MIB_INTRUNCATEDPKTS, /* InTruncatedPkts */ > IPSTATS_MIB_INDISCARDS, /* InDiscards */ > IPSTATS_MIB_INDELIVERS, /* InDelivers */ > IPSTATS_MIB_OUTFORWDATAGRAMS, /* OutForwDatagrams */ > IPSTATS_MIB_OUTREQUESTS, /* OutRequests */ > IPSTATS_MIB_OUTDISCARDS, /* OutDiscards */ > IPSTATS_MIB_OUTNOROUTES, /* OutNoRoutes */ > IPSTATS_MIB_REASMTIMEOUT, /* ReasmTimeout */ > IPSTATS_MIB_REASMREQDS, /* ReasmReqds */ > IPSTATS_MIB_REASMOKS, /* ReasmOKs */ > IPSTATS_MIB_REASMFAILS, /* ReasmFails */ > IPSTATS_MIB_FRAGOKS, /* FragOKs */ > IPSTATS_MIB_FRAGFAILS, /* FragFails */ > IPSTATS_MIB_FRAGCREATES, /* FragCreates */ > IPSTATS_MIB_INMCASTPKTS, /* InMcastPkts */ > IPSTATS_MIB_OUTMCASTPKTS, /* OutMcastPkts */ > IPSTATS_MIB_INBCASTPKTS, /* InBcastPkts */ > IPSTATS_MIB_OUTBCASTPKTS, /* OutBcastPkts */ > > TCP_MIB_RTOALGORITHM, /* RtoAlgorithm */ > TCP_MIB_RTOMIN, /* RtoMin */ > TCP_MIB_RTOMAX, /* RtoMax */ > TCP_MIB_MAXCONN, /* MaxConn */ > TCP_MIB_ACTIVEOPENS, /* ActiveOpens */ > TCP_MIB_PASSIVEOPENS, /* PassiveOpens */ > TCP_MIB_ATTEMPTFAILS, /* AttemptFails */ > TCP_MIB_ESTABRESETS, /* EstabResets */ > TCP_MIB_CURRESTAB, /* CurrEstab */ > TCP_MIB_INSEGS, /* InSegs */ > TCP_MIB_OUTSEGS, /* OutSegs */ > TCP_MIB_RETRANSSEGS, /* RetransSegs */ > TCP_MIB_INERRS, /* InErrs */ > TCP_MIB_OUTRSTS, /* OutRsts */ > I'll use these then. From ctung at NetEffect.com Wed Jun 18 15:23:13 2008 From: ctung at NetEffect.com (Chien Tung) Date: Wed, 18 Jun 2008 17:23:13 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int> Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> > Roland Dreier Wrote: > > Steve Wise Wrote: > > This patch adds a sysfs attribute group called > "proto_stats" under > /sys/class/infiniband/$device/ and > populates this group with protocol > statistics if they > exist for a given device. Currently, only iWARP > stats are > defined, but the code is designed to allow Infiniband > protocol > stats if they become available. These stats are > per-device and more > importantly -not- per port. > > Making this per-device makes sense I guess given that > protocol processing is not bound to a port -- ie in IB we > have alternate path migration that can make things switch > ports, and for iWARP RNICs presumably an offloaded connection > could migrate from one port to another if routing changes > (although I doubt we have in-kernel support for this). And I > would guess in most HW implementations, the > transport-handling logic has no idea what physical port the > packets it's working on is attached to. > > However I see that iw_nes has an ethtool stat for > "Retransmission count", which is per-netdev and hence > attached to a physical port. Is this a compromise to work > with ethool, or is this really per-port for nes devices? How > does Steve's approach work for you? The "Retransmission count" is per netdev and we support multiple netdev per physical port. We have designed our silicon to support stats at netdev level and it doesn't make sense for us to aggregate stats from all netdevs and report at a device level. Perhaps we can compromise, Chelsio can give device level stats at /sys/class/infiniband/$device and for NetEffect, we can provide netdev level stats below it? Chien From rdreier at cisco.com Wed Jun 18 15:36:41 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 15:36:41 -0700 Subject: [ofa-general] Re: [PATCH] ib_core: fix is_closed flag check in ib_uverbs_async_handler In-Reply-To: <200806171411.03001.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Tue, 17 Jun 2008 14:11:02 +0300") References: <200806171411.03001.jackm@dev.mellanox.co.il> Message-ID: thanks, applied. From rdreier at cisco.com Wed Jun 18 15:38:47 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Jun 2008 15:38:47 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This will get a couple of small fixes: Jack Morgenstein (1): IB/uverbs: Fix check of is_closed flag check in ib_uverbs_async_handler() Roland Dreier (1): RDMA/nes: Fix off-by-one in nes_reg_user_mr() error path drivers/infiniband/core/uverbs_main.c | 2 +- drivers/infiniband/hw/nes/nes_verbs.c | 4 +--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c index f806da1..caed42b 100644 --- a/drivers/infiniband/core/uverbs_main.c +++ b/drivers/infiniband/core/uverbs_main.c @@ -423,7 +423,7 @@ static void ib_uverbs_async_handler(struct ib_uverbs_file *file, unsigned long flags; spin_lock_irqsave(&file->async_file->lock, flags); - if (!file->async_file->is_closed) { + if (file->async_file->is_closed) { spin_unlock_irqrestore(&file->async_file->lock, flags); return; } diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c index 99b3c4a..d617da9 100644 --- a/drivers/infiniband/hw/nes/nes_verbs.c +++ b/drivers/infiniband/hw/nes/nes_verbs.c @@ -2456,10 +2456,8 @@ static struct ib_mr *nes_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, if ((page_count!=0)&&(page_count<<12)-(region->offset&(4096-1))>=region->length) goto enough_pages; if ((page_count&0x01FF) == 0) { - if (page_count>(1024*512)) { + if (page_count >= 1024 * 512) { ib_umem_release(region); - pci_free_consistent(nesdev->pcidev, 4096, vpbl.pbl_vbase, - vpbl.pbl_pbase); nes_free_resource(nesadapter, nesadapter->allocated_mrs, stag_index); kfree(nesmr); From rollerv at fh-landshut.de Wed Jun 18 16:24:40 2008 From: rollerv at fh-landshut.de (Beth Kent) Date: Wed, 18 Jun 2008 20:24:40 -0300 Subject: [ofa-general] Start earning the salary you deserve by obtaining the appropriate University Degree. g ku p Message-ID: <01c8d181$55c4b400$4ae729bd@rollerv> A Gen gnn uine Un wo ivers jo ity Deg esu ree in 4-6 we qo eks! Have you ever thought that the only thing stopping you from a great j yf ob and better p mr ay was a few let id ters behind your name? Well now you can get them! BA___B bq Sc___MA___MSc___MBA___PhD Within 4-6 weeks! No St pt udy Re vt qui vmf red! 100% Ver ckm ifia gkz ble! These are real, ge ki nuine degrees that include Bac ft helors, Ma cs sters, MBA and Do cqz ctorate Deg tq rees. They are fully ve joa rifia flx ble and ce dt rtif wxu ied transcripts are available. Just ca rgi ll the number below. You will thank me later 1-5 jr 20-2 ax 03-0 sn 21 mci 3Leave a message 24 hours a day, 7 days a week including Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaber at trash.net Wed Jun 18 18:19:14 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 19 Jun 2008 03:19:14 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <20080618182148.GD28285@obsidianresearch.com> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <20080614190358.GU22807@obsidianresearch.com> <48562A41.7000304@trash.net> <71d336490806180532g4ea7067akc157581cb1fa946d@mail.gmail.com> <485901A8.8060705@trash.net> <20080618182148.GD28285@obsidianresearch.com> Message-ID: <4859B412.7010204@trash.net> Jason Gunthorpe wrote: > On Wed, Jun 18, 2008 at 02:38:00PM +0200, Patrick McHardy wrote: > >>> Currently we register netdevice only after we have made sure that >>> we can connect to the EVIC and that there are enough resources on >>> the EVIC to support this virtual ethernet interface. But if we >>> register the netdevice without making sure that we can reach the >>> EVIC and that it has resources, there is a possibility that we >>> register a netdevice that can never become active. >> >> Thats similar to a ethernet device that never has a cable >> plugged in. You should register the device with its carrier >> turned off, then change the carrier state once the connection >> has been established. > > I would also encourage you to embrace this view of the VNIC as a > tunnel fully and let the administrator re-target an active netdevice > to a different VNIC without tearing down the netdevice.. That has nothing to do with when the device is registered. Look at the existing tunnel devices, they *all* follow the scheme I described (and all of them can be "retargeted"). The simple point is: not one device with a pure kernel driver uses asynchronous registration. The question you haven't answered is: why doesn't carrier state work for you? > The user space interface for that pretty much come for free with the > netlink implementation, one of the advantages I suppose.. Yes, a more or less clean interface (at least forceably following the major existing conventions) comes with it automatically (the driver private details still can be done well or less well of course). But as multiple people have already stated, we gladly offer assistance. From kaber at trash.net Wed Jun 18 18:26:45 2008 From: kaber at trash.net (Patrick McHardy) Date: Thu, 19 Jun 2008 03:26:45 +0200 Subject: [ofa-general] Re: [PATCH v4 14/14] QLogic VNIC: sysfs Documentation In-Reply-To: <4859B412.7010204@trash.net> References: <20080610205633.11186.45499.stgit@dale> <20080610210918.11186.64253.stgit@dale> <484F751F.7030407@trash.net> <48513E4E.4040601@trash.net> <20080614190358.GU22807@obsidianresearch.com> <48562A41.7000304@trash.net> <71d336490806180532g4ea7067akc157581cb1fa946d@mail.gmail.com> <485901A8.8060705@trash.net> <20080618182148.GD28285@obsidianresearch.com> <4859B412.7010204@trash.net> Message-ID: <4859B5D5.4090903@trash.net> Patrick McHardy wrote: > Jason Gunthorpe wrote: >> On Wed, Jun 18, 2008 at 02:38:00PM +0200, Patrick McHardy wrote: >> >>>> Currently we register netdevice only after we have made sure that >>>> we can connect to the EVIC and that there are enough resources on >>>> the EVIC to support this virtual ethernet interface. But if we >>>> register the netdevice without making sure that we can reach the >>>> EVIC and that it has resources, there is a possibility that we >>>> register a netdevice that can never become active. > >> >>> Thats similar to a ethernet device that never has a cable >>> plugged in. You should register the device with its carrier >>> turned off, then change the carrier state once the connection >>> has been established. >> >> I would also encourage you to embrace this view of the VNIC as a >> tunnel fully and let the administrator re-target an active netdevice >> to a different VNIC without tearing down the netdevice.. Maybe I misparsed this, sorry Jason :) From keshetti.mahesh at gmail.com Wed Jun 18 21:39:15 2008 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Thu, 19 Jun 2008 10:09:15 +0530 Subject: [ofa-general] Re: [OpenSM] How to find active routing algorithm ? In-Reply-To: <1213792191.13056.98.camel@hrosenstock-ws.xsigo.com> References: <829ded920806180123k4048533dw15ba78afad420b25@mail.gmail.com> <829ded920806180337y6fe5acfcg16190715d0f50066@mail.gmail.com> <1213787834.13056.94.camel@hrosenstock-ws.xsigo.com> <829ded920806180427m4c51e40fl2094fe7b61b3a94f@mail.gmail.com> <1213792191.13056.98.camel@hrosenstock-ws.xsigo.com> Message-ID: <829ded920806182139m22167e86xfeb643769954a6d1@mail.gmail.com> > Stepping back for a moment, why do other nodes need to know the routing > algorithm being used ? How would this information be used ? > Sometimes we observed changes in application performance when minhop and UPDN are running (which is obvious I guess). So users want to know what is the active routing algorithm before they run a job. -Mahesh From eli at dev.mellanox.co.il Thu Jun 19 00:10:07 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 10:10:07 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: forward congestion class MADs to the HCA In-Reply-To: <1213823412.13056.156.camel@hrosenstock-ws.xsigo.com> References: <20080618185317.GA8525@mtls03> <1213823412.13056.156.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080619071007.GA15786@mtls03> > > Does the current MLX4 firmware support CC ? > Yes, but it is not released yet. Current released FW versions will drop CC mads if they get them through MAD_IFC. > Also, just wondering if the intention is to head towards cc agent > support as well ? > Yes, we have it implemented but it is still not released. From tziporet at dev.mellanox.co.il Thu Jun 19 01:35:38 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 19 Jun 2008 11:35:38 +0300 Subject: [ofa-general] Several RDS patches In-Reply-To: <200806181156.05160.okir@lst.de> References: <200806171722.02780.olaf.kirch@oracle.com> <4858D507.5080503@dev.mellanox.co.il> <4858D9F8.4030300@mellanox.co.il> <200806181156.05160.okir@lst.de> Message-ID: <485A1A5A.3080506@mellanox.co.il> Olaf Kirch wrote: > Yes. I'm currently re-working my patch set for 1.4 and will > submit everything shortly. > > > Note that feature freeze date is July 15 but I prefer not to wait for the last moment Tziporet From vlad at lists.openfabrics.org Thu Jun 19 02:26:21 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 19 Jun 2008 02:26:21 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080619-0200 daily build status Message-ID: <20080619092622.0149AE61E1D@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.22 Build failed on i686 with linux-2.6.21.1 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080619-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From jackm at dev.mellanox.co.il Thu Jun 19 06:14:51 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Thu, 19 Jun 2008 16:14:51 +0300 Subject: [ofa-general] XRC inclusion plans Message-ID: <200806191614.52030.jackm@dev.mellanox.co.il> Roland, Are you planning to include XRC in Kernel 2.6.27? If (hopefully) yes, I will prepare the necessary kernel and userspace patches and re-submit. - Jack From rdreier at cisco.com Thu Jun 19 06:19:53 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 06:19:53 -0700 Subject: [ofa-general] XRC inclusion plans In-Reply-To: <200806191614.52030.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Thu, 19 Jun 2008 16:14:51 +0300") References: <200806191614.52030.jackm@dev.mellanox.co.il> Message-ID: > Are you planning to include XRC in Kernel 2.6.27? > If (hopefully) yes, I will prepare the necessary kernel and userspace patches and re-submit. Well, I had no plans since the patches had gotten so stale, but if you resubmit the patches we can take a look. - R. From ogerlitz at voltaire.com Thu Jun 19 07:05:35 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:05:35 +0300 (IDT) Subject: [ofa-general] [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support Message-ID: Roland, changes from v1 - use pointer to the net device instead the net device name. Only patches 1 and 3 were changed, 2 and 4 remained the same, I added v2 to their subject to prevent confusion. As I said, since some more changes are possible in the rdma-cm for 2.6.27, I think it would be better if you do this push, however, if you you prefer to push this through Jeff's tree, let it be. Or The four patches that follow implement the RDMA stack part of the ability to notify ulps which use the rdma-cm on networking changes that make them "non aligned" with the network stack wrt what links (eg ) are being used for communication. They passed review on the list. The patch sequence is dependent on the below three commits to the davem-next branch at Jeff Garzik netdev-2.6 tree, so my thought was that you can pull/merge this tree/branch into your for-2.6.27 branch, such that it compiles after accepting the rdma patches. commit 01f3109de49a889db8adf9116449727547ee497e Author: Or Gerlitz Date: Fri Jun 13 18:12:02 2008 -0700 bonding: deliver netdev event for fail-over under the active-backup mode commit 709f8a45e8521f2f4229e5fdf3ded1fb77e2ca4e Author: Or Gerlitz Date: Fri Jun 13 18:12:01 2008 -0700 bonding: bond_change_active_slave() cleanup under active-backup commit c1da4ac752b8b0411791d26c678fcf23d2eed242 Author: Or Gerlitz Date: Fri Jun 13 18:12:00 2008 -0700 net/core: add NETDEV_BONDING_FAILOVER event From ogerlitz at voltaire.com Thu Jun 19 07:06:11 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:06:11 +0300 (IDT) Subject: [ofa-general] [PATCH 1/4] rdma/addr: keep pointer to the netdevice in struct rdma_dev_addr In-Reply-To: References: Message-ID: Keep a pointer to the local (src) netdevice in struct rdma_dev_addr. In a High-Availability scheme this information can be used by the rdma-cm to align RDMA sessions to use the same links as the IP stack does under fail-over and route change cases. Signed-off-by: Or Gerlitz drivers/infiniband/core/addr.c | 1 + drivers/infiniband/core/cma.c | 8 +++++--- include/rdma/ib_addr.h | 1 + 3 files changed, 7 insertions(+), 3 deletions(-) Index: infiniband/drivers/infiniband/core/addr.c =================================================================== --- infiniband.orig/drivers/infiniband/core/addr.c +++ infiniband/drivers/infiniband/core/addr.c @@ -105,6 +105,7 @@ int rdma_copy_addr(struct rdma_dev_addr memcpy(dev_addr->broadcast, dev->broadcast, MAX_ADDR_LEN); if (dst_dev_addr) memcpy(dev_addr->dst_dev_addr, dst_dev_addr, MAX_ADDR_LEN); + dev_addr->src_dev = dev; return 0; } EXPORT_SYMBOL(rdma_copy_addr); Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -1002,6 +1002,7 @@ static struct rdma_id_private *cma_new_c union cma_ip_addr *src, *dst; __be16 port; u8 ip_ver; + int ret; if (cma_get_net_info(ib_event->private_data, listen_id->ps, &ip_ver, &port, &src, &dst)) @@ -1026,10 +1027,11 @@ static struct rdma_id_private *cma_new_c if (rt->num_paths == 2) rt->path_rec[1] = *ib_event->param.req_rcvd.alternate_path; - ib_addr_set_sgid(&rt->addr.dev_addr, &rt->path_rec[0].sgid); ib_addr_set_dgid(&rt->addr.dev_addr, &rt->path_rec[0].dgid); - ib_addr_set_pkey(&rt->addr.dev_addr, be16_to_cpu(rt->path_rec[0].pkey)); - rt->addr.dev_addr.dev_type = RDMA_NODE_IB_CA; + ret = rdma_translate_ip(&id->route.addr.src_addr, + &id->route.addr.dev_addr); + if (ret) + goto destroy_id; id_priv = container_of(id, struct rdma_id_private, id); id_priv->state = CMA_CONNECT; Index: infiniband/include/rdma/ib_addr.h =================================================================== --- infiniband.orig/include/rdma/ib_addr.h +++ infiniband/include/rdma/ib_addr.h @@ -61,6 +61,7 @@ struct rdma_dev_addr { unsigned char dst_dev_addr[MAX_ADDR_LEN]; unsigned char broadcast[MAX_ADDR_LEN]; enum rdma_node_type dev_type; + struct net_device *src_dev; }; /** From ogerlitz at voltaire.com Thu Jun 19 07:07:01 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:07:01 +0300 (IDT) Subject: [ofa-general] [PATCH v2 2/4] rdma/cma: simplify locking needed for serialization of callbacks In-Reply-To: References: Message-ID: The rdma-cm has some logic in place to make sure that callbacks on an ID are delivered to the consumer in serialized manner, specifically it has code to protect against a device removal racing with a callback now being delivered to the user. This patch simplifies this logic by using a mutex per ID instead of the wait queue and atomic variable, such that cma_disable_remove is now called cma_disable_callback and cma_enable_remove is now just mutex_unlock. Signed-off-by: Or Gerlitz drivers/infiniband/core/cma.c | 106 +++++++++++++++++++----------------------- 1 files changed, 50 insertions(+), 56 deletions(-) Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -130,8 +130,7 @@ struct rdma_id_private { struct completion comp; atomic_t refcount; - wait_queue_head_t wait_remove; - atomic_t dev_remove; + struct mutex handler_mutex; int backlog; int timeout_ms; @@ -355,26 +354,15 @@ static void cma_deref_id(struct rdma_id_ complete(&id_priv->comp); } -static int cma_disable_remove(struct rdma_id_private *id_priv, +static int cma_disable_callback(struct rdma_id_private *id_priv, enum cma_state state) { - unsigned long flags; - int ret; - - spin_lock_irqsave(&id_priv->lock, flags); - if (id_priv->state == state) { - atomic_inc(&id_priv->dev_remove); - ret = 0; - } else - ret = -EINVAL; - spin_unlock_irqrestore(&id_priv->lock, flags); - return ret; -} - -static void cma_enable_remove(struct rdma_id_private *id_priv) -{ - if (atomic_dec_and_test(&id_priv->dev_remove)) - wake_up(&id_priv->wait_remove); + mutex_lock(&id_priv->handler_mutex); + if (id_priv->state != state) { + mutex_unlock(&id_priv->handler_mutex); + return -EINVAL; + } + return 0; } static int cma_has_cm_dev(struct rdma_id_private *id_priv) @@ -399,8 +387,7 @@ struct rdma_cm_id *rdma_create_id(rdma_c mutex_init(&id_priv->qp_mutex); init_completion(&id_priv->comp); atomic_set(&id_priv->refcount, 1); - init_waitqueue_head(&id_priv->wait_remove); - atomic_set(&id_priv->dev_remove, 0); + mutex_init(&id_priv->handler_mutex); INIT_LIST_HEAD(&id_priv->listen_list); INIT_LIST_HEAD(&id_priv->mc_list); get_random_bytes(&id_priv->seq_num, sizeof id_priv->seq_num); @@ -927,7 +914,7 @@ static int cma_ib_handler(struct ib_cm_i struct rdma_cm_event event; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -984,12 +971,12 @@ static int cma_ib_handler(struct ib_cm_i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.ib = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -1101,7 +1088,7 @@ static int cma_req_handler(struct ib_cm_ int offset, ret; listen_id = cm_id->context; - if (cma_disable_remove(listen_id, CMA_LISTEN)) + if (cma_disable_callback(listen_id, CMA_LISTEN)) return -ECONNABORTED; memset(&event, 0, sizeof event); @@ -1122,7 +1109,7 @@ static int cma_req_handler(struct ib_cm_ goto out; } - atomic_inc(&conn_id->dev_remove); + mutex_lock(&conn_id->handler_mutex); mutex_lock(&lock); ret = cma_acquire_dev(conn_id); mutex_unlock(&lock); @@ -1144,7 +1131,7 @@ static int cma_req_handler(struct ib_cm_ !cma_is_ud_ps(conn_id->id.ps)) ib_send_cm_mra(cm_id, CMA_CM_MRA_SETTING, NULL, 0); mutex_unlock(&lock); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); goto out; } @@ -1153,11 +1140,11 @@ static int cma_req_handler(struct ib_cm_ release_conn_id: cma_exch(conn_id, CMA_DESTROYING); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(&conn_id->id); out: - cma_enable_remove(listen_id); + mutex_unlock(&listen_id->handler_mutex); return ret; } @@ -1223,7 +1210,7 @@ static int cma_iw_handler(struct iw_cm_i struct sockaddr_in *sin; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -1267,12 +1254,12 @@ static int cma_iw_handler(struct iw_cm_i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.iw = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -1288,7 +1275,7 @@ static int iw_conn_req_handler(struct iw struct ib_device_attr attr; listen_id = cm_id->context; - if (cma_disable_remove(listen_id, CMA_LISTEN)) + if (cma_disable_callback(listen_id, CMA_LISTEN)) return -ECONNABORTED; /* Create a new RDMA id for the new IW CM ID */ @@ -1300,19 +1287,19 @@ static int iw_conn_req_handler(struct iw goto out; } conn_id = container_of(new_cm_id, struct rdma_id_private, id); - atomic_inc(&conn_id->dev_remove); + mutex_lock(&conn_id->handler_mutex); conn_id->state = CMA_CONNECT; dev = ip_dev_find(&init_net, iw_event->local_addr.sin_addr.s_addr); if (!dev) { ret = -EADDRNOTAVAIL; - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } ret = rdma_copy_addr(&conn_id->id.route.addr.dev_addr, dev, NULL); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1321,7 +1308,7 @@ static int iw_conn_req_handler(struct iw ret = cma_acquire_dev(conn_id); mutex_unlock(&lock); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1337,7 +1324,7 @@ static int iw_conn_req_handler(struct iw ret = ib_query_device(conn_id->id.device, &attr); if (ret) { - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(new_cm_id); goto out; } @@ -1353,14 +1340,14 @@ static int iw_conn_req_handler(struct iw /* User wants to destroy the CM ID */ conn_id->cm_id.iw = NULL; cma_exch(conn_id, CMA_DESTROYING); - cma_enable_remove(conn_id); + mutex_unlock(&conn_id->handler_mutex); rdma_destroy_id(&conn_id->id); } out: if (dev) dev_put(dev); - cma_enable_remove(listen_id); + mutex_unlock(&listen_id->handler_mutex); return ret; } @@ -1592,7 +1579,7 @@ static void cma_work_handler(struct work struct rdma_id_private *id_priv = work->id; int destroy = 0; - atomic_inc(&id_priv->dev_remove); + mutex_lock(&id_priv->handler_mutex); if (!cma_comp_exch(id_priv, work->old_state, work->new_state)) goto out; @@ -1601,7 +1588,7 @@ static void cma_work_handler(struct work destroy = 1; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); if (destroy) rdma_destroy_id(&id_priv->id); @@ -1764,7 +1751,7 @@ static void addr_handler(int status, str struct rdma_cm_event event; memset(&event, 0, sizeof event); - atomic_inc(&id_priv->dev_remove); + mutex_lock(&id_priv->handler_mutex); /* * Grab mutex to block rdma_destroy_id() from removing the device while @@ -1793,13 +1780,13 @@ static void addr_handler(int status, str if (id_priv->id.event_handler(&id_priv->id, &event)) { cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); rdma_destroy_id(&id_priv->id); return; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); cma_deref_id(id_priv); } @@ -2126,7 +2113,7 @@ static int cma_sidr_rep_handler(struct i struct ib_cm_sidr_rep_event_param *rep = &ib_event->param.sidr_rep_rcvd; int ret = 0; - if (cma_disable_remove(id_priv, CMA_CONNECT)) + if (cma_disable_callback(id_priv, CMA_CONNECT)) return 0; memset(&event, 0, sizeof event); @@ -2167,12 +2154,12 @@ static int cma_sidr_rep_handler(struct i /* Destroy the CM ID by returning a non-zero value. */ id_priv->cm_id.ib = NULL; cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return ret; } out: - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return ret; } @@ -2570,9 +2557,12 @@ static int cma_ib_mc_handler(int status, int ret; id_priv = mc->id_priv; - if (cma_disable_remove(id_priv, CMA_ADDR_BOUND) && - cma_disable_remove(id_priv, CMA_ADDR_RESOLVED)) + mutex_lock(&id_priv->handler_mutex); + if ((id_priv->state != CMA_ADDR_BOUND) && + (id_priv->state != CMA_ADDR_RESOLVED)) { + mutex_unlock(&id_priv->handler_mutex); return 0; + } mutex_lock(&id_priv->qp_mutex); if (!status && id_priv->id.qp) @@ -2596,12 +2586,12 @@ static int cma_ib_mc_handler(int status, ret = id_priv->id.event_handler(&id_priv->id, &event); if (ret) { cma_exch(id_priv, CMA_DESTROYING); - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); rdma_destroy_id(&id_priv->id); return 0; } - cma_enable_remove(id_priv); + mutex_unlock(&id_priv->handler_mutex); return 0; } @@ -2760,6 +2750,7 @@ static int cma_remove_id_dev(struct rdma { struct rdma_cm_event event; enum cma_state state; + int ret = 0; /* Record that we want to remove the device */ state = cma_exch(id_priv, CMA_DEVICE_REMOVAL); @@ -2767,15 +2758,18 @@ static int cma_remove_id_dev(struct rdma return 0; cma_cancel_operation(id_priv, state); - wait_event(id_priv->wait_remove, !atomic_read(&id_priv->dev_remove)); + mutex_lock(&id_priv->handler_mutex); /* Check for destruction from another callback. */ if (!cma_comp(id_priv, CMA_DEVICE_REMOVAL)) - return 0; + goto out; memset(&event, 0, sizeof event); event.event = RDMA_CM_EVENT_DEVICE_REMOVAL; - return id_priv->id.event_handler(&id_priv->id, &event); + ret = id_priv->id.event_handler(&id_priv->id, &event); +out: + mutex_unlock(&id_priv->handler_mutex); + return ret; } static void cma_process_remove(struct cma_device *cma_dev) From ogerlitz at voltaire.com Thu Jun 19 07:07:27 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:07:27 +0300 (IDT) Subject: [ofa-general] [PATCH v2 3/4] rdma/cma: implement RDMA_CM_EVENT_ADDR_CHANGE notification In-Reply-To: References: Message-ID: RDMA_CM_EVENT_ADDR_CHANGE event can be used by rdma-cm consumers that wish to have their RDMA sessions always use the same links (eg ) as the IP stack does. In the current code, this does not happen when bonding is used and fail-over happened, but the IB link used by an already existing session is operating fine. Use netevent notification for sensing that a change has happened in the IP stack, then scan the rdma-cm IDs list to see if there is an ID that is "misaligned" in that respect with the IP stack, and deliver RDMA_CM_EVENT_ADDR_CHANGE for this ID. The user can act on the event or just ignore it Signed-off-by: Or Gerlitz drivers/infiniband/core/cma.c | 90 ++++++++++++++++++++++++++++++++++++++++++ include/rdma/rdma_cm.h | 3 - 2 files changed, 92 insertions(+), 1 deletion(-) Index: infiniband/drivers/infiniband/core/cma.c =================================================================== --- infiniband.orig/drivers/infiniband/core/cma.c +++ infiniband/drivers/infiniband/core/cma.c @@ -168,6 +168,12 @@ struct cma_work { struct rdma_cm_event event; }; +struct cma_ndev_work { + struct work_struct work; + struct rdma_id_private *id; + struct rdma_cm_event event; +}; + union cma_ip_addr { struct in6_addr ip6; struct { @@ -1595,6 +1601,29 @@ out: kfree(work); } +static void cma_ndev_work_handler(struct work_struct *_work) +{ + struct cma_ndev_work *work = container_of(_work, struct cma_ndev_work, work); + struct rdma_id_private *id_priv = work->id; + int destroy = 0; + + mutex_lock(&id_priv->handler_mutex); + if (id_priv->state == CMA_DESTROYING || + id_priv->state == CMA_DEVICE_REMOVAL) + goto out; + + if (id_priv->id.event_handler(&id_priv->id, &work->event)) { + cma_exch(id_priv, CMA_DESTROYING); + destroy = 1; + } +out: + mutex_unlock(&id_priv->handler_mutex); + cma_deref_id(id_priv); + if (destroy) + rdma_destroy_id(&id_priv->id); + kfree(work); +} + static int cma_resolve_ib_route(struct rdma_id_private *id_priv, int timeout_ms) { struct rdma_route *route = &id_priv->id.route; @@ -2723,6 +2752,64 @@ void rdma_leave_multicast(struct rdma_cm } EXPORT_SYMBOL(rdma_leave_multicast); +static int cma_netdev_change(struct net_device *ndev, struct rdma_id_private *id_priv) +{ + struct rdma_dev_addr *dev_addr; + struct cma_ndev_work *work; + + dev_addr = &id_priv->id.route.addr.dev_addr; + + if ((dev_addr->src_dev == ndev) && + memcmp(dev_addr->src_dev_addr, ndev->dev_addr, ndev->addr_len)) { + printk(KERN_INFO "RDMA CM addr change for ndev %s used by id %p\n", + ndev->name, &id_priv->id); + work = kzalloc(sizeof *work, GFP_KERNEL); + if (!work) + return -ENOMEM; + INIT_WORK(&work->work, cma_ndev_work_handler); + work->id = id_priv; + work->event.event = RDMA_CM_EVENT_ADDR_CHANGE; + atomic_inc(&id_priv->refcount); + queue_work(cma_wq, &work->work); + } + + return 0; +} + +static int cma_netdev_callback(struct notifier_block *self, unsigned long event, + void *ctx) +{ + struct net_device *ndev = (struct net_device *)ctx; + struct cma_device *cma_dev; + struct rdma_id_private *id_priv; + int ret = NOTIFY_DONE; + + if (dev_net(ndev) != &init_net) + return NOTIFY_DONE; + + if (event != NETDEV_BONDING_FAILOVER) + return NOTIFY_DONE; + + if (!(ndev->flags & IFF_MASTER) || !(ndev->priv_flags & IFF_BONDING)) + return NOTIFY_DONE; + + mutex_lock(&lock); + list_for_each_entry(cma_dev, &dev_list, list) + list_for_each_entry(id_priv, &cma_dev->id_list, list) { + ret = cma_netdev_change(ndev, id_priv); + if (ret) + goto out; + } + +out: + mutex_unlock(&lock); + return ret; +} + +static struct notifier_block cma_nb = { + .notifier_call = cma_netdev_callback +}; + static void cma_add_one(struct ib_device *device) { struct cma_device *cma_dev; @@ -2831,6 +2918,7 @@ static int cma_init(void) ib_sa_register_client(&sa_client); rdma_addr_register_client(&addr_client); + register_netdevice_notifier(&cma_nb); ret = ib_register_client(&cma_client); if (ret) @@ -2838,6 +2926,7 @@ static int cma_init(void) return 0; err: + unregister_netdevice_notifier(&cma_nb); rdma_addr_unregister_client(&addr_client); ib_sa_unregister_client(&sa_client); destroy_workqueue(cma_wq); @@ -2847,6 +2936,7 @@ err: static void cma_cleanup(void) { ib_unregister_client(&cma_client); + unregister_netdevice_notifier(&cma_nb); rdma_addr_unregister_client(&addr_client); ib_sa_unregister_client(&sa_client); destroy_workqueue(cma_wq); Index: infiniband/include/rdma/rdma_cm.h =================================================================== --- infiniband.orig/include/rdma/rdma_cm.h +++ infiniband/include/rdma/rdma_cm.h @@ -57,7 +57,8 @@ enum rdma_cm_event_type { RDMA_CM_EVENT_DISCONNECTED, RDMA_CM_EVENT_DEVICE_REMOVAL, RDMA_CM_EVENT_MULTICAST_JOIN, - RDMA_CM_EVENT_MULTICAST_ERROR + RDMA_CM_EVENT_MULTICAST_ERROR, + RDMA_CM_EVENT_ADDR_CHANGE }; enum rdma_port_space { From ogerlitz at voltaire.com Thu Jun 19 07:07:56 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:07:56 +0300 (IDT) Subject: [ofa-general] [PATCH v2 3/4] ib/iser: add support for the ADDR_CHANGE rdma-cm event In-Reply-To: References: Message-ID: enhance iser to act upon notification on network stack changes which makes its rdma connection unaligned with the link used by the stack for the IPs used to establish the connection. When RDMA_CM_EVENT_ADDR_CHANGE arrives, just disconnect the connection, following that the user space iscsid daemon would reconnect, and the new connection would be aligned with the IP stack. Signed-off-by: Or Gerlitz drivers/infiniband/ulp/iser/iser_verbs.c | 1 + 1 files changed, 1 insertion(+) Index: infiniband/drivers/infiniband/ulp/iser/iser_verbs.c =================================================================== --- infiniband.orig/drivers/infiniband/ulp/iser/iser_verbs.c +++ infiniband/drivers/infiniband/ulp/iser/iser_verbs.c @@ -472,6 +472,7 @@ static int iser_cma_handler(struct rdma_ break; case RDMA_CM_EVENT_DISCONNECTED: case RDMA_CM_EVENT_DEVICE_REMOVAL: + case RDMA_CM_EVENT_ADDR_CHANGE: iser_disconnected_handler(cma_id); break; default: From ogerlitz at voltaire.com Thu Jun 19 07:08:30 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:08:30 +0300 (IDT) Subject: [ofa-general] Re: [PATCH 1/4] rdma/addr: keep pointer to the netdevice in struct rdma_dev_addr In-Reply-To: References: Message-ID: Sorry, there was typo in the subject line, its v2 Or. From eli at dev.mellanox.co.il Thu Jun 19 07:15:20 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 17:15:20 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Check completion done when a command Message-ID: <20080619141520.GB15786@mtls03> timeouts Reply-To: >From 2ace4c00ef256a22a9a223ba658d62c1f6adf8e7 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Thu, 19 Jun 2008 16:54:43 +0300 Subject: [PATCH] IB/mlx4: Check completion done when a command timeouts When the system is busy it may happen that a command actually completed but it took more than the specified timeout till the task executing the command was actually given CPU time. This patch checks that the completion is really missing before failing. Signed-off-by: Eli Cohen --- Roland, we've been using this patch in ofed for a long time now. We have seen cases where there are many processes running and it takes quite some time before the task that executed the command gets CPU time. It seems that the semantics of wait_for_completion_timeout has loose relation to the actual time the event occurred -- however the done field itself gives a more accurate indication as for whether the command was completed or not. drivers/net/mlx4/cmd.c | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/net/mlx4/cmd.c b/drivers/net/mlx4/cmd.c index 70dff94..d50ee9a 100644 --- a/drivers/net/mlx4/cmd.c +++ b/drivers/net/mlx4/cmd.c @@ -279,10 +279,11 @@ static int mlx4_cmd_wait(struct mlx4_dev *dev, u64 in_param, u64 *out_param, mlx4_cmd_post(dev, in_param, out_param ? *out_param : 0, in_modifier, op_modifier, op, context->token, 1); - if (!wait_for_completion_timeout(&context->done, msecs_to_jiffies(timeout))) { - err = -EBUSY; - goto out; - } + if (!wait_for_completion_timeout(&context->done, msecs_to_jiffies(timeout))) + if (!context->done.done) { + err = -EBUSY; + goto out; + } err = context->result; if (err) -- 1.5.5.4 From eli at dev.mellanox.co.il Thu Jun 19 07:28:02 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 17:28:02 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Configure QP's max message size Message-ID: <20080619142802.GC15786@mtls03> >From 78f13ca2204743b0ab7ab82932c89c08e1ef674a Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Thu, 19 Jun 2008 17:24:07 +0300 Subject: [PATCH] IB/mlx4: Configure QP's max message size Configure QP's max message size according to the value queried from the HCA through QUERY_DEV_CAP. Signed-off-by: Eli Cohen --- drivers/infiniband/hw/mlx4/qp.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 8e02ecf..fe38c85 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -903,7 +903,8 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp, attr->path_mtu); goto out; } - context->mtu_msgmax = (attr->path_mtu << 5) | 31; + context->mtu_msgmax = (attr->path_mtu << 5) | + ilog2(dev->dev->caps.max_msg_sz); } if (qp->rq.wqe_cnt) -- 1.5.5.4 From hrosenstock at xsigo.com Thu Jun 19 07:35:28 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 19 Jun 2008 07:35:28 -0700 Subject: [ofa-general] [PATCHv2] ibsim: Support short RMPP packets Message-ID: <1213886129.13056.168.camel@hrosenstock-ws.xsigo.com> ibsim: Support for short RMPP packets (up thru 256 bytes total) Changes from v1: - Added length to simulation header (by reducing context size so as not to break ABI) and encoded length on sends and decoded and checked lengths on receive - Restored short/partial packet read check in umad2sim_read - In umad2sim_write, fixed setting of umad->length - Eliminated sim_req256 typedef Signed-off-by: Hal Rosenstock diff --git a/TODO b/TODO index 616ed71..b385ade 100644 --- a/TODO +++ b/TODO @@ -1,7 +1,7 @@ * transaction management - multiple transactions per client - hops simulation - - RMPP + - long RMPP * more attributes handled * MKey support * cleaner (less) debug/warning messages diff --git a/ibsim/ibsim.c b/ibsim/ibsim.c index e4ab6bc..cde9ac5 100644 --- a/ibsim/ibsim.c +++ b/ibsim/ibsim.c @@ -492,9 +492,9 @@ static int sim_read_pkt(int fd, int client) if (!dcl) continue; - VERB("%s %d bytes (%zu) to client %d fd %d", + VERB("%s %d bytes to client %d fd %d", dcl == cl ? "replying" : "forwarding", - size, sizeof(struct sim_request), dcl->id, dcl->fd); + size, dcl->id, dcl->fd); // reply ret = write(dcl->fd, buf, size); diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 675d95b..28fdd52 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -1128,14 +1129,14 @@ Smpfn *get_handle_fn(ib_rpc_t rpc, int response) return fn; } - return 0; // No MGTCLASS matched . + return 0; // No MGTCLASS matched. } int process_packet(Client * cl, void *p, int size, Client ** dcl) { struct sim_request *r = p; Port *port; - uint8_t data[256]; + uint8_t data[MAD_BLOCK_SIZE]; int status, tlid, tqp; int response; Smpfn *fn; @@ -1144,9 +1145,10 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) *dcl = cl; - DEBUG("client %d, size %d", cl->id, size); - if (size != sizeof(*r)) { - IBWARN("bad packet size %d (!= %zu)", size, sizeof(*r)); + DEBUG("client %d size %d", cl->id, size); + if (size > sizeof(*r) + MAD_BLOCK_SIZE) { + IBWARN("unsupported packet size %d (> %zu)", size, + sizeof(*r) + MAD_BLOCK_SIZE); return -1; } @@ -1183,7 +1185,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) VERB("forward pkt to client %d pid %d attr %d", (*dcl)->id, (*dcl)->pid, rpc.attr.id); forward_MAD(r->mad, &rpc, &path); - return sizeof(*r); // forward only + return size; // forward only } if (port->errrate && (random() % 100) < port->errrate) { @@ -1214,12 +1216,12 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) VERB("PKT roll back did not succeed"); goto _dropped; } - return sizeof(*r); + return sizeof(*r) + MAD_BLOCK_SIZE; _dropped: r->status = htonl(110); *dcl = cl; - return sizeof(*r); + return sizeof(*r) + MAD_BLOCK_SIZE; } static int encode_trap128(Port * port, char *data) @@ -1279,48 +1281,60 @@ static int encode_trap_header(char *buf) int send_trap(Port * port, int trapnum) { - struct sim_request req; + struct sim_request *req; Client *cl; int ret, lid = port->lid; - char *data = req.mad + 64; /* data offset */ EncodeTrapfn *encode_trapfn = encodetrap[trapnum]; Port *destport; if (!encode_trapfn) { - IBWARN("trap number %d not upported", trapnum); + IBWARN("trap number %d not supported", trapnum); return -1; } - memset(req.mad, 0, sizeof(req.mad)); - encode_trap_header(req.mad); - if (encode_trapfn(port, data) < 0) + req = malloc(sizeof(*req) + MAD_BLOCK_SIZE); + if (!req) { + IBWARN("no mem for trap MAD: %m"); return -1; + } + + memset(req->mad, 0, MAD_BLOCK_SIZE); + encode_trap_header(req->mad); + if (encode_trapfn(port, req->mad + 64) < 0) { + free(req); + return -1; + } if (!(destport = lid_route_MAD(port, port->smlid))) { + free(req); IBWARN("routing failed: no route to dest lid %d", port->smlid); return -1; } - req.dlid = htons(port->smlid); - req.slid = htons(lid); - req.sqp = 0; - req.dqp = 0; - req.status = 0; - req.context = 0; + req->dlid = htons(port->smlid); + req->slid = htons(lid); + req->sqp = 0; + req->dqp = 0; + req->status = 0; + req->length = MAD_BLOCK_SIZE; + req->context = 0; // find SM client cl = find_client(destport, 0, 1, 0); - if (!cl) + if (!cl) { + free(req); return 0; + } if (simverb > 2) { xdump(stdout, "--- packet ---\n", &req, 256); fflush(stdout); } - ret = write(cl->fd, &req, sizeof(req)); - if (ret == sizeof(req)) + ret = write(cl->fd, req, sizeof(*req) + MAD_BLOCK_SIZE); + free(req); + if (ret == sizeof(*req) + MAD_BLOCK_SIZE) return 0; if (ret < 0 && (errno == ECONNREFUSED || errno == ENOTCONN)) { diff --git a/include/ibsim.h b/include/ibsim.h index 84568e6..2c6a841 100644 --- a/include/ibsim.h +++ b/include/ibsim.h @@ -1,5 +1,6 @@ /* * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -61,14 +62,17 @@ struct sim_port { #define SIM_MAGIC 0xdeadbeef #define SIM_CTL_MAX_DATA 64 +#define MAD_BLOCK_SIZE 256 + struct sim_request { uint32_t dlid; uint32_t slid; uint32_t dqp; uint32_t sqp; uint32_t status; - uint64_t context; - char mad[256]; + uint32_t length; + uint32_t context; + char mad[0]; }; enum SIM_CTL_TYPES { diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c index 4cbf8da..6f0fbad 100644 --- a/umad2sim/umad2sim.c +++ b/umad2sim/umad2sim.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2006,2007 Voltaire, Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This file is part of ibsim. * @@ -376,30 +377,46 @@ static int dev_sysfs_create(struct umad2sim_dev *dev) static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) { - struct sim_request req; + struct sim_request *req; ib_user_mad_t *umad = (ib_user_mad_t *) buf; unsigned mgmt_class; int cnt; DEBUG("umad2sim_read: %zu...\n", count); + req = malloc(sizeof(*req) + MAD_BLOCK_SIZE); + if (!req) { + ERROR("umad2sim_read: no mem: %m\n"); + return 0; /* ???? */ + } - cnt = real_read(dev->sim_client.fd_pktin, &req, sizeof(req)); + cnt = real_read(dev->sim_client.fd_pktin, req, + sizeof(*req) + MAD_BLOCK_SIZE); DEBUG("umad2sim_read: got %d...\n", cnt); - if (cnt < sizeof(req)) { - ERROR("umad2sim_read: partial request - skip.\n"); + if (cnt < sizeof(*req)) { + free(req); + ERROR("umad2sim_read: sim header incomplete - skip.\n"); + umad->status = EAGAIN; + return umad_size(); + } + + if (cnt != sizeof(*req) + req->length) { + free(req); + ERROR("umad2sim_read: partial request - got %d length " + "in sim header %d - skip.\n", + cnt, sizeof(*req) + req->length); umad->status = EAGAIN; return umad_size(); } - mgmt_class = mad_get_field(req.mad, 0, IB_MAD_MGMTCLASS_F); + mgmt_class = mad_get_field(req->mad, 0, IB_MAD_MGMTCLASS_F); DEBUG("umad2sim_read: mad: method=%x, response=%x, mgmtclass=%x, " "attrid=%x, attrmod=%x\n", - mad_get_field(req.mad, 0, IB_MAD_METHOD_F), - mad_get_field(req.mad, 0, IB_MAD_RESPONSE_F), + mad_get_field(req->mad, 0, IB_MAD_METHOD_F), + mad_get_field(req->mad, 0, IB_MAD_RESPONSE_F), mgmt_class, - mad_get_field(req.mad, 0, IB_MAD_ATTRID_F), - mad_get_field(req.mad, 0, IB_MAD_ATTRMOD_F) + mad_get_field(req->mad, 0, IB_MAD_ATTRID_F), + mad_get_field(req->mad, 0, IB_MAD_ATTRMOD_F) ); if (mgmt_class >= arrsize(dev->agent_idx)) { @@ -407,23 +424,24 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) mgmt_class = 0; } + cnt -= sizeof(*req); umad->agent_id = dev->agent_idx[mgmt_class]; - umad->status = ntohl(req.status); + umad->status = ntohl(req->status); umad->timeout_ms = 0; umad->retries = 0; - umad->length = umad_size() + sizeof(req.mad); + umad->length = umad_size() + cnt; - umad->addr.qpn = req.sqp; + umad->addr.qpn = req->sqp; umad->addr.qkey = 0; // agent->qkey; - umad->addr.lid = req.slid; + umad->addr.lid = req->slid; umad->addr.sl = 0; // agent->sl; umad->addr.path_bits = 0; umad->addr.grh_present = 0; - cnt -= sizeof(req) - sizeof(req.mad); if (cnt > count - umad_size()) cnt = count - umad_size(); - memcpy(umad_get_mad(umad), req.mad, cnt); + memcpy(umad_get_mad(umad), req->mad, cnt); + free(req); return cnt + umad_size(); } @@ -431,9 +449,9 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) static ssize_t umad2sim_write(struct umad2sim_dev *dev, const void *buf, size_t count) { - struct sim_request req; + struct sim_request *req; ib_user_mad_t *umad = (ib_user_mad_t *) buf; - int cnt; + int cnt, ocnt; #ifdef SIMULATE_SEND_ERRORS { static int err_count; @@ -464,25 +482,33 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, mad_get_field(umad_get_mad(umad), 0, IB_MAD_ATTRMOD_F) ); - req.dlid = umad->addr.lid; - req.slid = req.dlid == 0xffff ? 0xffff : 0; /* 0 - means auto + cnt = count - umad_size(); + if (cnt > MAD_BLOCK_SIZE) + cnt = MAD_BLOCK_SIZE; + req = malloc(sizeof(*req) + cnt); + if (!req) { + ERROR("umad2sim_write: no mem for sim req: %m"); + return -1; + } + + req->dlid = umad->addr.lid; + req->slid = req->dlid == 0xffff ? 0xffff : 0; /* 0 - means auto (supported by ibsim) */ ; - req.dqp = umad->addr.qpn; - req.sqp = htonl(dev->agents[umad->agent_id].qpn); - req.status = 0; - req.context = 0; + req->dqp = umad->addr.qpn; + req->sqp = htonl(dev->agents[umad->agent_id].qpn); + req->status = 0; + req->length = cnt; + req->context = 0; - cnt = count - umad_size(); - if (cnt > sizeof(req.mad)) - cnt = sizeof(req.mad); - memcpy(req.mad, umad_get_mad(umad), cnt); + memcpy(req->mad, umad_get_mad(umad), cnt); - cnt = write(dev->sim_client.fd_pktout, (void *)&req, sizeof(req)); - if (cnt < 0) { + ocnt = write(dev->sim_client.fd_pktout, req, sizeof(*req) + cnt); + free(req); + if (ocnt < 0) { ERROR("umad2sim_write: cannot write\n"); return -1; } - if (cnt < sizeof(req)) { + if (ocnt < sizeof(*req) + cnt) { ERROR("umad2sim_write: partial write\n"); } From hrosenstock at xsigo.com Thu Jun 19 07:35:44 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 19 Jun 2008 07:35:44 -0700 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> <1213623942.14047.634.camel@hrosenstock-ws.xsigo.com> Message-ID: <1213886144.13056.169.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 06:45 -0700, Hal Rosenstock wrote: > > > > > - umad->length = umad_size() + sizeof(req.mad); > > > > > + umad->length = umad_size() + cnt; > > > > > > > > 'cnt' is normally actual MAD length + sim_request header size. > Is it > > > > correct to put it as is in umad->length (shouldn't be umad_size > () + cnt > > > > - sizeof(struct sim_request))? > > > > > > Not sure about this. I'll need to get back on this. Yes; what you stated is correct. -- Hal From ogerlitz at voltaire.com Thu Jun 19 07:39:05 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:39:05 +0300 Subject: [ofa-general][PATCH v2 1/2]mlx4: Multiple completion vectors support In-Reply-To: References: <483C2966.1080606@mellanox.co.il> <48440414.4080909@mellanox.co.il> <4844E40A.9080209@mellanox.co.il> <48511089.8050609@voltaire.com> <48512C94.1040306@voltaire.com> Message-ID: <485A6F89.4070908@voltaire.com> Roland Dreier wrote: > Well, both of your statements seem to be true: nothing sets the affinity for the interrupts created for multiple EQs, and there is not any simply way to guarantee that CQ vector 5 is sent to CPU 5 that I see. I start to understand now.. so the suggested patch does not implement interrupt affinity for the different EQs. What's needed to have such affinity? should it be a feature of the HW? I still don't follow the second part of your reply, what's the difference from EQ to "CQ vector", I was thinking that its just two ways to describe the same thing. Looking on the change-log of the initial commit from last year, I see that interrupt affinity is mentioned. Or. > commit f4fd0b224d60044d2da5ca02f8f2b5150c1d8731 > Author: Michael S. Tsirkin > Date: Thu May 3 13:48:47 2007 +0300 > > IB: Add CQ comp_vector support > > Add a num_comp_vectors member to struct ib_device and extend > ib_create_cq() to pass in a comp_vector parameter -- this parallels > the userspace libibverbs API. Update all hardware drivers to set > num_comp_vectors to 1 and have all ULPs pass 0 for the comp_vector > value. Pass the value of num_comp_vectors to userspace rather than > hard-coding a value of 1. > > We want multiple CQ event vector support (via MSI-X or similar for > adapters that can generate multiple interrupts), but it's not clear > how many vectors we want, or how we want to deal with policy issues > such as how to decide which vector to use or how to set up interrupt > affinity. This patch is useful for experimenting, since no core > changes will be necessary when updating a driver to support multiple > vectors, and we know that we want to make at least these changes > anyway. > > Signed-off-by: Michael S. Tsirkin > Signed-off-by: Roland Dreier > From ogerlitz at voltaire.com Thu Jun 19 07:51:18 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 17:51:18 +0300 Subject: [ofa-general] [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <20080618155100.GC11489@mtls03> References: <1213628310.5447.91.camel@mtls03> <485929A9.6030306@voltaire.com> <20080618155100.GC11489@mtls03> Message-ID: <485A7266.6050802@voltaire.com> Eli Cohen wrote: > I don't know of any. I guess the mlx4 Ethernet driver has it too and it uses same ideas. > This RFC refers only to multiple queues in the receive flow and should > not have any impact on future multiple send queue implementation. It seems that other drivers implement multi queuing in the send flow, where you come and suggest to implement it in the receive flow. What's the architectural conception behind this? What problem your patches come to solve? and maybe you can comment why you think there's no other implementation? is it since other Ethernet HW vendors don't support multiple receive queues or can not generate interrupts to multiple interrupt vectors through MSI-X as the connectx does? Or. From rdreier at cisco.com Thu Jun 19 07:53:14 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 07:53:14 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Configure QP's max message size In-Reply-To: <20080619142802.GC15786@mtls03> (Eli Cohen's message of "Thu, 19 Jun 2008 17:28:02 +0300") References: <20080619142802.GC15786@mtls03> Message-ID: > Configure QP's max message size according to the value queried > from the HCA through QUERY_DEV_CAP. Not enough info for a changelog entry: why do this? What does it fix/improve? Is this an urgent fix? - R. From rdreier at cisco.com Thu Jun 19 07:53:55 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 07:53:55 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Check completion done when a command In-Reply-To: <20080619141520.GB15786@mtls03> (Eli Cohen's message of "Thu, 19 Jun 2008 17:15:20 +0300") References: <20080619141520.GB15786@mtls03> Message-ID: > we've been using this patch in ofed for a long time now. We have seen > cases where there are many processes running and it takes quite some > time before the task that executed the command gets CPU time. It seems > that the semantics of wait_for_completion_timeout has loose relation > to the actual time the event occurred -- however the done field itself > gives a more accurate indication as for whether the command was > completed or not. I seem to recall we had the same discussion about mthca and concluded that it was a bug that should be fixed in wait_for_completion_timeout(). From amirv at mellanox.co.il Thu Jun 19 07:57:55 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 19 Jun 2008 07:57:55 -0700 Subject: [ofa-general] [PATCH 2/4] CMA/RDMA: Add support for event IB_CM_TIMEWAIT_EXIT In-Reply-To: <12138874774119-git-send-email-amirv@mellanox.co.il> References: sdp_gracefull_close <12138874774119-git-send-email-amirv@mellanox.co.il> Message-ID: <12138874771286-git-send-email-amirv@mellanox.co.il> Notify the ULP when a IB_CM_TIMEWAIT_EXIT occurs. Signed-off-by: Amir Vadai --- .../fixes/cma_0050_timewait_finished_event.patch | 45 ++++++++++++++++++++ 1 files changed, 45 insertions(+), 0 deletions(-) create mode 100644 kernel_patches/fixes/cma_0050_timewait_finished_event.patch diff --git a/kernel_patches/fixes/cma_0050_timewait_finished_event.patch b/kernel_patches/fixes/cma_0050_timewait_finished_event.patch new file mode 100644 index 0000000..d987529 --- /dev/null +++ b/kernel_patches/fixes/cma_0050_timewait_finished_event.patch @@ -0,0 +1,45 @@ +SDP needs to be notified when CM exits the TIMEWAIT state. +CM does notify the CMA but CMA doesn't pass it to the upper layer. + +Below is a patch for the CMA code. I wasn't sure if it is ok to set +CMA_DISCONNECT in the IB_CM_TIMEWAIT_EXIT instead of doing it in the +IB_CM_DREQ_RECEIVED/IB_CM_DREP_RECEIVED as it is done today. Instead +I changed the check at the beginning of cma_ib_handler() as you can +see in the patch. + +Signed-off-by: Amir Vadai + +--- a/include/rdma/rdma_cm.h 2008-06-18 15:04:22.560904000 +0300 ++++ b/include/rdma/rdma_cm.h 2008-06-11 11:55:26.758053000 +0300 +@@ -53,7 +53,8 @@ enum rdma_cm_event_type { + RDMA_CM_EVENT_DISCONNECTED, + RDMA_CM_EVENT_DEVICE_REMOVAL, + RDMA_CM_EVENT_MULTICAST_JOIN, +- RDMA_CM_EVENT_MULTICAST_ERROR ++ RDMA_CM_EVENT_MULTICAST_ERROR, ++ RDMA_CM_EVENT_TIMWAIT_EXIT + }; + + enum rdma_port_space { +--- a/drivers/infiniband/core/cma.c 2008-06-11 11:24:38.021543000 +0300 ++++ b/drivers/infiniband/core/cma.c 2008-06-18 13:27:08.098747000 +0300 +@@ -931,7 +931,8 @@ static int cma_ib_handler(struct ib_cm_i + struct rdma_cm_event event; + int ret = 0; + +- if (cma_disable_remove(id_priv, CMA_CONNECT)) ++ if ((ib_event->event != IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_CONNECT)) || ++ (ib_event->event == IB_CM_TIMEWAIT_EXIT && cma_disable_remove(id_priv, CMA_DISCONNECT))) + return 0; + + memset(&event, 0, sizeof event); +@@ -967,6 +968,8 @@ static int cma_ib_handler(struct ib_cm_i + event.event = RDMA_CM_EVENT_DISCONNECTED; + break; + case IB_CM_TIMEWAIT_EXIT: ++ event.event = RDMA_CM_EVENT_TIMWAIT_EXIT; ++ break; + case IB_CM_MRA_RECEIVED: + /* ignore event */ + goto out; + -- 1.5.2 From amirv at mellanox.co.il Thu Jun 19 07:57:54 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 19 Jun 2008 07:57:54 -0700 Subject: [ofa-general] [PATCH 1/4] sdp: Use sdp_set_state() In-Reply-To: sdp_gracefull_close References: sdp_gracefull_close Message-ID: <12138874774119-git-send-email-amirv@mellanox.co.il> Fix all places that change sk_state directly to use sdp_set_state() Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 12 ++++++------ drivers/infiniband/ulp/sdp/sdp_bcopy.c | 4 ++-- drivers/infiniband/ulp/sdp/sdp_cma.c | 6 +++--- drivers/infiniband/ulp/sdp/sdp_main.c | 18 +++++++++--------- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index bb2434a..5bd4041 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -218,13 +218,18 @@ static inline struct sdp_sock *sdp_sk(const struct sock *sk) return (struct sdp_sock *)sk; } +static inline void sdp_set_state(struct sock *sk, int state) +{ + sk->sk_state = state; +} + static inline void sdp_set_error(struct sock *sk, int err) { sk->sk_err = -err; if (sk->sk_socket) sk->sk_socket->state = SS_UNCONNECTED; - sk->sk_state = TCP_CLOSE; + sdp_set_state(sk, TCP_CLOSE); if (sdp_sk(sk)->time_wait) { sdp_dbg(sk, "%s: destroy in time wait state\n", __func__); @@ -235,11 +240,6 @@ static inline void sdp_set_error(struct sock *sk, int err) sk->sk_error_report(sk); } -static inline void sdp_set_state(struct sock *sk, int state) -{ - sk->sk_state = state; -} - extern struct workqueue_struct *sdp_workqueue; int sdp_cma_handler(struct rdma_cm_id *, struct rdma_cm_event *); diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index be7c598..05825bf 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -544,9 +544,9 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) BUG_ON(!skb); sdp_post_send(ssk, skb, SDP_MID_DISCONN); if (ssk->isk.sk.sk_state == TCP_FIN_WAIT1) - ssk->isk.sk.sk_state = TCP_FIN_WAIT2; + sdp_set_state(&ssk->isk.sk, TCP_FIN_WAIT2); else - ssk->isk.sk.sk_state = TCP_CLOSING; + sdp_set_state(&ssk->isk.sk, TCP_CLOSING); } } diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index 2e4ceb5..f2fb083 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -256,7 +256,7 @@ int sdp_connect_handler(struct sock *sk, struct rdma_cm_id *id, list_add_tail(&sdp_sk(child)->backlog_queue, &sdp_sk(sk)->backlog_queue); sdp_sk(child)->parent = sk; - child->sk_state = TCP_SYN_RECV; + sdp_set_state(child, TCP_SYN_RECV); /* child->sk_write_space(child); */ /* child->sk_data_ready(child, 0); */ @@ -272,7 +272,7 @@ static int sdp_response_handler(struct sock *sk, struct rdma_cm_id *id, struct sockaddr_in *dst_addr; sdp_dbg(sk, "%s\n", __func__); - sk->sk_state = TCP_ESTABLISHED; + sdp_set_state(sk, TCP_ESTABLISHED); if (sock_flag(sk, SOCK_KEEPOPEN)) sdp_start_keepalive_timer(sk); @@ -316,7 +316,7 @@ int sdp_connected_handler(struct sock *sk, struct rdma_cm_event *event) parent = sdp_sk(sk)->parent; BUG_ON(!parent); - sk->sk_state = TCP_ESTABLISHED; + sdp_set_state(sk, TCP_ESTABLISHED); if (sock_flag(sk, SOCK_KEEPOPEN)) sdp_start_keepalive_timer(sk); diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index 8f1e8b0..acd3aab 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -452,9 +452,9 @@ static int sdp_close_state(struct sock *sk) return 0; if (sk->sk_state == TCP_ESTABLISHED) - sk->sk_state = TCP_FIN_WAIT1; + sdp_set_state(sk, TCP_FIN_WAIT1); else if (sk->sk_state == TCP_CLOSE_WAIT) - sk->sk_state = TCP_LAST_ACK; + sdp_set_state(sk, TCP_LAST_ACK); else return 0; return 1; @@ -553,7 +553,7 @@ adjudge_to_death: if (sk->sk_state == TCP_FIN_WAIT2 && !sk->sk_send_head && sdp_sk(sk)->tx_head == sdp_sk(sk)->tx_tail) { - sk->sk_state = TCP_CLOSE; + sdp_set_state(sk, TCP_CLOSE); } if ((1 << sk->sk_state) & (TCPF_FIN_WAIT1 | TCPF_FIN_WAIT2)) { @@ -622,7 +622,7 @@ static int sdp_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len) return rc; } - sk->sk_state = TCP_SYN_SENT; + sdp_set_state(sk, TCP_SYN_SENT); return 0; } @@ -853,7 +853,7 @@ void sdp_time_wait_work(struct work_struct *work) sdp_dbg(sk, "%s: refcnt %d\n", __func__, atomic_read(&sk->sk_refcnt)); - sk->sk_state = TCP_CLOSE; + sdp_set_state(sk, TCP_CLOSE); sdp_sk(sk)->time_wait = 0; release_sock(sk); @@ -864,7 +864,7 @@ void sdp_time_wait_work(struct work_struct *work) void sdp_time_wait_destroy_sk(struct sdp_sock *ssk) { ssk->time_wait = 0; - ssk->isk.sk.sk_state = TCP_CLOSE; + sdp_set_state(&ssk->isk.sk, TCP_CLOSE); queue_work(sdp_workqueue, &ssk->destroy_work); } @@ -895,9 +895,9 @@ static void sdp_shutdown(struct sock *sk, int how) return; if (sk->sk_state == TCP_ESTABLISHED) - sk->sk_state = TCP_FIN_WAIT1; + sdp_set_state(sk, TCP_FIN_WAIT1); else if (sk->sk_state == TCP_CLOSE_WAIT) - sk->sk_state = TCP_LAST_ACK; + sdp_set_state(sk, TCP_LAST_ACK); else return; @@ -1991,7 +1991,7 @@ static int sdp_listen(struct sock *sk, int backlog) sdp_warn(sk, "rdma_listen failed: %d\n", rc); sdp_set_error(sk, rc); } else - sk->sk_state = TCP_LISTEN; + sdp_set_state(sk, TCP_LISTEN); return rc; } -- 1.5.2 From amirv at mellanox.co.il Thu Jun 19 07:57:56 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 19 Jun 2008 07:57:56 -0700 Subject: [ofa-general] [PATCH 3/4] sdp: Split sdp_handle_wc() to smaller functions In-Reply-To: <12138874771286-git-send-email-amirv@mellanox.co.il> References: sdp_gracefull_close <12138874774119-git-send-email-amirv@mellanox.co.il> <12138874771286-git-send-email-amirv@mellanox.co.il> Message-ID: <1213887477421-git-send-email-amirv@mellanox.co.il> Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 209 ++++++++++++++++++-------------- 1 files changed, 116 insertions(+), 93 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 05825bf..3a8d5ac 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -588,112 +588,135 @@ static void sdp_handle_resize_ack(struct sdp_sock *ssk, struct sdp_chrecvbuf *bu ssk->sent_request = 0; } -static void sdp_handle_wc(struct sdp_sock *ssk, struct ib_wc *wc) +static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) { + int frags; struct sk_buff *skb; struct sdp_bsdh *h; int pagesz, i; - if (wc->wr_id & SDP_OP_RECV) { - skb = sdp_recv_completion(ssk, wc->wr_id); - if (unlikely(!skb)) - return; + skb = sdp_recv_completion(ssk, wc->wr_id); + if (unlikely(!skb)) + return -1; - atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); + atomic_sub(SDP_MAX_SEND_SKB_FRAGS, &sdp_current_mem_usage); - if (unlikely(wc->status)) { - if (wc->status != IB_WC_WR_FLUSH_ERR) { - sdp_dbg(&ssk->isk.sk, - "Recv completion with error. " - "Status %d\n", wc->status); - sdp_reset(&ssk->isk.sk); - } - __kfree_skb(skb); - } else { - int frags; - - sdp_dbg_data(&ssk->isk.sk, - "Recv completion. ID %d Length %d\n", - (int)wc->wr_id, wc->byte_len); - if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { - printk("SDP BUG! byte_len %d < %zd\n", - wc->byte_len, sizeof(struct sdp_bsdh)); - __kfree_skb(skb); - return; - } - skb->len = wc->byte_len; - if (likely(wc->byte_len > SDP_HEAD_SIZE)) - skb->data_len = wc->byte_len - SDP_HEAD_SIZE; - else - skb->data_len = 0; - skb->data = skb->head; + if (unlikely(wc->status)) { + if (wc->status != IB_WC_WR_FLUSH_ERR) { + sdp_dbg(&ssk->isk.sk, + "Recv completion with error. " + "Status %d\n", wc->status); + sdp_reset(&ssk->isk.sk); + } + __kfree_skb(skb); + return 0; + } + + sdp_dbg_data(&ssk->isk.sk, "Recv completion. ID %d Length %d\n", + (int)wc->wr_id, wc->byte_len); + if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { + printk(KERN_WARNING "SDP BUG! byte_len %d < %zd\n", + wc->byte_len, sizeof(struct sdp_bsdh)); + __kfree_skb(skb); + return -1; + } + skb->len = wc->byte_len; + if (likely(wc->byte_len > SDP_HEAD_SIZE)) + skb->data_len = wc->byte_len - SDP_HEAD_SIZE; + else + skb->data_len = 0; + skb->data = skb->head; #ifdef NET_SKBUFF_DATA_USES_OFFSET - skb->tail = skb_headlen(skb); + skb->tail = skb_headlen(skb); #else - skb->tail = skb->head + skb_headlen(skb); + skb->tail = skb->head + skb_headlen(skb); #endif - h = (struct sdp_bsdh *)skb->data; - skb_reset_transport_header(skb); - ssk->mseq_ack = ntohl(h->mseq); - if (ssk->mseq_ack != (int)wc->wr_id) - printk("SDP BUG! mseq %d != wrid %d\n", - ssk->mseq_ack, (int)wc->wr_id); - ssk->bufs = ntohl(h->mseq_ack) - ssk->tx_head + 1 + - ntohs(h->bufs); - - frags = skb_shinfo(skb)->nr_frags; - pagesz = PAGE_ALIGN(skb->data_len); - skb_shinfo(skb)->nr_frags = pagesz / PAGE_SIZE; - - for (i = skb_shinfo(skb)->nr_frags; - i < frags; ++i) { - put_page(skb_shinfo(skb)->frags[i].page); - skb->truesize -= PAGE_SIZE; - } - - if (unlikely(h->flags & SDP_OOB_PEND)) - sk_send_sigurg(&ssk->isk.sk); - - skb_pull(skb, sizeof(struct sdp_bsdh)); - - if (likely(h->mid == SDP_MID_DATA) && - likely(skb->len > 0)) { - int oob = h->flags & SDP_OOB_PRES; - skb = sdp_sock_queue_rcv_skb(&ssk->isk.sk, skb); - if (unlikely(oob)) - sdp_urg(ssk, skb); - } else if (likely(h->mid == SDP_MID_DATA)) { - __kfree_skb(skb); - } else if (h->mid == SDP_MID_DISCONN) { - /* this will wake recvmsg */ - sdp_sock_queue_rcv_skb(&ssk->isk.sk, skb); - sdp_fin(&ssk->isk.sk); - } else if (h->mid == SDP_MID_CHRCVBUF) { - sdp_handle_resize_request(ssk, (struct sdp_chrecvbuf *)skb->data); - __kfree_skb(skb); - } else if (h->mid == SDP_MID_CHRCVBUF_ACK) { - sdp_handle_resize_ack(ssk, (struct sdp_chrecvbuf *)skb->data); - __kfree_skb(skb); - } else { - /* TODO: Handle other messages */ - printk("SDP: FIXME MID %d\n", h->mid); - __kfree_skb(skb); - } + h = (struct sdp_bsdh *)skb->data; + skb_reset_transport_header(skb); + ssk->mseq_ack = ntohl(h->mseq); + if (ssk->mseq_ack != (int)wc->wr_id) + printk(KERN_WARNING "SDP BUG! mseq %d != wrid %d\n", + ssk->mseq_ack, (int)wc->wr_id); + ssk->bufs = ntohl(h->mseq_ack) - ssk->tx_head + 1 + + ntohs(h->bufs); + + frags = skb_shinfo(skb)->nr_frags; + pagesz = PAGE_ALIGN(skb->data_len); + skb_shinfo(skb)->nr_frags = pagesz / PAGE_SIZE; + + for (i = skb_shinfo(skb)->nr_frags; + i < frags; ++i) { + put_page(skb_shinfo(skb)->frags[i].page); + skb->truesize -= PAGE_SIZE; + } + + if (unlikely(h->flags & SDP_OOB_PEND)) + sk_send_sigurg(&ssk->isk.sk); + + skb_pull(skb, sizeof(struct sdp_bsdh)); + + switch (h->mid) { + case SDP_MID_DATA: + if (unlikely(skb->len <= 0)) { + __kfree_skb(skb); + break; } + skb = sdp_sock_queue_rcv_skb(&ssk->isk.sk, skb); + if (unlikely(h->flags & SDP_OOB_PRES)) + sdp_urg(ssk, skb); + break; + case SDP_MID_DISCONN: + /* this will wake recvmsg */ + sdp_sock_queue_rcv_skb(&ssk->isk.sk, skb); + sdp_fin(&ssk->isk.sk); + break; + case SDP_MID_CHRCVBUF: + sdp_handle_resize_request(ssk, + (struct sdp_chrecvbuf *)skb->data); + __kfree_skb(skb); + break; + + case SDP_MID_CHRCVBUF_ACK: + sdp_handle_resize_ack(ssk, (struct sdp_chrecvbuf *)skb->data); + __kfree_skb(skb); + break; + default: + /* TODO: Handle other messages */ + printk(KERN_WARNING "SDP: FIXME MID %d\n", h->mid); + __kfree_skb(skb); + } + return 0; +} + +static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) +{ + struct sk_buff *skb; + + skb = sdp_send_completion(ssk, wc->wr_id); + if (unlikely(!skb)) + return -1; + sk_wmem_free_skb(&ssk->isk.sk, skb); + if (unlikely(wc->status)) { + if (wc->status != IB_WC_WR_FLUSH_ERR) { + sdp_dbg(&ssk->isk.sk, + "Send completion with error. " + "Status %d\n", wc->status); + sdp_set_error(&ssk->isk.sk, -ECONNRESET); + wake_up(&ssk->wq); + } + } + + return 0; +} + +static void sdp_handle_wc(struct sdp_sock *ssk, struct ib_wc *wc) +{ + if (wc->wr_id & SDP_OP_RECV) { + if (sdp_handle_recv_comp(ssk, wc)) + return; } else if (likely(wc->wr_id & SDP_OP_SEND)) { - skb = sdp_send_completion(ssk, wc->wr_id); - if (unlikely(!skb)) + if (sdp_handle_send_comp(ssk, wc)) return; - sk_wmem_free_skb(&ssk->isk.sk, skb); - if (unlikely(wc->status)) { - if (wc->status != IB_WC_WR_FLUSH_ERR) { - sdp_dbg(&ssk->isk.sk, - "Send completion with error. " - "Status %d\n", wc->status); - sdp_set_error(&ssk->isk.sk, -ECONNRESET); - wake_up(&ssk->wq); - } - } } else { sdp_cnt(sdp_keepalive_probes_sent); -- 1.5.2 From amirv at mellanox.co.il Thu Jun 19 07:57:57 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 19 Jun 2008 07:57:57 -0700 Subject: [ofa-general] [PATCH 4/4] sdp: do gracefull close instead of always doing abortive close. In-Reply-To: <1213887477421-git-send-email-amirv@mellanox.co.il> References: sdp_gracefull_close <12138874774119-git-send-email-amirv@mellanox.co.il> <12138874771286-git-send-email-amirv@mellanox.co.il> <1213887477421-git-send-email-amirv@mellanox.co.il> Message-ID: <12138874773153-git-send-email-amirv@mellanox.co.il> Main changes: 1. when a close/shutdown syscall is called, instead of sending a DREQ, put last socket ref count and go to TCP_CLOSE state do: - take a socket reference count - set state to TCP_TIME_WAIT - start infiniband tear down - wait till got RDMA_CM_EVENT_TIMEWAIT_EXIT - set socket state to TCP_CLOSE - put last socket ref count - this will call sdp_destruct() 2. Use fin_wait_timeout to timeout a half closed connection which the peer didn't respond with SDP_MID_DISCONNECT 3. No need for sdp_time_wait 4. Abortive close will immedietly start infiniband teardown - will finilize the socket closing when CM finish. 5. when sdp_post_sends send SDP_MID_DISCONNECT according to socket state multiple DISCONNECT could be sent - changed it to be triggered by a flag. Signed-off-by: Amir Vadai --- drivers/infiniband/ulp/sdp/sdp.h | 20 +++--- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 137 +++++++++++++++++++++++++------- drivers/infiniband/ulp/sdp/sdp_cma.c | 10 ++- drivers/infiniband/ulp/sdp/sdp_main.c | 97 +++++++++++++---------- 4 files changed, 180 insertions(+), 84 deletions(-) diff --git a/drivers/infiniband/ulp/sdp/sdp.h b/drivers/infiniband/ulp/sdp/sdp.h index 5bd4041..3b2dffb 100644 --- a/drivers/infiniband/ulp/sdp/sdp.h +++ b/drivers/infiniband/ulp/sdp/sdp.h @@ -125,7 +125,7 @@ struct sdp_sock { int xmit_size_goal; int nonagle; - int time_wait; + int fin_wait_timeout; unsigned keepalive_time; @@ -133,6 +133,8 @@ struct sdp_sock { unsigned keepalive_tx_head; unsigned keepalive_rx_head; + int sdp_disconnect; + /* Data below will be reset on error */ /* rdma specific */ struct rdma_cm_id *id; @@ -227,15 +229,12 @@ static inline void sdp_set_error(struct sock *sk, int err) { sk->sk_err = -err; if (sk->sk_socket) - sk->sk_socket->state = SS_UNCONNECTED; - - sdp_set_state(sk, TCP_CLOSE); + sk->sk_socket->state = SS_DISCONNECTING; - if (sdp_sk(sk)->time_wait) { - sdp_dbg(sk, "%s: destroy in time wait state\n", __func__); - sdp_sk(sk)->time_wait = 0; - queue_work(sdp_workqueue, &sdp_sk(sk)->destroy_work); - } + if (sk->sk_state == TCP_SYN_SENT) + sdp_set_state(sk, TCP_CLOSE); + else + sdp_set_state(sk, TCP_TIME_WAIT); sk->sk_error_report(sk); } @@ -245,7 +244,6 @@ extern struct workqueue_struct *sdp_workqueue; int sdp_cma_handler(struct rdma_cm_id *, struct rdma_cm_event *); void sdp_reset(struct sock *sk); void sdp_reset_sk(struct sock *sk, int rc); -void sdp_time_wait_destroy_sk(struct sdp_sock *ssk); void sdp_completion_handler(struct ib_cq *cq, void *cq_context); void sdp_work(struct work_struct *work); int sdp_post_credits(struct sdp_sock *ssk); @@ -254,6 +252,8 @@ void sdp_post_recvs(struct sdp_sock *ssk); int sdp_poll_cq(struct sdp_sock *ssk, struct ib_cq *cq); void sdp_post_sends(struct sdp_sock *ssk, int nonagle); void sdp_destroy_work(struct work_struct *work); +void sdp_cancel_fin_wait_timeout(struct sdp_sock *ssk); +void sdp_fin_work(struct work_struct *work); void sdp_time_wait_work(struct work_struct *work); struct sk_buff *sdp_recv_completion(struct sdp_sock *ssk, int id); struct sk_buff *sdp_send_completion(struct sdp_sock *ssk, int mseq); diff --git a/drivers/infiniband/ulp/sdp/sdp_bcopy.c b/drivers/infiniband/ulp/sdp/sdp_bcopy.c index 3a8d5ac..d3f50b5 100644 --- a/drivers/infiniband/ulp/sdp/sdp_bcopy.c +++ b/drivers/infiniband/ulp/sdp/sdp_bcopy.c @@ -96,7 +96,7 @@ void sdp_remove_large_sock(struct sdp_sock *ssk) } } -/* Like tcp_fin */ +/* Like tcp_fin - called when SDP_MID_DISCONNECT is received */ static void sdp_fin(struct sock *sk) { sdp_dbg(sk, "%s\n", __func__); @@ -104,6 +104,42 @@ static void sdp_fin(struct sock *sk) sk->sk_shutdown |= RCV_SHUTDOWN; sock_set_flag(sk, SOCK_DONE); + switch (sk->sk_state) { + case TCP_SYN_RECV: + case TCP_ESTABLISHED: + sdp_set_state(sk, TCP_CLOSE_WAIT); + break; + + case TCP_FIN_WAIT1: + /* This case occurs when a simultaneous close + * happens, we must ack the received FIN and + * enter the CLOSING state. + */ + sdp_set_state(sk, TCP_CLOSING); + break; + + case TCP_FIN_WAIT2: + /* Received a reply FIN - start Infiniband tear down */ + sdp_set_state(sk, TCP_TIME_WAIT); + sdp_dbg(sk, "%s: Starting Infiniband tear down sending DREQ\n", + __func__); + + if (sdp_sk(sk)->id) { + rdma_disconnect(sdp_sk(sk)->id); + } else { + sdp_warn(sk, "%s: sdp_sk(sk)->id is NULL\n", __func__); + BUG(); + } + break; + case TCP_TIME_WAIT: + case TCP_CLOSE: + break; + default: + sdp_warn(sk, "%s: FIN in unexpected state. sk->sk_state=%d\n", + __func__, sk->sk_state); + break; + } + sk_mem_reclaim(sk); @@ -524,7 +560,9 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) if (unlikely(c < ssk->rx_head - ssk->rx_tail) && likely(ssk->bufs > 1) && - likely(ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE)) { + likely(ssk->tx_head - ssk->tx_tail < SDP_TX_SIZE) && + likely((1 << ssk->isk.sk.sk_state) & + (TCPF_ESTABLISHED | TCPF_FIN_WAIT1 | TCPF_FIN_WAIT2))) { skb = sdp_stream_alloc_skb(&ssk->isk.sk, sizeof(struct sdp_bsdh), GFP_KERNEL); @@ -533,20 +571,16 @@ void sdp_post_sends(struct sdp_sock *ssk, int nonagle) sdp_post_send(ssk, skb, SDP_MID_DATA); } - if (unlikely((1 << ssk->isk.sk.sk_state) & - (TCPF_FIN_WAIT1 | TCPF_LAST_ACK)) && + if (unlikely(ssk->sdp_disconnect) && !ssk->isk.sk.sk_send_head && ssk->bufs > (ssk->remote_credits >= ssk->rx_head - ssk->rx_tail)) { + ssk->sdp_disconnect = 0; skb = sdp_stream_alloc_skb(&ssk->isk.sk, sizeof(struct sdp_bsdh), gfp_page); /* FIXME */ BUG_ON(!skb); sdp_post_send(ssk, skb, SDP_MID_DISCONN); - if (ssk->isk.sk.sk_state == TCP_FIN_WAIT1) - sdp_set_state(&ssk->isk.sk, TCP_FIN_WAIT2); - else - sdp_set_state(&ssk->isk.sk, TCP_CLOSING); } } @@ -590,6 +624,7 @@ static void sdp_handle_resize_ack(struct sdp_sock *ssk, struct sdp_chrecvbuf *bu static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) { + struct sock *sk = &ssk->isk.sk; int frags; struct sk_buff *skb; struct sdp_bsdh *h; @@ -603,16 +638,15 @@ static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) if (unlikely(wc->status)) { if (wc->status != IB_WC_WR_FLUSH_ERR) { - sdp_dbg(&ssk->isk.sk, - "Recv completion with error. " - "Status %d\n", wc->status); - sdp_reset(&ssk->isk.sk); + sdp_dbg(sk, "Recv completion with error. Status %d\n", + wc->status); + sdp_reset(sk); } __kfree_skb(skb); return 0; } - sdp_dbg_data(&ssk->isk.sk, "Recv completion. ID %d Length %d\n", + sdp_dbg_data(sk, "Recv completion. ID %d Length %d\n", (int)wc->wr_id, wc->byte_len); if (unlikely(wc->byte_len < sizeof(struct sdp_bsdh))) { printk(KERN_WARNING "SDP BUG! byte_len %d < %zd\n", @@ -651,7 +685,7 @@ static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) } if (unlikely(h->flags & SDP_OOB_PEND)) - sk_send_sigurg(&ssk->isk.sk); + sk_send_sigurg(sk); skb_pull(skb, sizeof(struct sdp_bsdh)); @@ -661,21 +695,36 @@ static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) __kfree_skb(skb); break; } - skb = sdp_sock_queue_rcv_skb(&ssk->isk.sk, skb); + + if (unlikely(sk->sk_shutdown & RCV_SHUTDOWN)) { + /* got data in RCV_SHUTDOWN */ + if ((1 << sk->sk_state) & + (TCPF_FIN_WAIT1 | TCPF_FIN_WAIT2)) { + /* go into abortive close */ + sdp_set_state(sk, TCP_TIME_WAIT); + + sk->sk_prot->disconnect(sk, 0); + } + + __kfree_skb(skb); + break; + } + skb = sdp_sock_queue_rcv_skb(sk, skb); if (unlikely(h->flags & SDP_OOB_PRES)) sdp_urg(ssk, skb); break; case SDP_MID_DISCONN: - /* this will wake recvmsg */ - sdp_sock_queue_rcv_skb(&ssk->isk.sk, skb); - sdp_fin(&ssk->isk.sk); + if (ssk->fin_wait_timeout) + sdp_cancel_fin_wait_timeout(ssk); + + __kfree_skb(skb); + sdp_fin(sk); break; case SDP_MID_CHRCVBUF: sdp_handle_resize_request(ssk, - (struct sdp_chrecvbuf *)skb->data); + (struct sdp_chrecvbuf *)skb->data); __kfree_skb(skb); break; - case SDP_MID_CHRCVBUF_ACK: sdp_handle_resize_ack(ssk, (struct sdp_chrecvbuf *)skb->data); __kfree_skb(skb); @@ -685,27 +734,62 @@ static int sdp_handle_recv_comp(struct sdp_sock *ssk, struct ib_wc *wc) printk(KERN_WARNING "SDP: FIXME MID %d\n", h->mid); __kfree_skb(skb); } + return 0; } static int sdp_handle_send_comp(struct sdp_sock *ssk, struct ib_wc *wc) { struct sk_buff *skb; + struct sdp_bsdh *h; skb = sdp_send_completion(ssk, wc->wr_id); if (unlikely(!skb)) return -1; - sk_wmem_free_skb(&ssk->isk.sk, skb); + if (unlikely(wc->status)) { if (wc->status != IB_WC_WR_FLUSH_ERR) { - sdp_dbg(&ssk->isk.sk, - "Send completion with error. " + sdp_dbg(&ssk->isk.sk, "Send completion with error. " "Status %d\n", wc->status); sdp_set_error(&ssk->isk.sk, -ECONNRESET); wake_up(&ssk->wq); } + goto out; } + h = (struct sdp_bsdh *)skb->data; + + if (likely(h->mid != SDP_MID_DISCONN)) + goto out; + + switch (ssk->isk.sk.sk_state) { + case TCP_FIN_WAIT1: + /* sdp_set_state(&ssk->isk.sk, + TCP_FIN_WAIT2); */ + break; + case TCP_CLOSING: + case TCP_LAST_ACK: + sdp_set_state(&ssk->isk.sk, TCP_TIME_WAIT); + + sdp_dbg(&ssk->isk.sk, + "%s: waiting for Infiniband tear down\n", __func__); + + /* We will not issue a DREQ here - because DREQ is sent on + the CM QP - and could reach the peer before the + SDP_MID_DISCONNECT This will put the peer in abortive close + state - without any real reason. + TODO: maybe a timeout whould be used here - what if the + peer won't send a DREQ? */ + break; + default: + sdp_dbg(&ssk->isk.sk, + "%s: sent DISCONNECT from unexpected state %d\n", + __func__, ssk->isk.sk.sk_state); + } + +out: + sk_wmem_free_skb(&ssk->isk.sk, skb); + return 0; } @@ -734,13 +818,6 @@ static void sdp_handle_wc(struct sdp_sock *ssk, struct ib_wc *wc) return; } - - if (ssk->time_wait && !ssk->isk.sk.sk_send_head && - ssk->tx_head == ssk->tx_tail) { - sdp_dbg(&ssk->isk.sk, "%s: destroy in time wait state\n", - __func__); - sdp_time_wait_destroy_sk(ssk); - } } void sdp_completion_handler(struct ib_cq *cq, void *cq_context) diff --git a/drivers/infiniband/ulp/sdp/sdp_cma.c b/drivers/infiniband/ulp/sdp/sdp_cma.c index f2fb083..8759bac 100644 --- a/drivers/infiniband/ulp/sdp/sdp_cma.c +++ b/drivers/infiniband/ulp/sdp/sdp_cma.c @@ -498,9 +498,17 @@ int sdp_cma_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) ((struct sockaddr_in *)&id->route.addr.src_addr)->sin_addr.s_addr; rc = sdp_connected_handler(sk, event); break; - case RDMA_CM_EVENT_DISCONNECTED: + case RDMA_CM_EVENT_DISCONNECTED: /* This means DREQ/DREP received */ sdp_dbg(sk, "RDMA_CM_EVENT_DISCONNECTED\n"); rdma_disconnect(id); + + if (sk->sk_state != TCP_TIME_WAIT) { + sdp_set_error(sk, EPIPE); + rc = sdp_disconnected_handler(sk); + } + break; + case RDMA_CM_EVENT_TIMWAIT_EXIT: + sdp_dbg(sk, "RDMA_CM_EVENT_TIMEWAIT_EXIT\n"); rc = sdp_disconnected_handler(sk); break; case RDMA_CM_EVENT_DEVICE_REMOVAL: diff --git a/drivers/infiniband/ulp/sdp/sdp_main.c b/drivers/infiniband/ulp/sdp/sdp_main.c index acd3aab..960ec90 100644 --- a/drivers/infiniband/ulp/sdp/sdp_main.c +++ b/drivers/infiniband/ulp/sdp/sdp_main.c @@ -342,10 +342,7 @@ void sdp_reset_sk(struct sock *sk, int rc) memset((void *)&ssk->id, 0, sizeof(*ssk) - offsetof(typeof(*ssk), id)); - if (ssk->time_wait) { - sdp_dbg(sk, "%s: destroy in time wait state\n", __func__); - sdp_time_wait_destroy_sk(ssk); - } + queue_work(sdp_workqueue, &ssk->destroy_work); sk->sk_state_change(sk); @@ -436,9 +433,10 @@ done: sdp_dbg(sk, "%s done\n", __func__); } -static void sdp_send_active_reset(struct sock *sk, gfp_t priority) +static void sdp_send_disconnect(struct sock *sk) { - sk->sk_prot->disconnect(sk, 0); + sdp_sk(sk)->sdp_disconnect = 1; + sdp_post_sends(sdp_sk(sk), 0); } /* @@ -452,7 +450,7 @@ static int sdp_close_state(struct sock *sk) return 0; if (sk->sk_state == TCP_ESTABLISHED) - sdp_set_state(sk, TCP_FIN_WAIT1); + sdp_set_state(sk, TCP_FIN_WAIT2);/* should be TCP_FIN_WAIT1 */ else if (sk->sk_state == TCP_CLOSE_WAIT) sdp_set_state(sk, TCP_LAST_ACK); else @@ -466,7 +464,13 @@ static void sdp_close(struct sock *sk, long timeout) struct sk_buff *skb; int data_was_unread = 0; + if ((1 << sk->sk_state) & (TCPF_TIME_WAIT | TCPF_CLOSE)) { + sock_put(sk); + return; + } + lock_sock(sk); + sock_hold(sk); sdp_dbg(sk, "%s\n", __func__); @@ -504,18 +508,23 @@ static void sdp_close(struct sock *sk, long timeout) if (data_was_unread) { /* Unread data was tossed, zap the connection. */ NET_INC_STATS_USER(LINUX_MIB_TCPABORTONCLOSE); - sdp_set_state(sk, TCP_CLOSE); - sdp_send_active_reset(sk, GFP_KERNEL); + sdp_set_state(sk, TCP_TIME_WAIT); + + /* Go into abortive close */ + sk->sk_prot->disconnect(sk, 0); } else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) { /* Check zero linger _after_ checking for unread data. */ - sk->sk_prot->disconnect(sk, 0); NET_INC_STATS_USER(LINUX_MIB_TCPABORTONDATA); + sdp_set_state(sk, TCP_TIME_WAIT); + + /* Go into abortive close */ + sk->sk_prot->disconnect(sk, 0); } else if (sdp_close_state(sk)) { /* We FIN if the application ate all the data before * zapping the connection. */ - sdp_post_sends(sdp_sk(sk), 0); + sdp_send_disconnect(sk); } /* TODO: state should move to CLOSE or CLOSE_WAIT etc on disconnect. @@ -533,7 +542,6 @@ adjudge_to_death: */ lock_sock(sk); - sock_hold(sk); sock_orphan(sk); /* This is a (useful) BSD violating of the RFC. There is a @@ -549,15 +557,8 @@ adjudge_to_death: * consume significant resources. Let's do it with special * linger2 option. --ANK */ - - if (sk->sk_state == TCP_FIN_WAIT2 && - !sk->sk_send_head && - sdp_sk(sk)->tx_head == sdp_sk(sk)->tx_tail) { - sdp_set_state(sk, TCP_CLOSE); - } - if ((1 << sk->sk_state) & (TCPF_FIN_WAIT1 | TCPF_FIN_WAIT2)) { - sdp_sk(sk)->time_wait = 1; + sdp_sk(sk)->fin_wait_timeout = 1; /* TODO: liger2 unimplemented. We should wait 3.5 * rto. How do I know rto? */ /* TODO: tcp_fin_time to get timeout */ @@ -566,18 +567,20 @@ adjudge_to_death: atomic_inc(sk->sk_prot->orphan_count); queue_delayed_work(sdp_workqueue, &sdp_sk(sk)->time_wait_work, TCP_FIN_TIMEOUT); - goto out; } /* TODO: limit number of orphaned sockets. TCP has sysctl_tcp_mem and sysctl_tcp_max_orphans */ - sock_put(sk); - /* Otherwise, socket is reprieved until protocol close. */ -out: - sdp_dbg(sk, "%s: last socket put %d\n", __func__, - atomic_read(&sk->sk_refcnt)); + if (sk->sk_state != TCP_CLOSE) { + sdp_dbg(sk, "Hold socket till end of Infiniband tear-down\n"); + /* Hold socket till end of Infiniband tear-down */ + sock_hold(sk); + } + release_sock(sk); + + sock_put(sk); sk_common_release(sk); } @@ -635,11 +638,13 @@ static int sdp_disconnect(struct sock *sk, int flags) struct rdma_cm_id *id; sdp_dbg(sk, "%s\n", __func__); - if (ssk->id) - rc = rdma_disconnect(ssk->id); - if (old_state != TCP_LISTEN) + if (old_state != TCP_LISTEN) { + if (ssk->id) + rc = rdma_disconnect(ssk->id); + return rc; + } sdp_set_state(sk, TCP_CLOSE); id = ssk->id; @@ -827,14 +832,23 @@ static int sdp_ioctl(struct sock *sk, int cmd, unsigned long arg) return put_user(answ, (int __user *)arg); } +void sdp_cancel_fin_wait_timeout(struct sdp_sock *ssk) +{ + ssk->fin_wait_timeout = 0; + cancel_delayed_work(&ssk->time_wait_work); + atomic_dec(ssk->isk.sk.sk_prot->orphan_count); +} + void sdp_destroy_work(struct work_struct *work) { struct sdp_sock *ssk = container_of(work, struct sdp_sock, destroy_work); struct sock *sk = &ssk->isk.sk; sdp_dbg(sk, "%s: refcnt %d\n", __func__, atomic_read(&sk->sk_refcnt)); - cancel_delayed_work(&sdp_sk(sk)->time_wait_work); - atomic_dec(sk->sk_prot->orphan_count); + if (ssk->fin_wait_timeout) + sdp_cancel_fin_wait_timeout(ssk); + + sdp_set_state(sk, TCP_CLOSE); sock_put(sk); } @@ -846,26 +860,20 @@ void sdp_time_wait_work(struct work_struct *work) lock_sock(sk); sdp_dbg(sk, "%s\n", __func__); - if (!sdp_sk(sk)->time_wait) { + if (!sdp_sk(sk)->fin_wait_timeout) { release_sock(sk); return; } sdp_dbg(sk, "%s: refcnt %d\n", __func__, atomic_read(&sk->sk_refcnt)); - sdp_set_state(sk, TCP_CLOSE); - sdp_sk(sk)->time_wait = 0; + sdp_sk(sk)->fin_wait_timeout = 0; release_sock(sk); atomic_dec(sk->sk_prot->orphan_count); - sock_put(sk); -} -void sdp_time_wait_destroy_sk(struct sdp_sock *ssk) -{ - ssk->time_wait = 0; - sdp_set_state(&ssk->isk.sk, TCP_CLOSE); - queue_work(sdp_workqueue, &ssk->destroy_work); + if (sdp_sk(sk)->id) + rdma_disconnect(sdp_sk(sk)->id); } static int sdp_init_sock(struct sock *sk) @@ -880,6 +888,9 @@ static int sdp_init_sock(struct sock *sk) INIT_WORK(&ssk->destroy_work, sdp_destroy_work); sk->sk_route_caps |= NETIF_F_SG | NETIF_F_NO_CSUM; + + ssk->sdp_disconnect = 0; + return 0; } @@ -895,7 +906,7 @@ static void sdp_shutdown(struct sock *sk, int how) return; if (sk->sk_state == TCP_ESTABLISHED) - sdp_set_state(sk, TCP_FIN_WAIT1); + sdp_set_state(sk, TCP_FIN_WAIT2);/* should be TCP_FIN_WAIT1 */ else if (sk->sk_state == TCP_CLOSE_WAIT) sdp_set_state(sk, TCP_LAST_ACK); else @@ -910,7 +921,7 @@ static void sdp_shutdown(struct sock *sk, int how) if (ssk->nonagle & TCP_NAGLE_OFF) ssk->nonagle |= TCP_NAGLE_PUSH; - sdp_post_sends(ssk, 0); + sdp_send_disconnect(sk); } static void sdp_mark_push(struct sdp_sock *ssk, struct sk_buff *skb) -- 1.5.2 From ogerlitz at voltaire.com Thu Jun 19 08:01:21 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 18:01:21 +0300 Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event (second try) In-Reply-To: <48590FF2.8090507@mellanox.co.il> References: <48590FF2.8090507@mellanox.co.il> Message-ID: <485A74C1.5010804@voltaire.com> Amir Vadai wrote: > SDP needs to be notified when CM exits the TIMEWAIT state. > CM does notify the CMA but CMA doesn't pass it to the upper layer. > > Below is a patch for the CMA code. I wasn't sure if it is ok to set > CMA_DISCONNECT in the IB_CM_TIMEWAIT_EXIT instead of doing it in the > IB_CM_DREQ_RECEIVED/IB_CM_DREP_RECEIVED as it is done today. Instead > I changed the check at the beginning of cma_ib_handler() as you can > see in the patch. > So how many events would now be seen by the rdma-cm consumer? maybe it would be better to have this event delivered only to PS_SDP IDs? Or. From rdreier at cisco.com Thu Jun 19 08:05:36 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 08:05:36 -0700 Subject: [ofa-general] unsing ibverbs 1.0 with rdmacm In-Reply-To: (Thomas Eggestad's message of "Thu, 12 Jun 2008 16:51:33 +0200") References: Message-ID: > I do not believe this solves the scenario from my point of view. I > was not clear about the fact that the app binary is not recompiled. > The customer may be running older or newer OFED stack where > the changes described in my original scenario may have happened. I'm not sure I follow. If you build a binary against some version of OFED where librdmacm is linked against libibverbs 1.1, then everything should work fine against any older version of OFED where all the needed symbols are available. I can't predict what might happen with unreleased future versions of OFED but I would guess that should work. I guess if we ever change the libibverbs binary in to 1.2 or something, and then librdmacm links against that in a future OFED release, we'll have to figure out a solution for applications linked against old librdmacm/libibverbs-1.1, but that is all theoretical at the moment. I think the only way you get in trouble is if you go out of your way to break things using dlvsym() tricks or something like that. - R. From eli at dev.mellanox.co.il Thu Jun 19 08:07:48 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 18:07:48 +0300 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Configure QP's max message size In-Reply-To: References: <20080619142802.GC15786@mtls03> Message-ID: <20080619150748.GD15786@mtls03> On Thu, Jun 19, 2008 at 07:53:14AM -0700, Roland Dreier wrote: > > Configure QP's max message size according to the value queried > > from the HCA through QUERY_DEV_CAP. > > Not enough info for a changelog entry: why do this? What does it > fix/improve? Is this an urgent fix? > The HCA declares the maximum message size it is capable of transmitting through query dev cap. When modifying a QP we should set this value. For connected QPs where messages sizes can be quite large we should set this value to a value not larger then the value published. So this patch sets it to the maximum allowed value. I can't say this patch is urgent but I sent a few months ago and we do wish to see it in the upstream kernel after being reviewed. Based on that, let me know if you think we should push it and I can add more information to the changelog. From rdreier at cisco.com Thu Jun 19 08:07:44 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 08:07:44 -0700 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: (Or Gerlitz's message of "Thu, 19 Jun 2008 17:05:35 +0300 (IDT)") References: Message-ID: > As I said, since some more changes are possible in the rdma-cm > for 2.6.27, I think it would be better if you do this push, however, > if you you prefer to push this through Jeff's tree, let it be. It becomes a mess if I put patches from Jeff's tree into my tree too. What are the dependencies of your patches on pending changes for other trees? One simple possibility is just to wait on putting these patches into any tree until the prerequisites are upstream. - R. From rdreier at cisco.com Thu Jun 19 08:10:27 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 08:10:27 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Configure QP's max message size In-Reply-To: <20080619150748.GD15786@mtls03> (Eli Cohen's message of "Thu, 19 Jun 2008 18:07:48 +0300") References: <20080619142802.GC15786@mtls03> <20080619150748.GD15786@mtls03> Message-ID: > I can't say this patch is urgent but I sent a few months ago and we do > wish to see it in the upstream kernel after being reviewed. Based on > that, let me know if you think we should push it and I can add more > information to the changelog. The patch itself looks fine. But my question on urgency is whether this fixes a big problem and needs to go into 2.6.26, or whether we can wait for 2.6.27. Based on the fact that it's been around for a few months and no one seems to have noticed it's missing upstream, I'm guessing 2.6.27 is fine. I probably lost it the first time around because I didn't understand why the change mattered -- so yes, with a changelog that explains the problem with the current code and why this patch fixes it, I would merge it right away. From monis at Voltaire.COM Thu Jun 19 08:11:25 2008 From: monis at Voltaire.COM (Moni Shoua) Date: Thu, 19 Jun 2008 18:11:25 +0300 Subject: [ofa-general] [PATCH] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: <484F8A20.9070101@Voltaire.COM> References: <484F8A20.9070101@Voltaire.COM> Message-ID: <485A771D.9010104@Voltaire.COM> Moni Shoua wrote: > The patch tries to solve the problem of device going down and paths being > flushed on SM change event. The method is to mark the paths as candidates for > refresh (with a valid flag) and wait for ARP probe to start a flow of path > lookup that leads to patch query which ends up in path (and address handle) > refresh. > The solution requires a different and less intrusive handling of SM change > event. For that, the second argument of the flush function changed it's meaning > from boolean flag to a level. > In most cases, SM failover doesn't cause LID change so traffic won't stop. In > the rare cases of LID change, the remote host (the one that hadn't changed > its LID) will lose connectivity until paths are refreshed. This is no worse than > the current state. In fact, preventing the device from going down saves packets > that otherwise would be lost. > > Signed-off-by: Moni Levy > Signed-off-by: Moni Shoua > --- Hi Roland, This issue is important to us. I'd appreciate if you review it and make an opinion. I understand it's kind of late now to make it to 2.6.26 but we hope to have it in 2.6.27 thanks MoniS From ogerlitz at voltaire.com Thu Jun 19 08:13:49 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 18:13:49 +0300 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: <484E9264.4070708@voltaire.com> References: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> <484E9264.4070708@voltaire.com> Message-ID: <485A77AD.4010009@voltaire.com> Or Gerlitz wrote: > Roland Dreier wrote: >> Yes, it's adequate for the current use but I'm wondering why we want to >> choose a more complex implementation that hides hardware capabilities? > I tend to think the suggested way is somehow simpler (vs. the per > attach) see my reply on "patch 1/3" In the sockets world the sender sets the IP_MULTICAST_LOOP socket option which ip(7) says that it "Sets or reads a boolean integer argument whether sent multicast packets should be looped back to the local sockets." so one more point here is that the suggested implementation of setting this at the qp creation is one that brings the API closer to the socket one, which can be easier for users to understand. >> It's just as easy to add a flag to the multicast attach verb -- maybe >> easier, in fact, since there are 16 reserved bits in struct >> ib_uverbs_attach_mcast. > I am concerned about both libibverbs --- uverbs ABI and the app -- > libibverbs ABI > I understand that with this suggestion of yours, the libibverbs > API/ABI of MCG attach to apps would change, isn't it something we want > to avoid, eg through introducing a new verbs create_qp_ext that gets > also creation flag? > Roland, can you please comment on your preferences, that is on the cap bit vs hint and where you want the app to direct the HW how to behave, in the qp creation time as we suggest or on the time of attach. If you want to go the changing the qp attach verb way, can please provide us with some implementation directions regarding backward compatibility with apps that are built against versions of libibverbs which have different API for this verb, thanks. Or. From rdreier at cisco.com Thu Jun 19 08:16:09 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 08:16:09 -0700 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> (Chien Tung's message of "Wed, 18 Jun 2008 17:23:13 -0500") References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> Message-ID: > The "Retransmission count" is per netdev and we support multiple netdev > per physical port. We have designed our silicon to support stats at > netdev level and it doesn't make sense for us to aggregate stats from > all netdevs and report at a device level. Perhaps we can compromise, > Chelsio can give device level stats at /sys/class/infiniband/$device and > for NetEffect, we can provide netdev level stats below it? Hmm, kind of a mess if we can't achieve some uniformity... how do you handle stats if I change routing so an offloaded connection goes from physical port 1 to physical port 2? Or presumably you have support for port aggregation in hardware so an offloaded connection can use both physical ports at the same time? Actually it seems that you attach a struct ib_device to each netdev (eg a two-port NIC gets two RDMA devices with iw_nes) so Steve's scheme should work fine for you too? (I just looked on a system with a two-port NIC and I do see nes0 and nes1 devices) - R. From amirv at mellanox.co.il Thu Jun 19 08:17:47 2008 From: amirv at mellanox.co.il (Amir Vadai) Date: Thu, 19 Jun 2008 18:17:47 +0300 Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event (second try) In-Reply-To: <485A74C1.5010804@voltaire.com> References: <48590FF2.8090507@mellanox.co.il> <485A74C1.5010804@voltaire.com> Message-ID: <40FA0A8088E8A441973D37502F00933E16298B@mtlexch01.mtl.com> This event might be usefull for other ULPs. The reason I needed it in SDP, is because the QP could be destroyed only when the timewait finish. And for that I need a timewait exit event. - Amir -----Original Message----- From: Or Gerlitz [mailto:ogerlitz at voltaire.com] Sent: ה 19 יוני 2008 18:01 To: Amir Vadai Cc: Sean Hefty; Oren Duer; general at lists.openfabrics.org Subject: Re: [ofa-general] cma: add support for TIMEWAIT_EXIT event (second try) Amir Vadai wrote: > SDP needs to be notified when CM exits the TIMEWAIT state. > CM does notify the CMA but CMA doesn't pass it to the upper layer. > > Below is a patch for the CMA code. I wasn't sure if it is ok to set > CMA_DISCONNECT in the IB_CM_TIMEWAIT_EXIT instead of doing it in the > IB_CM_DREQ_RECEIVED/IB_CM_DREP_RECEIVED as it is done today. Instead I > changed the check at the beginning of cma_ib_handler() as you can see > in the patch. > So how many events would now be seen by the rdma-cm consumer? maybe it would be better to have this event delivered only to PS_SDP IDs? Or. -- Amir Vadai Mellanox Technologies LTD. SW Phone: +972 (3) 625 9500 (ext.539) Mobile: +972 (52) 632 4714 E-mail: amirv at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From ogerlitz at voltaire.com Thu Jun 19 08:19:31 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Jun 2008 18:19:31 +0300 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: References: Message-ID: <485A7903.4050407@voltaire.com> Roland Dreier wrote: > > As I said, since some more changes are possible in the rdma-cm > > for 2.6.27, I think it would be better if you do this push, however, > > if you you prefer to push this through Jeff's tree, let it be. > > It becomes a mess if I put patches from Jeff's tree into my tree too. > What are the dependencies of your patches on pending changes for other > trees? One simple possibility is just to wait on putting these patches > into any tree until the prerequisites are upstream. My patches are only dependent on the three commits to Jeff's tree which I have stated in the 0/4 email message. They are not dependent on other patches to the infiniband tree or any other tree. I do want them to be queued for push into 2.6.27 at some tree. So one option is that you pull Jeff's davem-next into a branch at your tree and once Jeff does his push you ask to pull this branch whose only diff is would be those three patches, I think it possible, and you did it with libiscsi on the time we pushed iser. If this becomes a mess, lets just push the four patches to Jeff. Or. From eli at dev.mellanox.co.il Thu Jun 19 08:21:29 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 18:21:29 +0300 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Check completion done when a command In-Reply-To: References: <20080619141520.GB15786@mtls03> Message-ID: <20080619152129.GE15786@mtls03> On Thu, Jun 19, 2008 at 07:53:55AM -0700, Roland Dreier wrote: > I seem to recall we had the same discussion about mthca and concluded > that it was a bug that should be fixed in wait_for_completion_timeout(). Maybe this functions should be fixed and maybe there should be another function which does what's needed. The question is shouldn't we have at least a workaround to this problem until there is better support from the kernel? From rdreier at cisco.com Thu Jun 19 08:21:32 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 08:21:32 -0700 Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event (second try) In-Reply-To: <40FA0A8088E8A441973D37502F00933E16298B@mtlexch01.mtl.com> (Amir Vadai's message of "Thu, 19 Jun 2008 18:17:47 +0300") References: <48590FF2.8090507@mellanox.co.il> <485A74C1.5010804@voltaire.com> <40FA0A8088E8A441973D37502F00933E16298B@mtlexch01.mtl.com> Message-ID: > The reason I needed it in SDP, is because the QP could be destroyed > only when the timewait finish. And for that I need a timewait exit > event. What is the current cma design? How are consumers supposed to take the IB timewait period into account when destroying QPs? - R. From rdreier at cisco.com Thu Jun 19 08:22:48 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 08:22:48 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Check completion done when a command In-Reply-To: <20080619152129.GE15786@mtls03> (Eli Cohen's message of "Thu, 19 Jun 2008 18:21:29 +0300") References: <20080619141520.GB15786@mtls03> <20080619152129.GE15786@mtls03> Message-ID: > Maybe this functions should be fixed and maybe there should be another > function which does what's needed. The question is shouldn't we have > at least a workaround to this problem until there is better support > from the kernel? Because if we're going to merge something upstream, we might as well merge the real fix (that fixes both mthca and mlx4) rather than merging a workaround and then leaving the bug for another year or however long it takes to hit it again. - R. From eli at dev.mellanox.co.il Thu Jun 19 08:27:41 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 18:27:41 +0300 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Configure QP's max message size In-Reply-To: References: <20080619142802.GC15786@mtls03> <20080619150748.GD15786@mtls03> Message-ID: <20080619152741.GF15786@mtls03> On Thu, Jun 19, 2008 at 08:10:27AM -0700, Roland Dreier wrote: > > The patch itself looks fine. But my question on urgency is whether this > fixes a big problem and needs to go into 2.6.26, or whether we can wait > for 2.6.27. Based on the fact that it's been around for a few months > and no one seems to have noticed it's missing upstream, I'm guessing > 2.6.27 is fine. > 2.6.27 is fine. I'll send the patch again. From swise at opengridcomputing.com Thu Jun 19 08:32:52 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 19 Jun 2008 10:32:52 -0500 Subject: [ofa-general] [PATCH RFCv3 0/2] RDMA Protocol Stats Message-ID: <20080619153252.15418.21923.stgit@dell3.ogc.int> The following series implements adding protocol stats attributes to the RDMA sysfs class and providing those stats in iw_cxgb3. Changes since V2: - align struct iw_protocol_stats with the TCP and IP MIB variables defined in linux/snmp.h. - make all the stats u64. - fixed composing u64 from the chelsio hi/lo u32 stats. -- Stevo. From swise at opengridcomputing.com Thu Jun 19 08:34:24 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 19 Jun 2008 10:34:24 -0500 Subject: [ofa-general] [PATCH RFCv3 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <20080619153252.15418.21923.stgit@dell3.ogc.int> References: <20080619153252.15418.21923.stgit@dell3.ogc.int> Message-ID: <20080619153424.15418.99244.stgit@dell3.ogc.int> This patch adds a sysfs attribute group called "proto_stats" under /sys/class/infiniband/$device/ and populates this group with protocol statistics if they exist for a given device. Currently, only iWARP stats are defined, but the code is designed to allow Infiniband protocol stats if they become available. These stats are per-device and more importantly -not- per port. Details: - Add union rdma_protocol_stats in ib_verbs.h. This union allows defining transport-specific stats. Currently only iwarp stats are defined. - Add struct iw_protocol_stats to define the current set of iwarp protocol stats. - Add new ib_device method called get_proto_stats() to return protocol statistics. - Add logic in core/sysfs.c to create iwarp protocol stats attributes if the device is an RNIC and has a get_proto_stats() method. Signed-off-by: Steve Wise --- drivers/infiniband/core/sysfs.c | 123 +++++++++++++++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 53 +++++++++++++++++ 2 files changed, 176 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 9575655..28d0ab9 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -665,6 +665,124 @@ static struct class ib_class = { .dev_uevent = ib_device_uevent, }; +/* Show a given an attribute in the statistics group */ +static ssize_t show_protocol_stat(const struct device *device, + struct device_attribute *attr, char *buf, + unsigned offset) +{ + struct ib_device *dev = container_of(device, struct ib_device, dev); + union rdma_protocol_stats stats; + ssize_t ret = -EINVAL; + + WARN_ON(offset > sizeof(union rdma_protocol_stats) || + offset % sizeof (u64) != 0); + + if (!dev->get_protocol_stats) + goto out; + ret = (*dev->get_protocol_stats)(dev, &stats); + if (ret) + goto out; + ret = sprintf(buf, "%llu\n", *(u64 *)(((u8 *)&stats) + offset)); +out: + return ret; +} + +/* generate a read-only iwarp statistics attribute */ +#define IW_STATS_ENTRY(name) \ +static ssize_t show_##name(struct device *device, \ + struct device_attribute *attr, char *buf) \ +{ \ + return show_protocol_stat(device, attr, buf, \ + offsetof(struct iw_protocol_stats, name)); \ +} \ +static DEVICE_ATTR(name, S_IRUGO, show_##name, NULL) + +IW_STATS_ENTRY(ipInReceives); +IW_STATS_ENTRY(ipInHdrErrors); +IW_STATS_ENTRY(ipInTooBigErrors); +IW_STATS_ENTRY(ipInNoRoutes); +IW_STATS_ENTRY(ipInAddrErrors); +IW_STATS_ENTRY(ipInUnknownProtos); +IW_STATS_ENTRY(ipInTruncatedPkts); +IW_STATS_ENTRY(ipInDiscards); +IW_STATS_ENTRY(ipInDelivers); +IW_STATS_ENTRY(ipOutForwDatagrams); +IW_STATS_ENTRY(ipOutRequests); +IW_STATS_ENTRY(ipOutDiscards); +IW_STATS_ENTRY(ipOutNoRoutes); +IW_STATS_ENTRY(ipReasmTimeout); +IW_STATS_ENTRY(ipReasmReqds); +IW_STATS_ENTRY(ipReasmOKs); +IW_STATS_ENTRY(ipReasmFails); +IW_STATS_ENTRY(ipFragOKs); +IW_STATS_ENTRY(ipFragFails); +IW_STATS_ENTRY(ipFragCreates); +IW_STATS_ENTRY(ipInMcastPkts); +IW_STATS_ENTRY(ipOutMcastPkts); +IW_STATS_ENTRY(ipInBcastPkts); +IW_STATS_ENTRY(ipOutBcastPkts); +IW_STATS_ENTRY(tcpRtoAlgorithm); +IW_STATS_ENTRY(tcpRtoMin); +IW_STATS_ENTRY(tcpRtoMax); +IW_STATS_ENTRY(tcpMaxConn); +IW_STATS_ENTRY(tcpActiveOpens); +IW_STATS_ENTRY(tcpPassiveOpens); +IW_STATS_ENTRY(tcpAttemptFails); +IW_STATS_ENTRY(tcpEstabResets); +IW_STATS_ENTRY(tcpCurrEstab); +IW_STATS_ENTRY(tcpInSegs); +IW_STATS_ENTRY(tcpOutSegs); +IW_STATS_ENTRY(tcpRetransSegs); +IW_STATS_ENTRY(tcpInErrs); +IW_STATS_ENTRY(tcpOutRsts); + +static struct attribute *iw_proto_stats_attrs[] = { + &dev_attr_ipInReceives.attr, + &dev_attr_ipInHdrErrors.attr, + &dev_attr_ipInTooBigErrors.attr, + &dev_attr_ipInNoRoutes.attr, + &dev_attr_ipInAddrErrors.attr, + &dev_attr_ipInUnknownProtos.attr, + &dev_attr_ipInTruncatedPkts.attr, + &dev_attr_ipInDiscards.attr, + &dev_attr_ipInDelivers.attr, + &dev_attr_ipOutForwDatagrams.attr, + &dev_attr_ipOutRequests.attr, + &dev_attr_ipOutDiscards.attr, + &dev_attr_ipOutNoRoutes.attr, + &dev_attr_ipReasmTimeout.attr, + &dev_attr_ipReasmReqds.attr, + &dev_attr_ipReasmOKs.attr, + &dev_attr_ipReasmFails.attr, + &dev_attr_ipFragOKs.attr, + &dev_attr_ipFragFails.attr, + &dev_attr_ipFragCreates.attr, + &dev_attr_ipInMcastPkts.attr, + &dev_attr_ipOutMcastPkts.attr, + &dev_attr_ipInBcastPkts.attr, + &dev_attr_ipOutBcastPkts.attr, + &dev_attr_tcpRtoAlgorithm.attr, + &dev_attr_tcpRtoMin.attr, + &dev_attr_tcpRtoMax.attr, + &dev_attr_tcpMaxConn.attr, + &dev_attr_tcpActiveOpens.attr, + &dev_attr_tcpPassiveOpens.attr, + &dev_attr_tcpAttemptFails.attr, + &dev_attr_tcpEstabResets.attr, + &dev_attr_tcpCurrEstab.attr, + &dev_attr_tcpInSegs.attr, + &dev_attr_tcpOutSegs.attr, + &dev_attr_tcpRetransSegs.attr, + &dev_attr_tcpInErrs.attr, + &dev_attr_tcpOutRsts.attr, + NULL +}; + +static struct attribute_group iw_stats_group = { + .name = "proto_stats", + .attrs = iw_proto_stats_attrs, +}; + int ib_device_register_sysfs(struct ib_device *device) { struct device *class_dev = &device->dev; @@ -707,6 +825,11 @@ int ib_device_register_sysfs(struct ib_device *device) } } + if (device->node_type == RDMA_NODE_RNIC && device->get_protocol_stats) { + ret = sysfs_create_group(&class_dev->kobj, &iw_stats_group); + if (ret) + goto err_put; + } return 0; err_put: diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 62e5bcd..6f2c180 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -53,6 +53,57 @@ #include #include +struct ib_protocol_stats { + /* TBD... */ +}; + +struct iw_protocol_stats { + u64 ipInReceives; + u64 ipInHdrErrors; + u64 ipInTooBigErrors; + u64 ipOutNoRoutes; + u64 ipInAddrErrors; + u64 ipInUnknownProtos; + u64 ipInTruncatedPkts; + u64 ipInDiscards; + u64 ipInDelivers; + u64 ipOutForwDatagrams; + u64 ipOutRequests; + u64 ipOutDiscards; + u64 ipOutNoRoutes; + u64 ipReasmTimeout; + u64 ipReasmReqds; + u64 ipReasmOKs; + u64 ipReasmFails; + u64 ipFragOKs; + u64 ipFragFails; + u64 ipFragCreates; + u64 ipInMcastPkts; + u64 ipOutMcastPkts; + u64 ipInBcastPkts; + u64 ipOutBcastPkts; + + u64 tcpRtoAlgorithm; + u64 tcpRtoMin; + u64 tcpRtoMax; + u64 tcpMaxConn; + u64 tcpActiveOpens; + u64 tcpPassiveOpens; + u64 tcpAttemptFails; + u64 tcpEstabResets; + u64 tcpCurrEstab; + u64 tcpInSegs; + u64 tcpOutSegs; + u64 tcpRetransSegs; + u64 tcpInErrs; + u64 tcpOutRsts; +}; + +union rdma_protocol_stats { + struct ib_protocol_stats ib; + struct iw_protocol_stats iw; +}; + union ib_gid { u8 raw[16]; struct { @@ -947,6 +998,8 @@ struct ib_device { struct iw_cm_verbs *iwcm; + int (*get_protocol_stats)(struct ib_device *device, + union rdma_protocol_stats *stats); int (*query_device)(struct ib_device *device, struct ib_device_attr *device_attr); int (*query_port)(struct ib_device *device, From swise at opengridcomputing.com Thu Jun 19 08:34:26 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 19 Jun 2008 10:34:26 -0500 Subject: [ofa-general] [PATCH RFCv3 2/2] RDMA/cxgb3: Add support for protocol statistics. In-Reply-To: <20080619153252.15418.21923.stgit@dell3.ogc.int> References: <20080619153252.15418.21923.stgit@dell3.ogc.int> Message-ID: <20080619153426.15418.30544.stgit@dell3.ogc.int> - Add a new rdma ctl command called RDMA_GET_MIB to the cxgb3 low level driver to obtain the protocol mib from the rnic hardware. - Add new iw_cxgb3 provider method to get the MIB from the low level driver. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 62 +++++++++++++++++++++++++-- drivers/net/cxgb3/cxgb3_ctl_defs.h | 1 drivers/net/cxgb3/cxgb3_offload.c | 7 +++ include/rdma/ib_verbs.h | 2 - 4 files changed, 67 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 193dfe7..2c1c477 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -56,6 +56,7 @@ #include "iwch_provider.h" #include "iwch_cm.h" #include "iwch_user.h" +#include "common.h" static int iwch_modify_port(struct ib_device *ibdev, u8 port, int port_modify_mask, @@ -1210,6 +1211,61 @@ static ssize_t show_board(struct device *dev, struct device_attribute *attr, iwch_dev->rdev.rnic_info.pdev->device); } +static int iwch_get_mib(struct ib_device *ibdev, + union rdma_protocol_stats *stats) +{ + struct iwch_dev *dev; + struct tp_mib_stats m; + int ret; + + PDBG("%s ibdev %p\n", __func__, ibdev); + dev = to_iwch_dev(ibdev); + ret = dev->rdev.t3cdev_p->ctl(dev->rdev.t3cdev_p, RDMA_GET_MIB, &m); + if (ret) + return -ENOSYS; + + memset(stats, 0, sizeof *stats); + stats->iw.ipInReceives = ((u64) m.ipInReceive_hi << 32) + + m.ipInReceive_lo; + stats->iw.ipInHdrErrors = ((u64) m.ipInHdrErrors_hi << 32) + + m.ipInHdrErrors_lo; + stats->iw.ipInAddrErrors = ((u64) m.ipInAddrErrors_hi << 32) + + m.ipInAddrErrors_lo; + stats->iw.ipInUnknownProtos = ((u64) m.ipInUnknownProtos_hi << 32) + + m.ipInUnknownProtos_lo; + stats->iw.ipInDiscards = ((u64) m.ipInDiscards_hi << 32) + + m.ipInDiscards_lo; + stats->iw.ipInDelivers = ((u64) m.ipInDelivers_hi << 32) + + m.ipInDelivers_lo; + stats->iw.ipOutRequests = ((u64) m.ipOutRequests_hi << 32) + + m.ipOutRequests_lo; + stats->iw.ipOutDiscards = ((u64) m.ipOutDiscards_hi << 32) + + m.ipOutDiscards_lo; + stats->iw.ipOutNoRoutes = ((u64) m.ipOutNoRoutes_hi << 32) + + m.ipOutNoRoutes_lo; + stats->iw.ipReasmTimeout = (u64) m.ipReasmTimeout; + stats->iw.ipReasmReqds = (u64) m.ipReasmReqds; + stats->iw.ipReasmOKs = (u64) m.ipReasmOKs; + stats->iw.ipReasmFails = (u64) m.ipReasmFails; + stats->iw.tcpActiveOpens = (u64) m.tcpActiveOpens; + stats->iw.tcpPassiveOpens = (u64) m.tcpPassiveOpens; + stats->iw.tcpAttemptFails = (u64) m.tcpAttemptFails; + stats->iw.tcpEstabResets = (u64) m.tcpEstabResets; + stats->iw.tcpOutRsts = (u64) m.tcpOutRsts; + stats->iw.tcpCurrEstab = (u64) m.tcpCurrEstab; + stats->iw.tcpInSegs = ((u64) m.tcpInSegs_hi << 32) + + m.tcpInSegs_lo; + stats->iw.tcpOutSegs = ((u64) m.tcpOutSegs_hi << 32) + + m.tcpOutSegs_lo; + stats->iw.tcpRetransSegs = ((u64) m.tcpRetransSeg_hi << 32) + + m.tcpRetransSeg_lo; + stats->iw.tcpInErrs = ((u64) m.tcpInErrs_hi << 32) + + m.tcpInErrs_lo; + stats->iw.tcpRtoMin = (u64) m.tcpRtoMin; + stats->iw.tcpRtoMax = (u64) m.tcpRtoMax; + return 0; +} + static DEVICE_ATTR(hw_rev, S_IRUGO, show_rev, NULL); static DEVICE_ATTR(fw_ver, S_IRUGO, show_fw_ver, NULL); static DEVICE_ATTR(hca_type, S_IRUGO, show_hca, NULL); @@ -1219,7 +1275,7 @@ static struct device_attribute *iwch_class_attributes[] = { &dev_attr_hw_rev, &dev_attr_fw_ver, &dev_attr_hca_type, - &dev_attr_board_id + &dev_attr_board_id, }; int iwch_register_device(struct iwch_dev *dev) @@ -1290,15 +1346,13 @@ int iwch_register_device(struct iwch_dev *dev) dev->ibdev.alloc_fast_reg_mr = iwch_alloc_fast_reg_mr; dev->ibdev.alloc_fast_reg_page_list = iwch_alloc_fastreg_pbl; dev->ibdev.free_fast_reg_page_list = iwch_free_fastreg_pbl; - dev->ibdev.attach_mcast = iwch_multicast_attach; dev->ibdev.detach_mcast = iwch_multicast_detach; dev->ibdev.process_mad = iwch_process_mad; - dev->ibdev.req_notify_cq = iwch_arm_cq; dev->ibdev.post_send = iwch_post_send; dev->ibdev.post_recv = iwch_post_receive; - + dev->ibdev.get_protocol_stats = iwch_get_mib; dev->ibdev.iwcm = kmalloc(sizeof(struct iw_cm_verbs), GFP_KERNEL); if (!dev->ibdev.iwcm) diff --git a/drivers/net/cxgb3/cxgb3_ctl_defs.h b/drivers/net/cxgb3/cxgb3_ctl_defs.h index 6c4f320..ed0ecd9 100644 --- a/drivers/net/cxgb3/cxgb3_ctl_defs.h +++ b/drivers/net/cxgb3/cxgb3_ctl_defs.h @@ -54,6 +54,7 @@ enum { RDMA_CQ_DISABLE = 16, RDMA_CTRL_QP_SETUP = 17, RDMA_GET_MEM = 18, + RDMA_GET_MIB = 19, GET_RX_PAGE_INFO = 50, }; diff --git a/drivers/net/cxgb3/cxgb3_offload.c b/drivers/net/cxgb3/cxgb3_offload.c index ff9c013..cf26968 100644 --- a/drivers/net/cxgb3/cxgb3_offload.c +++ b/drivers/net/cxgb3/cxgb3_offload.c @@ -303,6 +303,12 @@ static int cxgb_rdma_ctl(struct adapter *adapter, unsigned int req, void *data) spin_unlock_irq(&adapter->sge.reg_lock); break; } + case RDMA_GET_MIB: { + spin_lock(&adapter->stats_lock); + t3_tp_get_mib_stats(adapter, (struct tp_mib_stats *)data); + spin_unlock(&adapter->stats_lock); + break; + } default: ret = -EOPNOTSUPP; } @@ -381,6 +387,7 @@ static int cxgb_offload_ctl(struct t3cdev *tdev, unsigned int req, void *data) case RDMA_CQ_DISABLE: case RDMA_CTRL_QP_SETUP: case RDMA_GET_MEM: + case RDMA_GET_MIB: if (!offload_running(adapter)) return -EAGAIN; return cxgb_rdma_ctl(adapter, req, data); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 6f2c180..d5e30d4 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -61,7 +61,7 @@ struct iw_protocol_stats { u64 ipInReceives; u64 ipInHdrErrors; u64 ipInTooBigErrors; - u64 ipOutNoRoutes; + u64 ipInNoRoutes; u64 ipInAddrErrors; u64 ipInUnknownProtos; u64 ipInTruncatedPkts; From eli at dev.mellanox.co.il Thu Jun 19 08:38:19 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 18:38:19 +0300 Subject: [ofa-general] [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <485A7266.6050802@voltaire.com> References: <1213628310.5447.91.camel@mtls03> <485929A9.6030306@voltaire.com> <20080618155100.GC11489@mtls03> <485A7266.6050802@voltaire.com> Message-ID: <20080619153819.GG15786@mtls03> On Thu, Jun 19, 2008 at 05:51:18PM +0300, Or Gerlitz wrote: > It seems that other drivers implement multi queuing in the send flow, where > you come and suggest to implement it in the receive flow. What's the > architectural conception behind this? As I mentioned that in the RFC, this patch should distribute the work on received packets amongst the available CPUs so you have more power in the receive. If you want to read more you can google search on receive side scaling. > > What problem your patches come to solve? and maybe you can comment why you > think there's no other implementation? That's not what I said. I said I don't know of any other implementation except for mlx4_net but that's not in the upstream kernel yet. > is it since other Ethernet HW > vendors don't support multiple receive queues or can not generate > interrupts to multiple interrupt vectors through MSI-X as the connectx > does? > From eli at dev.mellanox.co.il Thu Jun 19 08:52:51 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 19 Jun 2008 18:52:51 +0300 Subject: [ofa-general] [PATCH v2] IB/mlx4: Configure QP's max message size Message-ID: <20080619155251.GH15786@mtls03> >From d215882c1305d9c964c9c71ff1cf29a04395234f Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Thu, 19 Jun 2008 17:24:07 +0300 Subject: [PATCH] IB/mlx4: Configure QP's max message size ConnectX declares the max message size it supports through QUERY_DEV_CAP. When modifying a QP to RTR, the max message size for the QP must be specified. This value must not exceed the value declared through QUERY_DEV_CAP. The current code ignores the max allowed size and sets this value 2^31. This patch sets to all the QPs the max value allowed as queried. Signed-off-by: Eli Cohen --- Changes: updated the changelog to be more verbose. drivers/infiniband/hw/mlx4/qp.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 8e02ecf..fe38c85 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -903,7 +903,8 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp, attr->path_mtu); goto out; } - context->mtu_msgmax = (attr->path_mtu << 5) | 31; + context->mtu_msgmax = (attr->path_mtu << 5) | + ilog2(dev->dev->caps.max_msg_sz); } if (qp->rq.wqe_cnt) -- 1.5.5.4 From sean.hefty at intel.com Thu Jun 19 10:04:22 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 19 Jun 2008 10:04:22 -0700 Subject: [ofa-general] [PATCH 1/4] sdp: Use sdp_set_state() In-Reply-To: <12138874774119-git-send-email-amirv@mellanox.co.il> References: sdp_gracefull_close <12138874774119-git-send-email-amirv@mellanox.co.il> Message-ID: <000001c8d22e$8508e3a0$bb37170a@amr.corp.intel.com> >+static inline void sdp_set_state(struct sock *sk, int state) >+{ >+ sk->sk_state = state; >+} Why have this abstraction? >- sk->sk_state = TCP_CLOSE; >+ sdp_set_state(sk, TCP_CLOSE); IMO, the previous code is simpler to maintain. - Sean From sean.hefty at intel.com Thu Jun 19 10:29:53 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 19 Jun 2008 10:29:53 -0700 Subject: [ofa-general] cma: add support for TIMEWAIT_EXIT event (second try) In-Reply-To: References: <48590FF2.8090507@mellanox.co.il> <485A74C1.5010804@voltaire.com><40FA0A8088E8A441973D37502F00933E16298B@mtlexch01.mtl.com> Message-ID: <000101c8d232$15f78580$bb37170a@amr.corp.intel.com> >What is the current cma design? How are consumers supposed to take the >IB timewait period into account when destroying QPs? The cma just drops the event at the moment. Exposing the event is probably the right thing to do. I need to look at the proposed solution in more detail to see if there are any issues. - Sean From amirv.mellanox at gmail.com Thu Jun 19 10:40:00 2008 From: amirv.mellanox at gmail.com (Amir Mellanox) Date: Thu, 19 Jun 2008 20:40:00 +0300 Subject: [ofa-general] [PATCH 1/4] sdp: Use sdp_set_state() In-Reply-To: <000001c8d22e$8508e3a0$bb37170a@amr.corp.intel.com> References: <12138874774119-git-send-email-amirv@mellanox.co.il> <000001c8d22e$8508e3a0$bb37170a@amr.corp.intel.com> Message-ID: <18e64aac0806191040l3238bab6ua869e2416b1cb8af@mail.gmail.com> It is easier to add debug prints for sdp state machine On Thu, Jun 19, 2008 at 8:04 PM, Sean Hefty wrote: > >+static inline void sdp_set_state(struct sock *sk, int state) > >+{ > >+ sk->sk_state = state; > >+} > > Why have this abstraction? > > >- sk->sk_state = TCP_CLOSE; > >+ sdp_set_state(sk, TCP_CLOSE); > > IMO, the previous code is simpler to maintain. > > - Sean > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Thu Jun 19 10:49:20 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 19 Jun 2008 10:49:20 -0700 Subject: [ofa-general] [PATCH] IB/mlx4: forward congestion class MADs to the HCA In-Reply-To: <20080619071007.GA15786@mtls03> References: <20080618185317.GA8525@mtls03> <1213823412.13056.156.camel@hrosenstock-ws.xsigo.com> <20080619071007.GA15786@mtls03> Message-ID: <1213897760.13056.175.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-19 at 10:10 +0300, Eli Cohen wrote: > > Does the current MLX4 firmware support CC ? > > > Yes, but it is not released yet. Current released FW versions will > drop CC mads if they get them through MAD_IFC. Is updated switch microcode also needed for CC ? -- Hal From hrosenstock at xsigo.com Thu Jun 19 11:04:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 19 Jun 2008 11:04:29 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] libibmad/rpc.c: Better error message in madrpc_init Message-ID: <1213898669.13056.180.camel@hrosenstock-ws.xsigo.com> libibmad/rpc.c: Better error message in madrpc_init Signed-off-by: Hal Rosenstock diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 17f43b8..cb1940b 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -314,7 +314,7 @@ madrpc_init(char *dev_name, int dev_port, int *mgmt_classes, int num_classes) if (mgmt == IB_SA_CLASS) rmpp_version = 1; if (mad_register_client(mgmt, rmpp_version) < 0) - IBPANIC("client_register for mgmt %d failed", mgmt); + IBPANIC("client_register for mgmt class %d failed", mgmt); } } From pw at osc.edu Thu Jun 19 12:44:43 2008 From: pw at osc.edu (Pete Wyckoff) Date: Thu, 19 Jun 2008 15:44:43 -0400 Subject: [ofa-general] Re: mthca MR attrs userspace change In-Reply-To: References: <20080527180004.GA15444@osc.edu> <20080603204407.GA7840@osc.edu> Message-ID: <20080619194443.GA13112@osc.edu> rdreier at cisco.com wrote on Fri, 06 Jun 2008 12:01 -0700: > > Nor am I. It never showed up in updates-testing or the real > > updates, a week after you built it. I can't find any developer docs > > on how this is supposed to work. Maybe someone on the list knows > > what button to push? > > OK, I think I figured it out... I need to do a "make update" to get it > submitted to the Fedora "bodhi" tool, where it gets into testing > eventually (a manual process that may take some time). The request is > at https://admin.fedoraproject.org/updates/F9/pending/libmthca-1.0.5-1.fc9 > for the moment. I tried libmthca-1.0.5-1.fc9.x86_64 that showed up in my mirror's update/testing/9. An iSER userspace daemon works, and the kernel no longer tells me to go update libmthca. I'll leave this library on the nodes for others to play with. Maybe this one data point counts as "testing" for Fedora? -- Pete From rdreier at cisco.com Thu Jun 19 14:44:23 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 14:44:23 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: Check completion done when a command In-Reply-To: (Roland Dreier's message of "Thu, 19 Jun 2008 08:22:48 -0700") References: <20080619141520.GB15786@mtls03> <20080619152129.GE15786@mtls03> Message-ID: > Because if we're going to merge something upstream, we might as well > merge the real fix (that fixes both mthca and mlx4) rather than merging > a workaround and then leaving the bug for another year or however long > it takes to hit it again. Oh and of course the fact that the patch requires driver code to look at internal implementation details of struct completion is a big hint that this is the wrong approach too. - R. From rdreier at cisco.com Thu Jun 19 15:04:07 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 15:04:07 -0700 Subject: [ofa-general] wait_for_completion_timeout() spurious failure under heavy load? Message-ID: It seems that the current implementaton of wait_for_completion_timeout() has a small problem under very high load for the common pattern: if (!wait_for_completion_timeout(&done, timeout)) /* handle failure */ because the implementation very roughly does (lots of code deleted to show the basic flow): static inline long __sched do_wait_for_common(struct completion *x, long timeout, int state) { if (x->done) return timeout; do { timeout = schedule_timeout(timeout); if (!timeout) return timeout; } while (!x->done); return timeout; } so if the system is very busy and x->done is not set when do_wait_for_common() is entered, it is possible that the first call to schedule_timeout() returns 0 because the task doing wait_for_completion doesn't get rescheduled for a long time, even if it is woken up early enough. In this case, wait_for_completion_timeout() returns 0 without even checking x->done again, and the code above falls into its failure case purely for scheduler reasons, even if the hardware event or whatever was being waited for happened early enough. So would it make sense to add an extra test to do_wait_for() in the timeout case and, say, return 1 if x->done is actually set? Something like the patch below? A quick audit (not exhaustive) of wait_for_completion_timeout() callers seems to indicate that no one actually cares about the return value in the success case -- they just test for 0 (timed out) versus non-zero (wait succeeded). Thanks, Roland diff --git a/kernel/sched.c b/kernel/sched.c index eaf6751..3d04ec1 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -4405,7 +4405,12 @@ do_wait_for_common(struct completion *x, long timeout, int state) spin_lock_irq(&x->wait.lock); if (!timeout) { __remove_wait_queue(&x->wait, &wait); - return timeout; + if (x->done) { + x->done--; + return 1; + } else { + return 0; + } } } while (!x->done); __remove_wait_queue(&x->wait, &wait); From rdreier at cisco.com Thu Jun 19 15:11:13 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 15:11:13 -0700 Subject: [ofa-general] Re: [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <1213628310.5447.91.camel@mtls03> (Eli Cohen's message of "Mon, 16 Jun 2008 17:58:30 +0300") References: <1213628310.5447.91.camel@mtls03> Message-ID: > /** > + * ib_create_qp_range - Creates a range of QPs associated with the > + * specified protection domain. > + * @pd: The protection domain associated with the QP. > + * @qp_init_attr: A list of initial attributes required to create the > + * QPs. If QP creation succeeds, then the attributes are updated to > + * the actual capabilities of the created QP. > + * @nqps: The number of required QPs > + * @align: Required alignment for the allocated QP numbers. > + * @list: pointer to an array to hold allocated QPs > + */ > +int ib_create_qp_range(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, > + int nqps, int align, struct ib_qp *list[]); Well, RSS for IPoIB is OK in theory but I think we really are getting to the point of adding a lot of extra complexity to the RDMA core where IPoIB is the only user. And this new interface for a QP range seems like the wrong level of abstraction -- it is exposing the mlx4 implementation details of RSS. If the next HCA that comes out has a different RSS scheme then we end up having to change all this. So the details of how RSS works should be in low-level driver code. By the way, Michael Chan from Broadcom just posted multi-rx-ring support for bnx2 to netdev. - R. From rdreier at cisco.com Thu Jun 19 15:13:08 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 15:13:08 -0700 Subject: [ofa-general] Re: mthca MR attrs userspace change In-Reply-To: <20080619194443.GA13112@osc.edu> (Pete Wyckoff's message of "Thu, 19 Jun 2008 15:44:43 -0400") References: <20080527180004.GA15444@osc.edu> <20080603204407.GA7840@osc.edu> <20080619194443.GA13112@osc.edu> Message-ID: > I tried libmthca-1.0.5-1.fc9.x86_64 that showed up in my mirror's > update/testing/9. An iSER userspace daemon works, and the kernel no > longer tells me to go update libmthca. I'll leave this library on > the nodes for others to play with. Maybe this one data point counts > as "testing" for Fedora? I think you can go to https://admin.fedoraproject.org/updates/F9/FEDORA-2008-5154 and give a +1 to the package. And I think if enough people (3?) test it it moves to F-9 proper. - R. From fogeqxas at boatcom.de Thu Jun 19 15:13:25 2008 From: fogeqxas at boatcom.de (Miguel Toney) Date: Fri, 20 Jun 2008 10:13:25 +1200 Subject: [ofa-general] Luxury Message-ID: <01c8d2be$46975080$92976dcb@fogeqxas> We are now bringing forward the replicas of the watches,Accessories, boxsets, Cufflinks, Keychains, Lighters, Pens, Tiffany & CoJewelry. A replica presupposes the exact copy of the original, that is thequality and all the other features the original possesses. For instance,youmight get a $18000 Rolex replica for a $229.00 price only! Our replica is ofsuch an amazing quality that it can be destinguished from the original only inthe jewellery workshop.You're anxiously wanting to be glamorous enough fordemanding high soiciety standarts, but your earnings are not that sky high? Youare willing to reward your lover with an outrageous gift but you don't wantthat to empty your wallet? Come, visit our shop! http://brilkean[dot]com/ Note! This link is unclickable! Copy this link and paste into your browser.When pasting don’t forget to change [dot] for ‘.’P.S When applying for a well-paid job, when flirting with agirl, when asking for a credit in the bank you get your chances doubled with ARolex President replica on your hand! PSS: The only missing thing here is a luxury suit! menireoplis poBqsyfpm7D7BqBmzlE -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Thu Jun 19 15:33:45 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 15:33:45 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support In-Reply-To: <20080612220443.3491.22057.stgit@dell3.ogc.int> (Steve Wise's message of "Thu, 12 Jun 2008 17:04:43 -0500") References: <20080612220443.3491.22057.stgit@dell3.ogc.int> Message-ID: Thanks, applied, with one minor fix: > +static int fw_supports_fastreg(struct iwch_dev *iwch_dev) ... > + return (fw_maj >= 6 && fw_min > 0); You really mean return fw_maj > 6 || (fw_maj == 6 && fw_min > 0); right? and also I changed the 'p' pointer to __be64 in iwch_build_fastreg(), but that still leaves drivers/infiniband/hw/cxgb3/iwch_qp.c:174:6: warning: incorrect type in assignment (different base types) drivers/infiniband/hw/cxgb3/iwch_qp.c:174:6: expected restricted unsigned long long [usertype] *[assigned] p drivers/infiniband/hw/cxgb3/iwch_qp.c:174:6: got unsigned long long * for the line p = &wqe->flit[1]; because flit[] is u64. What's the best way to fix this? - R. From rdreier at cisco.com Thu Jun 19 15:35:40 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 15:35:40 -0700 Subject: [ofa-general] Re: [PATCH REPOST #2] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806131655.43410.ossrosch@linux.vnet.ibm.com> (Stefan Roscher's message of "Fri, 13 Jun 2008 16:55:41 +0200") References: <200806131655.43410.ossrosch@linux.vnet.ibm.com> Message-ID: > During corner case testing, we noticed that some versions of ehca > do not properly transition to interrupt done in special load situations. > This can be resolved by periodically triggering EOI through H_EOI, > if eqes are pending. > > Signed-off-by: Stefan Roscher > --- > As firmware team suggested I moved the call of the EOI h_call into > the handler function, this ensures that we will call EOI only when we > find a valid eqe on the event queue. > Additionally I changed the calculation of the xirr value as Roland suggested. paulus / benh -- does this version still get your ack? Seems that fw team is OK with it according to Stefan... If so I will add this to my tree for 2.6.27. > diff --git a/drivers/infiniband/hw/ehca/ehca_irq.c b/drivers/infiniband/hw/ehca/ehca_irq.c > index ce1ab05..0792d93 100644 > --- a/drivers/infiniband/hw/ehca/ehca_irq.c > +++ b/drivers/infiniband/hw/ehca/ehca_irq.c > @@ -531,7 +531,7 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq) > { > struct ehca_eq *eq = &shca->eq; > struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache; > - u64 eqe_value; > + u64 eqe_value, ret; > unsigned long flags; > int eqe_cnt, i; > int eq_empty = 0; > @@ -583,8 +583,13 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq) > ehca_dbg(&shca->ib_device, > "No eqe found for irq event"); > goto unlock_irq_spinlock; > - } else if (!is_irq) > + } else if (!is_irq) { > + ret = hipz_h_eoi(eq->ist); > + if (ret != H_SUCCESS) > + ehca_err(&shca->ib_device, > + "bad return code EOI -rc = %ld\n", ret); > ehca_dbg(&shca->ib_device, "deadman found %x eqe", eqe_cnt); > + } > if (unlikely(eqe_cnt == EHCA_EQE_CACHE_SIZE)) > ehca_dbg(&shca->ib_device, "too many eqes for one irq event"); > /* enable irq for new packets */ > diff --git a/drivers/infiniband/hw/ehca/hcp_if.c b/drivers/infiniband/hw/ehca/hcp_if.c > index 5245e13..415d3a4 100644 > --- a/drivers/infiniband/hw/ehca/hcp_if.c > +++ b/drivers/infiniband/hw/ehca/hcp_if.c > @@ -933,3 +933,13 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, > r_cb, > 0, 0, 0, 0); > } > + > +u64 hipz_h_eoi(int irq) > +{ > + unsigned long xirr; > + > + iosync(); > + xirr = (0xffULL << 24) | irq; > + > + return plpar_hcall_norets(H_EOI, xirr); > +} > diff --git a/drivers/infiniband/hw/ehca/hcp_if.h b/drivers/infiniband/hw/ehca/hcp_if.h > index 60ce02b..2c3c6e0 100644 > --- a/drivers/infiniband/hw/ehca/hcp_if.h > +++ b/drivers/infiniband/hw/ehca/hcp_if.h > @@ -260,5 +260,6 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, > const u64 ressource_handle, > void *rblock, > unsigned long *byte_count); > +u64 hipz_h_eoi(int irq); > > #endif /* __HCP_IF_H__ */ > -- > 1.5.5 > > From sriram at pnl.gov Thu Jun 19 18:42:47 2008 From: sriram at pnl.gov (Krishnamoorthy, Sriram) Date: Thu, 19 Jun 2008 18:42:47 -0700 Subject: [ofa-general] IBV_WC_RETRY_EXC_ERR causes Message-ID: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF983@EMAIL02.pnl.gov> Can someone please explain what can cause IBV_WC_RETRY_EXC_ERR? I am using a combination of send-receive and RDMA. I have the reliable connection queue pairs initialized as: qp_attr.timeout = 18; qp_attr.retry_cnt = 7; qp_attr.rnr_retry = 7; >From the documentation, I assumed a value of 7 meant infinite retry. Can lack of receive buffers cause this error? I understand IBV_WC_RNR_RETRY_EXC_ERR to be the error caused by lack of receive buffers. Could it be congestion in the network? I could not find much from earlier queries related to this error. It often occurs in the middle of the computation on large (>=1024) processor counts, when I try to have multiple outstanding send-recvs between a pair of processes (each pair of processes has a RC queue pair initialized). I do not have a small test case yet that can repeat this error. Thanks for any help, Sriram.K -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Thu Jun 19 22:29:19 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 22:29:19 -0700 Subject: [ofa-general] [PATCH v2] IB/mlx4: Configure QP's max message size In-Reply-To: <20080619155251.GH15786@mtls03> (Eli Cohen's message of "Thu, 19 Jun 2008 18:52:51 +0300") References: <20080619155251.GH15786@mtls03> Message-ID: thanks, applied (although the changelog could still be improved by saying what goes wrong if we use the current code) From rdreier at cisco.com Thu Jun 19 22:31:20 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 19 Jun 2008 22:31:20 -0700 Subject: [ofa-general] Re: [PATCH] IB/mlx4: forward congestion class MADs to the HCA In-Reply-To: <20080618185317.GA8525@mtls03> (Eli Cohen's message of "Wed, 18 Jun 2008 21:53:17 +0300") References: <20080618185317.GA8525@mtls03> Message-ID: thanks, applied From jirislaby at gmail.com Thu Jun 19 23:40:49 2008 From: jirislaby at gmail.com (Jiri Slaby) Date: Fri, 20 Jun 2008 08:40:49 +0200 Subject: [ofa-general] Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: References: Message-ID: <485B50F1.2020802@gmail.com> Roland Dreier napsal(a): > It seems that the current implementaton of wait_for_completion_timeout() > has a small problem under very high load for the common pattern: > > if (!wait_for_completion_timeout(&done, timeout)) > /* handle failure */ > > because the implementation very roughly does (lots of code deleted to > show the basic flow): > > static inline long __sched > do_wait_for_common(struct completion *x, long timeout, int state) > { > if (x->done) > return timeout; > > do { > timeout = schedule_timeout(timeout); > > if (!timeout) > return timeout; > > } while (!x->done); > > return timeout; > } > > so if the system is very busy and x->done is not set when > do_wait_for_common() is entered, it is possible that the first call to > schedule_timeout() returns 0 because the task doing wait_for_completion Sorry, but how can schedule_timeout return 0 before the timeout expiration? From vlad at lists.openfabrics.org Fri Jun 20 02:25:08 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 20 Jun 2008 02:25:08 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080620-0200 daily build status Message-ID: <20080620092508.CF7C2E28ADC@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080620-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From dotanba at gmail.com Fri Jun 20 03:49:23 2008 From: dotanba at gmail.com (Dotan Barak) Date: Fri, 20 Jun 2008 12:49:23 +0200 Subject: [ofa-general] IBV_WC_RETRY_EXC_ERR causes In-Reply-To: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF983@EMAIL02.pnl.gov> References: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF983@EMAIL02.pnl.gov> Message-ID: <485B8B33.5080801@gmail.com> Krishnamoorthy, Sriram wrote: > > Can someone please explain what can cause IBV_WC_RETRY_EXC_ERR? I am > using a combination of send-receive and RDMA. I have the reliable > connection queue pairs initialized as: > IBV_WC_RETRY_EXE_ERR means that there wasn't any ack by the receiver after 4.096*(2 power 18) * 7 usec. It can happen because of several reasons: 1) bad QP attributes 2) the remote side wasn't exists or it is in bad state 3) rare, but congestion in the network can causes this too > > qp_attr.timeout = 18; > qp_attr.retry_cnt = 7; > qp_attr.rnr_retry = 7; > > From the documentation, I assumed a value of 7 meant infinite retry. > Can lack of receive buffers cause this error? I understand > IBV_WC_RNR_RETRY_EXC_ERR to be the error caused by lack of receive > buffers. Could it be congestion in the network? > 7 means infinite retry only for RNR flow, for retry flow 7 is the number of time of the retransmission. > > I could not find much from earlier queries related to this error. It > often occurs in the middle of the computation on large (>=1024) > processor counts, when I try to have multiple outstanding send-recvs > between a pair of processes (each pair of processes has a RC queue > pair initialized). I do not have a small test case yet that can repeat > this error. > How do you connect the both sides? maybe the sender send messages to QP wasn't transfered to (at least) RTR state? Dotan From mingo at elte.hu Fri Jun 20 04:20:42 2008 From: mingo at elte.hu (Ingo Molnar) Date: Fri, 20 Jun 2008 13:20:42 +0200 Subject: [ofa-general] Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: <485B50F1.2020802@gmail.com> References: <485B50F1.2020802@gmail.com> Message-ID: <20080620112042.GE7439@elte.hu> * Jiri Slaby wrote: >> do { >> timeout = schedule_timeout(timeout); >> >> if (!timeout) >> return timeout; >> >> } while (!x->done); >> >> return timeout; >> } >> >> so if the system is very busy and x->done is not set when >> do_wait_for_common() is entered, it is possible that the first call to >> schedule_timeout() returns 0 because the task doing wait_for_completion > > Sorry, but how can schedule_timeout return 0 before the timeout > expiration? the point would be that due to high load, the completion wakeup happens first, but then due to scheduling delays the timeout also occurs (later), before the wakeup related to x->done has managed to do its task. I.e. due to scheduling delays we report a spurious "timeout" failure, despite the completion occuring before the timeout. The timeout is really intended to be related to the delay of the completion event, not the delay of scheduling after that event happened. seems like a well-spotted race to me, i agree it's more robust to ignore the timeout if we can make progress on x->done, and return a 1 jiffy 'timeout remaining' value. Oleg, do you concur? but i'd do it not like this: > if (!timeout) { > __remove_wait_queue(&x->wait, &wait); > - return timeout; > + if (x->done) { > + x->done--; > + return 1; > + } else { > + return 0; > + } but like in the commit below. Agreed? Ingo --------------------> commit bb10ed0994927d433f6dbdf274fdb26cfcf516b7 Author: Roland Dreier Date: Thu Jun 19 15:04:07 2008 -0700 sched: fix wait_for_completion_timeout() spurious failure under heavy load It seems that the current implementaton of wait_for_completion_timeout() has a small problem under very high load for the common pattern: if (!wait_for_completion_timeout(&done, timeout)) /* handle failure */ because the implementation very roughly does (lots of code deleted to show the basic flow): static inline long __sched do_wait_for_common(struct completion *x, long timeout, int state) { if (x->done) return timeout; do { timeout = schedule_timeout(timeout); if (!timeout) return timeout; } while (!x->done); return timeout; } so if the system is very busy and x->done is not set when do_wait_for_common() is entered, it is possible that the first call to schedule_timeout() returns 0 because the task doing wait_for_completion doesn't get rescheduled for a long time, even if it is woken up early enough. In this case, wait_for_completion_timeout() returns 0 without even checking x->done again, and the code above falls into its failure case purely for scheduler reasons, even if the hardware event or whatever was being waited for happened early enough. It would make sense to add an extra test to do_wait_for() in the timeout case and return 1 if x->done is actually set. A quick audit (not exhaustive) of wait_for_completion_timeout() callers seems to indicate that no one actually cares about the return value in the success case -- they just test for 0 (timed out) versus non-zero (wait succeeded). Signed-off-by: Ingo Molnar diff --git a/kernel/sched.c b/kernel/sched.c index 4a3cb06..577f160 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -4405,6 +4405,16 @@ do_wait_for_common(struct completion *x, long timeout, int state) spin_unlock_irq(&x->wait.lock); timeout = schedule_timeout(timeout); spin_lock_irq(&x->wait.lock); + + /* + * If the completion has arrived meanwhile + * then return 1 jiffy time left: + */ + if (x->done && !timeout) { + timeout = 1; + break; + } + if (!timeout) { __remove_wait_queue(&x->wait, &wait); return timeout; From a.p.zijlstra at chello.nl Fri Jun 20 04:30:02 2008 From: a.p.zijlstra at chello.nl (Peter Zijlstra) Date: Fri, 20 Jun 2008 13:30:02 +0200 Subject: [ofa-general] ***SPAM*** Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: <20080620112042.GE7439@elte.hu> References: <485B50F1.2020802@gmail.com> <20080620112042.GE7439@elte.hu> Message-ID: <1213961403.3223.142.camel@lappy.programming.kicks-ass.net> On Fri, 2008-06-20 at 13:20 +0200, Ingo Molnar wrote: > * Jiri Slaby wrote: > > >> do { > >> timeout = schedule_timeout(timeout); > >> > >> if (!timeout) > >> return timeout; > >> > >> } while (!x->done); > >> > >> return timeout; > >> } > >> > >> so if the system is very busy and x->done is not set when > >> do_wait_for_common() is entered, it is possible that the first call to > >> schedule_timeout() returns 0 because the task doing wait_for_completion > > > > Sorry, but how can schedule_timeout return 0 before the timeout > > expiration? > > the point would be that due to high load, the completion wakeup happens > first, but then due to scheduling delays the timeout also occurs > (later), before the wakeup related to x->done has managed to do its > task. > > I.e. due to scheduling delays we report a spurious "timeout" failure, > despite the completion occuring before the timeout. The timeout is > really intended to be related to the delay of the completion event, not > the delay of scheduling after that event happened. > > seems like a well-spotted race to me, i agree it's more robust to ignore > the timeout if we can make progress on x->done, and return a 1 jiffy > 'timeout remaining' value. Oleg, do you concur? > > but i'd do it not like this: > > > if (!timeout) { > > __remove_wait_queue(&x->wait, &wait); > > - return timeout; > > + if (x->done) { > > + x->done--; > > + return 1; > > + } else { > > + return 0; > > + } > > but like in the commit below. Agreed? > > Ingo > > --------------------> > commit bb10ed0994927d433f6dbdf274fdb26cfcf516b7 > Author: Roland Dreier > Date: Thu Jun 19 15:04:07 2008 -0700 > > sched: fix wait_for_completion_timeout() spurious failure under heavy load > > It seems that the current implementaton of wait_for_completion_timeout() > has a small problem under very high load for the common pattern: > > if (!wait_for_completion_timeout(&done, timeout)) > /* handle failure */ > > because the implementation very roughly does (lots of code deleted to > show the basic flow): > > static inline long __sched > do_wait_for_common(struct completion *x, long timeout, int state) > { > if (x->done) > return timeout; > > do { > timeout = schedule_timeout(timeout); > > if (!timeout) > return timeout; > > } while (!x->done); > > return timeout; > } > > so if the system is very busy and x->done is not set when > do_wait_for_common() is entered, it is possible that the first call to > schedule_timeout() returns 0 because the task doing wait_for_completion > doesn't get rescheduled for a long time, even if it is woken up early > enough. > > In this case, wait_for_completion_timeout() returns 0 without even > checking x->done again, and the code above falls into its failure case > purely for scheduler reasons, even if the hardware event or whatever was > being waited for happened early enough. > > It would make sense to add an extra test to do_wait_for() in the timeout > case and return 1 if x->done is actually set. > > A quick audit (not exhaustive) of wait_for_completion_timeout() callers > seems to indicate that no one actually cares about the return value in > the success case -- they just test for 0 (timed out) versus non-zero > (wait succeeded). > > Signed-off-by: Ingo Molnar Good catch, Acked-by: Peter Zijlstra > diff --git a/kernel/sched.c b/kernel/sched.c > index 4a3cb06..577f160 100644 > --- a/kernel/sched.c > +++ b/kernel/sched.c > @@ -4405,6 +4405,16 @@ do_wait_for_common(struct completion *x, long timeout, int state) > spin_unlock_irq(&x->wait.lock); > timeout = schedule_timeout(timeout); > spin_lock_irq(&x->wait.lock); > + > + /* > + * If the completion has arrived meanwhile > + * then return 1 jiffy time left: > + */ > + if (x->done && !timeout) { > + timeout = 1; > + break; > + } > + > if (!timeout) { > __remove_wait_queue(&x->wait, &wait); > return timeout; From mdidomenico4 at gmail.com Fri Jun 20 05:14:33 2008 From: mdidomenico4 at gmail.com (Michael Di Domenico) Date: Fri, 20 Jun 2008 08:14:33 -0400 Subject: [ofa-general] ***SPAM*** set node/port description Message-ID: Is it possible to change the ib node/port description with OFED? The ports currently show up as " HCA-1". I'd like to alter the description if possible. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Fri Jun 20 06:31:59 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 06:31:59 -0700 Subject: [ofa-general] ***SPAM*** set node/port description In-Reply-To: References: Message-ID: <1213968719.13056.187.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-20 at 08:14 -0400, Michael Di Domenico wrote: > Is it possible to change the ib node/port description with OFED? The > ports currently show up as " HCA-1". I'd like to alter the > description if possible. Node Description is writable on the host. You can write to: /sys/class/infiniband//node_desc e.g. (for Mellanox HCAs) /sys/class/infiniband/mlx4_0/node_desc /sys/class/infiniband/mthca0/node_desc There's a diag script which does this: set_nodedesc.sh At older diag versions, it had a slightly different name. -- Hal > Thanks > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Fri Jun 20 06:37:46 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 16:37:46 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/include/osm_mtree.h: Eliminate unneeded include In-Reply-To: <1213798180.13056.121.camel@hrosenstock-ws.xsigo.com> References: <1213798180.13056.121.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620133746.GD3158@sashak.voltaire.com> On 07:09 Wed 18 Jun , Hal Rosenstock wrote: > opensm/include/osm_mtree.h: Eliminate unneeded include > > Also, other cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Four patches are applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 20 06:39:53 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 16:39:53 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620133953.GE3158@sashak.voltaire.com> Hi Hal, On 08:18 Wed 18 Jun , Hal Rosenstock wrote: > pensm/osm_port_profile: No need to inline osm_port_prof_is_ignored_port > and osm_port_prof_set_ignored_port functions What is the reason for doing this (and the next inline removing patch)? > Also, some cosmetic formatting changes Would be nice to not mix in one patch. Sasha From sashak at voltaire.com Fri Jun 20 06:42:01 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 16:42:01 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] libibmad/rpc.c: Better error message in madrpc_init In-Reply-To: <1213898669.13056.180.camel@hrosenstock-ws.xsigo.com> References: <1213898669.13056.180.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620134201.GF3158@sashak.voltaire.com> On 11:04 Thu 19 Jun , Hal Rosenstock wrote: > libibmad/rpc.c: Better error message in madrpc_init > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hrosenstock at xsigo.com Fri Jun 20 06:56:53 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 06:56:53 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_port.c: Cosmetic commentary changes Message-ID: <1213970213.13056.190.camel@hrosenstock-ws.xsigo.com> opensm/osm_port.c: Cosmetic commentary changes Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_port.c b/opensm/opensm/osm_port.c index b213e89..4b71fa9 100644 --- a/opensm/opensm/osm_port.c +++ b/opensm/opensm/osm_port.c @@ -756,14 +756,14 @@ osm_physp_set_pkey_tbl(IN osm_log_t * p_log, CL_ASSERT(p_pkey_tbl); /* - (14.2.5.7) - the block number valid values are 0-2047, and are further - limited by the size of the P_Key table specified by the PartitionCap on the - node. + (14.2.5.7) - the block number valid values are 0-2047, and are + further limited by the size of the P_Key table specified by + the PartitionCap on the node. */ if (!p_physp->p_node->sw || p_physp->port_num == 0) /* - The maximum blocks is defined in the node info: partition cap for CA, - routers and switch management ports. + The maximum blocks is defined in the node info: partition cap + for CA, router, and switch management ports. */ max_blocks = (cl_ntoh16(p_physp->p_node->node_info.partition_cap) + @@ -771,8 +771,9 @@ osm_physp_set_pkey_tbl(IN osm_log_t * p_log, / IB_NUM_PKEY_ELEMENTS_IN_BLOCK; else /* - This is a switch, and not a management port. The maximum blocks is defined - in the switch info: partition enforcement cap. + This is a switch, and not a management port. The maximum + blocks is defined in the switch info: partition enforcement + cap. */ max_blocks = (cl_ntoh16(p_physp->p_node->sw->switch_info.enforce_cap) + From oleg at tv-sign.ru Fri Jun 20 07:14:00 2008 From: oleg at tv-sign.ru (Oleg Nesterov) Date: Fri, 20 Jun 2008 18:14:00 +0400 Subject: [ofa-general] Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: <20080620112042.GE7439@elte.hu> References: <485B50F1.2020802@gmail.com> <20080620112042.GE7439@elte.hu> Message-ID: <20080620141400.GA411@tv-sign.ru> On 06/20, Ingo Molnar wrote: > > --- a/kernel/sched.c > +++ b/kernel/sched.c > @@ -4405,6 +4405,16 @@ do_wait_for_common(struct completion *x, long timeout, int state) > spin_unlock_irq(&x->wait.lock); > timeout = schedule_timeout(timeout); > spin_lock_irq(&x->wait.lock); > + > + /* > + * If the completion has arrived meanwhile > + * then return 1 jiffy time left: > + */ > + if (x->done && !timeout) { > + timeout = 1; > + break; > + } > + > if (!timeout) { > __remove_wait_queue(&x->wait, &wait); > return timeout; This is the real nitpick, but I wonder what is the right behaviour of wait_for_completion_timeout(x, 0) when x->done != 0. Perhaps we can return 1 in that case too, just for the consistency? IOW, how about the patch below? this also makes the code a bit simpler because we factor out __remove_wait_queue(). Oleg. --- kernel/sched.c +++ kernel/sched.c @@ -4746,15 +4746,13 @@ do_wait_for_common(struct completion *x, spin_unlock_irq(&x->wait.lock); timeout = schedule_timeout(timeout); spin_lock_irq(&x->wait.lock); - if (!timeout) { - __remove_wait_queue(&x->wait, &wait); - return timeout; - } - } while (!x->done); + } while (!x->done && timeout); __remove_wait_queue(&x->wait, &wait); + if (!x->done) + return 0; } x->done--; - return timeout; + return timeout ?: 1; } static long __sched From hrosenstock at xsigo.com Fri Jun 20 07:24:43 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 07:24:43 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_switch: Cosmetic formatting changes Message-ID: <1213971883.13056.194.camel@hrosenstock-ws.xsigo.com> opensm/osm_switch: Cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index 90119b9..1c0f6e9 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -251,7 +251,6 @@ osm_switch_get_hop_count(IN const osm_switch_t * const p_sw, return (lid_ho > p_sw->max_lid_ho || !p_sw->hops[lid_ho]) ? OSM_NO_PATH : p_sw->hops[lid_ho][port_num]; } - /* * PARAMETERS * p_sw @@ -285,7 +284,6 @@ static inline osm_fwd_tbl_t *osm_switch_get_fwd_tbl_ptr(IN const osm_switch_t * { return ((osm_fwd_tbl_t *) & p_sw->fwd_tbl); } - /* * PARAMETERS * p_sw @@ -371,7 +369,6 @@ osm_switch_get_least_hops(IN const osm_switch_t * const p_sw, return (lid_ho > p_sw->max_lid_ho || !p_sw->hops[lid_ho]) ? OSM_NO_PATH : p_sw->hops[lid_ho][0]; } - /* * PARAMETERS * p_sw @@ -437,7 +434,6 @@ osm_switch_get_port_by_lid(IN const osm_switch_t * const p_sw, { return (osm_fwd_tbl_get(&p_sw->fwd_tbl, lid_ho)); } - /* * PARAMETERS * p_sw @@ -517,7 +513,6 @@ static inline osm_physp_t *osm_switch_get_route_by_lid(IN const osm_switch_t * else return NULL; } - /* * PARAMETERS * p_sw @@ -554,7 +549,6 @@ osm_switch_sp0_is_lmc_capable(IN const osm_switch_t * const p_sw, return (p_subn->opt.lmc_esp0 && ib_switch_info_is_enhanced_port0(&p_sw->switch_info)) ? 1 : 0; } - /* * PARAMETERS * p_sw @@ -588,7 +582,6 @@ osm_switch_get_max_block_id(IN const osm_switch_t * const p_sw) return ((uint32_t) (osm_fwd_tbl_get_size(&p_sw->fwd_tbl) / osm_fwd_tbl_get_lids_per_block(&p_sw->fwd_tbl))); } - /* * PARAMETERS * p_sw @@ -620,7 +613,6 @@ osm_switch_get_max_block_id_in_use(IN const osm_switch_t * const p_sw) cl_ntoh16(p_sw->switch_info. lin_top))); } - /* * PARAMETERS * p_sw @@ -684,7 +676,6 @@ osm_switch_supports_mcast(IN const osm_switch_t * const p_sw) { return (p_sw->switch_info.mcast_cap != 0); } - /* * PARAMETERS * p_sw @@ -716,7 +707,6 @@ osm_switch_set_switch_info(IN osm_switch_t * const p_sw, CL_ASSERT(p_si); p_sw->switch_info = *p_si; } - /* * PARAMETERS * p_sw @@ -747,7 +737,6 @@ osm_switch_count_path(IN osm_switch_t * const p_sw, IN const uint8_t port) { osm_port_prof_path_count_inc(&p_sw->p_prof[port]); } - /* * PARAMETERS * p_sw @@ -781,7 +770,6 @@ osm_switch_set_ft_block(IN osm_switch_t * const p_sw, CL_ASSERT(p_sw); return (osm_fwd_tbl_set_block(&p_sw->fwd_tbl, p_block, block_num)); } - /* * PARAMETERS * p_sw @@ -819,7 +807,6 @@ osm_switch_set_mft_block(IN osm_switch_t * const p_sw, return (osm_mcast_tbl_set_block(&p_sw->mcast_tbl, p_block, block_num, position)); } - /* * PARAMETERS * p_sw @@ -861,7 +848,6 @@ osm_switch_get_mft_block(IN osm_switch_t * const p_sw, return (osm_mcast_tbl_get_block(&p_sw->mcast_tbl, block_num, position, p_block)); } - /* * PARAMETERS * p_sw @@ -901,7 +887,6 @@ osm_switch_get_mft_max_block(IN osm_switch_t * const p_sw) CL_ASSERT(p_sw); return (osm_mcast_tbl_get_max_block(&p_sw->mcast_tbl)); } - /* * PARAMETERS * p_sw @@ -925,7 +910,6 @@ osm_switch_get_mft_max_block_in_use(IN osm_switch_t * const p_sw) CL_ASSERT(p_sw); return (osm_mcast_tbl_get_max_block_in_use(&p_sw->mcast_tbl)); } - /* * PARAMETERS * p_sw @@ -955,7 +939,6 @@ osm_switch_get_mft_max_position(IN osm_switch_t * const p_sw) CL_ASSERT(p_sw); return (osm_mcast_tbl_get_max_position(&p_sw->mcast_tbl)); } - /* * PARAMETERS * p_sw @@ -1070,7 +1053,6 @@ osm_switch_get_fwd_tbl_size(IN const osm_switch_t * const p_sw) { return (osm_fwd_tbl_get_size(&p_sw->fwd_tbl)); } - /* * PARAMETERS * p_sw @@ -1098,7 +1080,6 @@ osm_switch_get_mcast_fwd_tbl_size(IN const osm_switch_t * const p_sw) { return (cl_ntoh16(p_sw->switch_info.mcast_cap)); } - /* * PARAMETERS * p_sw @@ -1127,7 +1108,6 @@ osm_switch_path_count_get(IN const osm_switch_t * const p_sw, { return (osm_port_prof_path_count_get(&p_sw->p_prof[port_num])); } - /* * PARAMETERS * p_sw @@ -1186,7 +1166,6 @@ static inline osm_mcast_tbl_t *osm_switch_get_mcast_tbl_ptr(IN const { return ((osm_mcast_tbl_t *) & p_sw->mcast_tbl); } - /* * PARAMETERS * p_sw @@ -1222,7 +1201,6 @@ osm_switch_is_in_mcast_tree(IN const osm_switch_t * const p_sw, else return (FALSE); } - /* * PARAMETERS * p_sw diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index a9d13c8..1fd91ee 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -370,13 +370,15 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, osm_switch_get_hop_count(p_sw, base_lid, port_num); /* - If we aren't using pre-defined user routes function, then - we need to make sure that the current path is the minimum one. - In case of having such a user function - this check will not - be done, and the old routing will be used. - Note: This means that it is the user's job to clean all data - in the forwarding tables that he wants to be overridden by the - minimum hop function. + If we aren't using pre-defined user routes + function, then we need to make sure that the + current path is the minimum one. In case of + having such a user function - this check will + not be done, and the old routing will be used. + Note: This means that it is the user's job to + clean all data in the forwarding tables that + he wants to be overridden by the minimum + hop function. */ if (hops == least_hops) return (port_num); From oleg at tv-sign.ru Fri Jun 20 07:32:20 2008 From: oleg at tv-sign.ru (Oleg Nesterov) Date: Fri, 20 Jun 2008 18:32:20 +0400 Subject: [ofa-general] Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: <20080620141400.GA411@tv-sign.ru> References: <485B50F1.2020802@gmail.com> <20080620112042.GE7439@elte.hu> <20080620141400.GA411@tv-sign.ru> Message-ID: <20080620143220.GA441@tv-sign.ru> On 06/20, Oleg Nesterov wrote: > > On 06/20, Ingo Molnar wrote: > > > > --- a/kernel/sched.c > > +++ b/kernel/sched.c > > @@ -4405,6 +4405,16 @@ do_wait_for_common(struct completion *x, long timeout, int state) > > spin_unlock_irq(&x->wait.lock); > > timeout = schedule_timeout(timeout); > > spin_lock_irq(&x->wait.lock); > > + > > + /* > > + * If the completion has arrived meanwhile > > + * then return 1 jiffy time left: > > + */ > > + if (x->done && !timeout) { > > + timeout = 1; > > + break; > > + } > > + > > if (!timeout) { > > __remove_wait_queue(&x->wait, &wait); > > return timeout; > > This is the real nitpick, but I wonder what is the right behaviour > of wait_for_completion_timeout(x, 0) when x->done != 0. Perhaps we > can return 1 in that case too, just for the consistency? > > IOW, how about the patch below? this also makes the code a bit > simpler because we factor out __remove_wait_queue(). Even better, we can kill the first __remove_wait_queue() as well. Oleg. --- kernel/sched.c +++ kernel/sched.c @@ -4739,22 +4739,20 @@ do_wait_for_common(struct completion *x, signal_pending(current)) || (state == TASK_KILLABLE && fatal_signal_pending(current))) { - __remove_wait_queue(&x->wait, &wait); - return -ERESTARTSYS; + timeout = -ERESTARTSYS; + break; } __set_current_state(state); spin_unlock_irq(&x->wait.lock); timeout = schedule_timeout(timeout); spin_lock_irq(&x->wait.lock); - if (!timeout) { - __remove_wait_queue(&x->wait, &wait); - return timeout; - } - } while (!x->done); + } while (!x->done && timeout); __remove_wait_queue(&x->wait, &wait); + if (!x->done) + return timeout; } x->done--; - return timeout; + return timeout ?: 1; } static long __sched From hrosenstock at xsigo.com Fri Jun 20 07:34:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 07:34:38 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_subnet.h: Cosmetic formatting changes Message-ID: <1213972478.13056.197.camel@hrosenstock-ws.xsigo.com> opensm/osm_subnet.h: Cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index cbc1596..ef9996c 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -223,7 +223,7 @@ typedef struct osm_subn_opt { * The port guid that the SM is binding to. * * m_key -* M_Key value sent to all ports qualifing all Set(PortInfo). +* M_Key value sent to all ports qualifying all Set(PortInfo). * * sm_key * SM_Key value of the SM used for SM authentication. @@ -313,7 +313,8 @@ typedef struct osm_subn_opt { * * packet_life_time * The maximal time a packet can stay in a switch. -* The value is send to all switches as SubnSet(SwitchInfo.life_state) +* The value is send to all switches as +* SubnSet(SwitchInfo.life_state) * * dump_files_dir * The directory to be used for opensm-subnet.lst, opensm.fdbs, @@ -337,7 +338,8 @@ typedef struct osm_subn_opt { * * accum_log_file * If TRUE (default) - the log file will be accumulated. -* If FALSE - the log file will be erased before starting current opensm run. +* If FALSE - the log file will be erased before starting +* current opensm run. * * port_prof_ignore_file * Name of file with port guids to be ignored by port profiling. @@ -382,7 +384,8 @@ typedef struct osm_subn_opt { * Name of the SA database file. * * exit_on_fatal -* If TRUE (default) - SM will exit on fatal subnet initialization issues. +* If TRUE (default) - SM will exit on fatal subnet initialization +* issues. * If FALSE - SM will not exit. * Fatal initialization issues: * a. SM recognizes 2 different nodes with the same guid, or @@ -666,8 +669,8 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn); * NOTES * Performs any necessary cleanup of the specified Subnet object. * Further operations should not be attempted on the destroyed object. -* This function should only be called after a call to osm_subn_construct or -* osm_subn_init. +* This function should only be called after a call to osm_subn_construct +* or osm_subn_init. * * SEE ALSO * Subnet object, osm_subn_construct, osm_subn_init From rdreier at cisco.com Fri Jun 20 07:33:02 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 07:33:02 -0700 Subject: [ofa-general] Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: <20080620112042.GE7439@elte.hu> (Ingo Molnar's message of "Fri, 20 Jun 2008 13:20:42 +0200") References: <485B50F1.2020802@gmail.com> <20080620112042.GE7439@elte.hu> Message-ID: > but like in the commit below. Agreed? > + > + /* > + * If the completion has arrived meanwhile > + * then return 1 jiffy time left: > + */ > + if (x->done && !timeout) { > + timeout = 1; > + break; > + } > + Sure, that looks nice to me. - R. From swise at opengridcomputing.com Fri Jun 20 08:18:21 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 20 Jun 2008 10:18:21 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support In-Reply-To: References: <20080612220443.3491.22057.stgit@dell3.ogc.int> Message-ID: <485BCA3D.1010903@opengridcomputing.com> Roland Dreier wrote: > Thanks, applied, with one minor fix: > > > +static int fw_supports_fastreg(struct iwch_dev *iwch_dev) > > ... > > > + return (fw_maj >= 6 && fw_min > 0); > > You really mean > > return fw_maj > 6 || (fw_maj == 6 && fw_min > 0); > > right? > > yes. > and also I changed the 'p' pointer to __be64 in iwch_build_fastreg(), > but that still leaves > > drivers/infiniband/hw/cxgb3/iwch_qp.c:174:6: warning: incorrect type in assignment (different base types) > drivers/infiniband/hw/cxgb3/iwch_qp.c:174:6: expected restricted unsigned long long [usertype] *[assigned] p > drivers/infiniband/hw/cxgb3/iwch_qp.c:174:6: got unsigned long long * > > for the line > > p = &wqe->flit[1]; > > because flit[] is u64. What's the best way to fix this? > > I think flit[] should be __be64. From mingo at elte.hu Fri Jun 20 08:21:24 2008 From: mingo at elte.hu (Ingo Molnar) Date: Fri, 20 Jun 2008 17:21:24 +0200 Subject: [ofa-general] Re: wait_for_completion_timeout() spurious failure under heavy load? In-Reply-To: <20080620143220.GA441@tv-sign.ru> References: <485B50F1.2020802@gmail.com> <20080620112042.GE7439@elte.hu> <20080620141400.GA411@tv-sign.ru> <20080620143220.GA441@tv-sign.ru> Message-ID: <20080620152124.GC17373@elte.hu> * Oleg Nesterov wrote: > > IOW, how about the patch below? this also makes the code a bit > > simpler because we factor out __remove_wait_queue(). > > Even better, we can kill the first __remove_wait_queue() as well. nice, thanks - applied it in the form below to tip/sched/urgent. Ingo ------------------> commit 6b8464474776dccf619283ee5510b0b795382dfb Author: Oleg Nesterov Date: Fri Jun 20 18:32:20 2008 +0400 sched: refactor wait_for_completion_timeout() Simplify the code and fix the boundary condition of wait_for_completion_timeout(,0). We can kill the first __remove_wait_queue() as well. Signed-off-by: Ingo Molnar diff --git a/kernel/sched.c b/kernel/sched.c index 577f160..bebf978 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -4398,32 +4398,20 @@ do_wait_for_common(struct completion *x, long timeout, int state) signal_pending(current)) || (state == TASK_KILLABLE && fatal_signal_pending(current))) { - __remove_wait_queue(&x->wait, &wait); - return -ERESTARTSYS; + timeout = -ERESTARTSYS; + break; } __set_current_state(state); spin_unlock_irq(&x->wait.lock); timeout = schedule_timeout(timeout); spin_lock_irq(&x->wait.lock); - - /* - * If the completion has arrived meanwhile - * then return 1 jiffy time left: - */ - if (x->done && !timeout) { - timeout = 1; - break; - } - - if (!timeout) { - __remove_wait_queue(&x->wait, &wait); - return timeout; - } - } while (!x->done); + } while (!x->done && timeout); __remove_wait_queue(&x->wait, &wait); + if (!x->done) + return timeout; } x->done--; - return timeout; + return timeout ?: 1; } static long __sched From sashak at voltaire.com Fri Jun 20 08:42:35 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 18:42:35 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_subnet.h: Cosmetic formatting changes In-Reply-To: <1213972478.13056.197.camel@hrosenstock-ws.xsigo.com> References: <1213972478.13056.197.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620154235.GI3158@sashak.voltaire.com> On 07:34 Fri 20 Jun , Hal Rosenstock wrote: > opensm/osm_subnet.h: Cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Recent three applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 20 09:13:40 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 19:13:40 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <48566664.1010004@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> Message-ID: <20080620161340.GJ3158@sashak.voltaire.com> Hi Yevgeny, On 16:11 Mon 16 Jun , Yevgeny Kliteynik wrote: > > Adding log message for matched criteria of > the QoS policy rule. Could you explain what is the purpose of this code? And how it should be useful in run-time? I'm asking because for me it looks like "pure" debug stuff which tries to debug qos rules matching code. Sasha From vfpresident at claxson.com.uy Fri Jun 20 09:15:58 2008 From: vfpresident at claxson.com.uy (Anne Lento) Date: Fri, 20 Jun 2008 11:15:58 -0500 Subject: [ofa-general] Exclusive man's health Message-ID: <000f01c8d2c7$03aa33b0$01cc8b04@grisales> High-quality low-priced medicines ED medicines ! Holidays special for the your size as yours and health! New business is opened for a day! In our LICENSED pharmacies buy medicines from the warehouse! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Fri Jun 20 09:36:22 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 09:36:22 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support In-Reply-To: <485BCA3D.1010903@opengridcomputing.com> (Steve Wise's message of "Fri, 20 Jun 2008 10:18:21 -0500") References: <20080612220443.3491.22057.stgit@dell3.ogc.int> <485BCA3D.1010903@opengridcomputing.com> Message-ID: > > p = &wqe->flit[1]; > I think flit[] should be __be64. actually would it make more sense to add a t3_fastreg_pbl_wr or something like that to the union for the continuation into a second WQE? - R. From rdreier at cisco.com Fri Jun 20 09:39:13 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 09:39:13 -0700 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: <485A77AD.4010009@voltaire.com> (Or Gerlitz's message of "Thu, 19 Jun 2008 18:13:49 +0300") References: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> <484E9264.4070708@voltaire.com> <485A77AD.4010009@voltaire.com> Message-ID: > In the sockets world the sender sets the IP_MULTICAST_LOOP socket > option which ip(7) says that it "Sets or reads a boolean integer > argument whether sent multicast packets should be looped back to the > local sockets." so one more point here is that the suggested > implementation of setting this at the qp creation is one that brings > the API closer to the socket one, which can be easier for users to > understand. OK, that is a reasonably convincing argument. > Roland, can you please comment on your preferences, that is on the cap > bit vs hint and where you want the app to direct the HW how to behave, > in the qp creation time as we suggest or on the time of attach. If you > want to go the changing the qp attach verb way, can please provide us > with some implementation directions regarding backward compatibility > with apps that are built against versions of libibverbs which have > different API for this verb, thanks. I guess the implementation you posted is OK. But I don't know how to expose this to userspace anyway, since no one has figured out the QP flags issues as far as I know. - R. From rdreier at cisco.com Fri Jun 20 09:42:25 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 09:42:25 -0700 Subject: [ofa-general] Re: [PATCH] IB/core - reset to error state transition not allowed In-Reply-To: <1208972583.2232.107.camel@brick.pathscale.com> (Ralph Campbell's message of "Wed, 23 Apr 2008 10:43:03 -0700") References: <1208972583.2232.107.camel@brick.pathscale.com> Message-ID: thanks, I applied this for 2.6.27 From swise at opengridcomputing.com Fri Jun 20 09:46:01 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 20 Jun 2008 11:46:01 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support In-Reply-To: References: <20080612220443.3491.22057.stgit@dell3.ogc.int> <485BCA3D.1010903@opengridcomputing.com> Message-ID: <485BDEC9.6040903@opengridcomputing.com> Roland Dreier wrote: > > > p = &wqe->flit[1]; > > > I think flit[] should be __be64. > > actually would it make more sense to add a t3_fastreg_pbl_wr or > something like that to the union for the continuation into a second WQE? > > - R. > or t3_pbl_fragment But flit[] should really still be __be64. Lemme resolve these issues and build with C=1 and CF=-D__CHECK_INDIAN__ and I'll repost. Thanks, Stevo From rdreier at cisco.com Fri Jun 20 09:51:25 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 09:51:25 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support In-Reply-To: <485BDEC9.6040903@opengridcomputing.com> (Steve Wise's message of "Fri, 20 Jun 2008 11:46:01 -0500") References: <20080612220443.3491.22057.stgit@dell3.ogc.int> <485BCA3D.1010903@opengridcomputing.com> <485BDEC9.6040903@opengridcomputing.com> Message-ID: > or t3_pbl_fragment sure... names aren't always my strong suit ;) > But flit[] should really still be __be64. OK... I see another places where it gets assigned a wrid: wqe->flit[T3_SQ_COOKIE_FLIT] = mw_bind->wr_id; and mw_bind->wr_id is u64. So I think that's the other place you'll have to fix up. - R. From rdreier at cisco.com Fri Jun 20 09:52:02 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 09:52:02 -0700 Subject: [ofa-general] Re: [PATCH] IB/core - reset to error state transition not allowed In-Reply-To: (Roland Dreier's message of "Fri, 20 Jun 2008 09:42:25 -0700") References: <1208972583.2232.107.camel@brick.pathscale.com> Message-ID: By the way, I also added these follow-up patches to remove now-unneeded code from mthca and mlx4: commit 97621c3f6ccdee3e4927346dfc8315397c1e4f5b Author: Roland Dreier Date: Fri Jun 20 09:47:42 2008 -0700 IB/mthca: Remove extra code for RESET->ERR QP state transition Commit b18aad71 ("IB/mthca: Fix RESET to ERROR transition") added some extra code to handle a QP state transition from RESET to ERROR. However, the latest 1.2.1 version of the IB spec has clarified that this transition is actually not allowed, so we can remove this extra code again. Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/hw/mthca/mthca_qp.c b/drivers/infiniband/hw/mthca/mthca_qp.c index 3b1c5ba..77b52b3 100644 --- a/drivers/infiniband/hw/mthca/mthca_qp.c +++ b/drivers/infiniband/hw/mthca/mthca_qp.c @@ -848,23 +848,6 @@ out: return err; } -static const struct ib_qp_attr dummy_init_attr = { .port_num = 1 }; -static const int dummy_init_attr_mask[] = { - [IB_QPT_UD] = (IB_QP_PKEY_INDEX | - IB_QP_PORT | - IB_QP_QKEY), - [IB_QPT_UC] = (IB_QP_PKEY_INDEX | - IB_QP_PORT | - IB_QP_ACCESS_FLAGS), - [IB_QPT_RC] = (IB_QP_PKEY_INDEX | - IB_QP_PORT | - IB_QP_ACCESS_FLAGS), - [IB_QPT_SMI] = (IB_QP_PKEY_INDEX | - IB_QP_QKEY), - [IB_QPT_GSI] = (IB_QP_PKEY_INDEX | - IB_QP_QKEY), -}; - int mthca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask, struct ib_udata *udata) { @@ -926,15 +909,6 @@ int mthca_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask, goto out; } - if (cur_state == IB_QPS_RESET && new_state == IB_QPS_ERR) { - err = __mthca_modify_qp(ibqp, &dummy_init_attr, - dummy_init_attr_mask[ibqp->qp_type], - IB_QPS_RESET, IB_QPS_INIT); - if (err) - goto out; - cur_state = IB_QPS_INIT; - } - err = __mthca_modify_qp(ibqp, attr, attr_mask, cur_state, new_state); out: commit 1387b8c045dc59b1ef98e710d09bc5683c79c56d Author: Roland Dreier Date: Fri Jun 20 09:49:25 2008 -0700 IB/mlx4: Remove extra code for RESET->ERR QP state transition Commit 65adfa91 ("IB/mlx4: Fix RESET to RESET and RESET to ERROR transitions") added some extra code to handle a QP state transition from RESET to ERROR. However, the latest 1.2.1 version of the IB spec has clarified that this transition is actually not allowed, so we can remove this extra code again. Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c index 76054a7..44bbd6c 100644 --- a/drivers/infiniband/hw/mlx4/qp.c +++ b/drivers/infiniband/hw/mlx4/qp.c @@ -1132,23 +1132,6 @@ out: return err; } -static const struct ib_qp_attr mlx4_ib_qp_attr = { .port_num = 1 }; -static const int mlx4_ib_qp_attr_mask_table[IB_QPT_UD + 1] = { - [IB_QPT_UD] = (IB_QP_PKEY_INDEX | - IB_QP_PORT | - IB_QP_QKEY), - [IB_QPT_UC] = (IB_QP_PKEY_INDEX | - IB_QP_PORT | - IB_QP_ACCESS_FLAGS), - [IB_QPT_RC] = (IB_QP_PKEY_INDEX | - IB_QP_PORT | - IB_QP_ACCESS_FLAGS), - [IB_QPT_SMI] = (IB_QP_PKEY_INDEX | - IB_QP_QKEY), - [IB_QPT_GSI] = (IB_QP_PKEY_INDEX | - IB_QP_QKEY), -}; - int mlx4_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, int attr_mask, struct ib_udata *udata) { @@ -1191,15 +1174,6 @@ int mlx4_ib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr, goto out; } - if (cur_state == IB_QPS_RESET && new_state == IB_QPS_ERR) { - err = __mlx4_ib_modify_qp(ibqp, &mlx4_ib_qp_attr, - mlx4_ib_qp_attr_mask_table[ibqp->qp_type], - IB_QPS_RESET, IB_QPS_INIT); - if (err) - goto out; - cur_state = IB_QPS_INIT; - } - err = __mlx4_ib_modify_qp(ibqp, attr, attr_mask, cur_state, new_state); out: From rdreier at cisco.com Fri Jun 20 10:26:34 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 20 Jun 2008 10:26:34 -0700 Subject: [ofa-general] Re: [PATCH] IB/ehca: Reject recv WRs if QP is in RESET state In-Reply-To: <200806091324.20869.fenkes@de.ibm.com> (Joachim Fenkes's message of "Mon, 9 Jun 2008 13:24:20 +0200") References: <200806061835.43802.fenkes@de.ibm.com> <48499C11.7030504@gmail.com> <200806091324.20869.fenkes@de.ibm.com> Message-ID: thanks, applied. From swise at opengridcomputing.com Fri Jun 20 10:53:22 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 20 Jun 2008 12:53:22 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fix up some of the cxgb3 ib_device_attr fields. Message-ID: <20080620175322.27941.64381.stgit@dell3.ogc.int> - set fw_ver - set hw_ver - set max_qp_wr to something reasonable - set max_cqe to something reasonable Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.h | 3 ++- drivers/infiniband/hw/cxgb3/iwch.c | 4 ++-- drivers/infiniband/hw/cxgb3/iwch_provider.c | 21 +++++++++++++++++++++ 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 4bb5926..cdc6ac9 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -45,13 +45,14 @@ #define T3_CTRL_QP_SIZE_LOG2 8 #define T3_CTRL_CQ_ID 0 -/* TBD */ #define T3_MAX_NUM_RI (1<<15) #define T3_MAX_NUM_QP (1<<15) #define T3_MAX_NUM_CQ (1<<15) #define T3_MAX_NUM_PD (1<<15) #define T3_MAX_PBL_SIZE 256 #define T3_MAX_RQ_SIZE 1024 +#define T3_MAX_QP_DEPTH (T3_MAX_RQ_SIZE-1) +#define T3_MAX_CQ_DEPTH 8192 #define T3_MAX_NUM_STAG (1<<15) #define T3_MAX_MR_SIZE 0x100000000ULL diff --git a/drivers/infiniband/hw/cxgb3/iwch.c b/drivers/infiniband/hw/cxgb3/iwch.c index 71554ea..e09cc1a 100644 --- a/drivers/infiniband/hw/cxgb3/iwch.c +++ b/drivers/infiniband/hw/cxgb3/iwch.c @@ -74,11 +74,11 @@ static void rnic_init(struct iwch_dev *rnicp) rnicp->attr.vendor_id = 0x168; rnicp->attr.vendor_part_id = 7; rnicp->attr.max_qps = T3_MAX_NUM_QP - 32; - rnicp->attr.max_wrs = (1UL << 24) - 1; + rnicp->attr.max_wrs = T3_MAX_QP_DEPTH; rnicp->attr.max_sge_per_wr = T3_MAX_SGE; rnicp->attr.max_sge_per_rdma_write_wr = T3_MAX_SGE; rnicp->attr.max_cqs = T3_MAX_NUM_CQ - 1; - rnicp->attr.max_cqes_per_cq = (1UL << 24) - 1; + rnicp->attr.max_cqes_per_cq = T3_MAX_CQ_DEPTH; rnicp->attr.max_mem_regs = cxio_num_stags(&rnicp->rdev); rnicp->attr.max_phys_buf_entries = T3_MAX_PBL_SIZE; rnicp->attr.max_pds = T3_MAX_NUM_PD - 1; diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 2c1c477..5f61c3c 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1093,6 +1093,25 @@ static int iwch_query_gid(struct ib_device *ibdev, u8 port, return 0; } +static u64 fw_vers_string_to_u64(struct iwch_dev *iwch_dev) +{ + struct ethtool_drvinfo info; + struct net_device *lldev = iwch_dev->rdev.t3cdev_p->lldev; + char *cp, *next; + unsigned fw_maj, fw_min, fw_mic; + + lldev->ethtool_ops->get_drvinfo(lldev, &info); + next = info.fw_version+1; + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_maj); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_min); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_mic); + return (((u64)fw_maj & 0xffff) << 32) | ((fw_min & 0xffff) << 16) | + (fw_mic & 0xffff); +} + static int iwch_query_device(struct ib_device *ibdev, struct ib_device_attr *props) { @@ -1103,6 +1122,8 @@ static int iwch_query_device(struct ib_device *ibdev, dev = to_iwch_dev(ibdev); memset(props, 0, sizeof *props); memcpy(&props->sys_image_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); + props->hw_ver = dev->rdev.t3cdev_p->type + props->fw_ver = fw_vers_string_to_u64(dev); props->device_cap_flags = dev->device_cap_flags; props->vendor_id = (u32)dev->rdev.rnic_info.pdev->vendor; props->vendor_part_id = (u32)dev->rdev.rnic_info.pdev->device; From sashak at voltaire.com Fri Jun 20 10:58:33 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 20:58:33 +0300 Subject: [ofa-general] Re: [PATCHv2] ibsim: Support short RMPP packets In-Reply-To: <1213886129.13056.168.camel@hrosenstock-ws.xsigo.com> References: <1213886129.13056.168.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620175833.GK3158@sashak.voltaire.com> On 07:35 Thu 19 Jun , Hal Rosenstock wrote: > ibsim: Support for short RMPP packets (up thru 256 bytes total) > > Changes from v1: > - Added length to simulation header (by reducing context size so as not > to break ABI) and encoded length on sends and decoded and checked > lengths on receive RMPP supports transactions up to 2^^32 256 byte packets in size. So length must be more that 32-bit (or later we will need to break ABI). Also it should transferred in network byte order. > - Restored short/partial packet read check in umad2sim_read > - In umad2sim_write, fixed setting of umad->length > - Eliminated sim_req256 typedef Entire patch still be huge for what it tries to accomplish, I think it is mostly due to "to malloc()" and data types conversions and it still unclear for me why it was needed if packets are limited in size anyway. For me it looks that something simpler like a patch below will do the same. What do you think? Sasha diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 675d95b..6c5add9 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1306,7 +1306,7 @@ int send_trap(Port * port, int trapnum) req.sqp = 0; req.dqp = 0; req.status = 0; - req.context = 0; + req.length = htonll(sizeof(req.mad)); // find SM client cl = find_client(destport, 0, 1, 0); diff --git a/include/ibsim.h b/include/ibsim.h index 84568e6..ba1e3dd 100644 --- a/include/ibsim.h +++ b/include/ibsim.h @@ -67,7 +67,7 @@ struct sim_request { uint32_t dqp; uint32_t sqp; uint32_t status; - uint64_t context; + uint64_t length; char mad[256]; }; diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c index 4cbf8da..f8fa50a 100644 --- a/umad2sim/umad2sim.c +++ b/umad2sim/umad2sim.c @@ -411,7 +411,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) umad->status = ntohl(req.status); umad->timeout_ms = 0; umad->retries = 0; - umad->length = umad_size() + sizeof(req.mad); + umad->length = umad_size() + ntohll(req.length); umad->addr.qpn = req.sqp; umad->addr.qkey = 0; // agent->qkey; @@ -470,13 +470,14 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, req.dqp = umad->addr.qpn; req.sqp = htonl(dev->agents[umad->agent_id].qpn); req.status = 0; - req.context = 0; cnt = count - umad_size(); if (cnt > sizeof(req.mad)) cnt = sizeof(req.mad); memcpy(req.mad, umad_get_mad(umad), cnt); + req.length = htonll(cnt); + cnt = write(dev->sim_client.fd_pktout, (void *)&req, sizeof(req)); if (cnt < 0) { ERROR("umad2sim_write: cannot write\n"); From swise at opengridcomputing.com Fri Jun 20 11:05:43 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 20 Jun 2008 13:05:43 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: MEM_MGT_EXTENSIONS support In-Reply-To: References: <20080612220443.3491.22057.stgit@dell3.ogc.int> <485BCA3D.1010903@opengridcomputing.com> <485BDEC9.6040903@opengridcomputing.com> Message-ID: <485BF177.9090804@opengridcomputing.com> Roland Dreier wrote: > > or t3_pbl_fragment > > sure... names aren't always my strong suit ;) > > > But flit[] should really still be __be64. > > OK... I see another places where it gets assigned a wrid: > > wqe->flit[T3_SQ_COOKIE_FLIT] = mw_bind->wr_id; > > Actually that line can be removed. The wr_id is really stored in the sw sq. Steve. From gstreiff at NetEffect.com Fri Jun 20 11:40:15 2008 From: gstreiff at NetEffect.com (Glenn Streiff) Date: Fri, 20 Jun 2008 13:40:15 -0500 Subject: [Fwd: [ofa-general] ofa_1_4_kernel 20080616-0200 daily build status] In-Reply-To: <5E701717F2B2ED4EA60F87C8AA57B7CC0795026C@venom2> Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC0795026D@venom2> Hi, Vlad. Noticed the daily build for ofed1.4 was still broken. Since ofed_1_4/linux-2.6.git hasn't been updated in a couple days, I assume you were not able to pick up the commit below. Let us know if there is anything more to do on the NetEffect side. Thanks, Glenn > -----Original Message----- > From: Glenn Streiff > Sent: Thursday, June 19, 2008 9:41 PM > To: 'vlad at mellanox.co.il'; Chien Tung > Subject: RE: [Fwd: [ofa-general] ofa_1_4_kernel 20080616-0200 daily > build status] > > > > > Hello Glenn, > > Please arrange backport patches for NES driver for OFED-1.4. > > > > git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git > > git_branch: ofed_kernel > > > > See attached email for details. > > > > Regards, > > Vladimir > > > > Vlad, > > Please pull from: > > git://git.openfabrics.org/~glenn/ofed-1.4.git ofed_kernel > > Chien Tung(1) > commit c0afe5ef5661bcf8925dc47d4c1100eb26b1e72b > RDMA/nes: Add ofed1.4 backports > > So heads up on the new naming convention for neteffect git > tree...I liked Steve's convention. > > Maybe this is a script impact for you for neteffect git_url? > > The 1.3 tree is still ~glenn/scm/linux-2.6.git for the moment, > but I should probably change that to be uniform: > ~glenn/scm/ofed-1.3.git > I'm thinking git is cool with me just doing a directory rename > on this, right? Then I just need to get gitweb updated to reflect the > new name. > > Glenn > > Snipped out all packages but --with-core-mod and --with-nes-mod to > get a clean build. > > Passed on i686 with linux-2.6.13 > Passed on i686 with linux-2.6.12 > Passed on i686 with linux-2.6.14 > Passed on i686 with linux-2.6.15 > Passed on i686 with linux-2.6.16 > Passed on i686 with linux-2.6.18 > Passed on i686 with linux-2.6.19 > Passed on i686 with linux-2.6.17 > Passed on i686 with linux-2.6.22 > Passed on i686 with linux-2.6.21.1 > Passed on x86_64 with linux-2.6.12 > Passed on x86_64 with linux-2.6.14 > Passed on x86_64 with linux-2.6.13 > Passed on x86_64 with linux-2.6.15 > Passed on x86_64 with linux-2.6.16 > Passed on x86_64 with linux-2.6.16.43-0.3-smp > Passed on x86_64 with linux-2.6.16.21-0.8-smp > Passed on x86_64 with linux-2.6.18 > Passed on x86_64 with linux-2.6.16.60-0.21-smp > Passed on x86_64 with linux-2.6.17 > Passed on x86_64 with linux-2.6.18-53.el5 > Passed on x86_64 with linux-2.6.18-8.el5 > Passed on x86_64 with linux-2.6.18-1.2798.fc6 > Passed on x86_64 with linux-2.6.19 > Passed on x86_64 with linux-2.6.20 > Passed on x86_64 with linux-2.6.18-93.el5 > Passed on x86_64 with linux-2.6.21.1 > Passed on x86_64 with linux-2.6.22 > Passed on x86_64 with linux-2.6.22.5-31-default > Passed on x86_64 with linux-2.6.24 > Passed on x86_64 with linux-2.6.9-42.ELsmp > Passed on x86_64 with linux-2.6.9-67.ELsmp > Passed on x86_64 with linux-2.6.9-55.ELsmp > Passed on ppc64 with linux-2.6.12 > Passed on ppc64 with linux-2.6.14 > Passed on ppc64 with linux-2.6.13 > Passed on ppc64 with linux-2.6.15 > Passed on ppc64 with linux-2.6.17 > Passed on ppc64 with linux-2.6.16 > Passed on ppc64 with linux-2.6.19 > Passed on ppc64 with linux-2.6.18 > Passed on ppc64 with linux-2.6.18-8.el5 > Passed on ppc64 with linux-2.6.24 > From arlin.r.davis at intel.com Fri Jun 20 11:41:23 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:41:23 -0700 Subject: [ofa-general] [PATCH 0/7][v1.2, v2.0] uDAPL patch set to enable scalability to 1000+ nodes/10000+ cores Message-ID: We managed to get access to a large cluster with 1000+ nodes and 10,000+ cores for testing/benchmarking. I am happy to say that uDAPL successfully scaled out to more then 14,000 cores. However, when running Intel MPI and uDAPL (OFED 1.2.5, mlx4 DDR) we discovered that the uDAPL rdma_cm provider would not scale beyond 256 nodes so we had to move back to a socket cm provider to setup the QP's. This patch set brings back socket cm (slight redesign) with some fixes and cleanup. For the record, the basic reason for rdma_cm scaling problems was path record queries. Until there is consensus on IB path record caching solutions that scales and is moved upstream I am recommending that uDAPL IB consumers needing large scale-out use socket cm provider (libdaplscm.so) in leiu of rdma_cm (libdaplcma.so). iWARP support will remain via uDAPL rdma_cm provider. -arlin From arlin.r.davis at intel.com Fri Jun 20 11:42:20 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:42:20 -0700 Subject: [ofa-general] [PATCH 1/7][v1.2] dapl: endpoint pending request count is wrong Message-ID: The code assumes every cookie allocated during posting of requests gets completed. This incorrect assumption results in wrong pending count. Remove request_pending field and replace with direct call, dapl_cb_pending, to provide accurate data to consumer. Add debug print if consumer overruns request queue. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/common/dapl_cookie.c | 28 ++++++++++++++++++++++++++++ dapl/common/dapl_cookie.h | 4 ++++ dapl/common/dapl_ep_get_status.c | 4 ++-- dapl/common/dapl_ep_post_recv.c | 7 ------- dapl/common/dapl_ep_util.c | 22 ++++++++++------------ dapl/common/dapl_evd_util.c | 16 ++++------------ dapl/common/dapl_rmr_bind.c | 14 -------------- 7 files changed, 48 insertions(+), 47 deletions(-) diff --git a/dapl/common/dapl_cookie.c b/dapl/common/dapl_cookie.c index ee12ce2..5b0b598 100644 --- a/dapl/common/dapl_cookie.c +++ b/dapl/common/dapl_cookie.c @@ -260,6 +260,34 @@ dapls_cb_put ( return DAT_SUCCESS; } +/* + * dapls_cb_pending + * + * snapshot of active entries on cookie ring buffer + * + * Input: + * buffer pointer to DAPL_COOKIE_BUFFER + * + * Returns: + * DAT_COUNT number of active/pending cookies + * + */ +extern DAT_COUNT +dapls_cb_pending ( + DAPL_COOKIE_BUFFER *buffer ) +{ + DAT_COUNT head, tail; + + head = dapl_os_atomic_read(&buffer->head); + tail = dapl_os_atomic_read(&buffer->tail); + + if (head == tail) + return 0; + else if (head > tail) + return (head - tail); + else + return ((buffer->pool_size - tail) + head); +} /* * dapls_rmr_cookie_alloc diff --git a/dapl/common/dapl_cookie.h b/dapl/common/dapl_cookie.h index 20fbec6..f953b28 100644 --- a/dapl/common/dapl_cookie.h +++ b/dapl/common/dapl_cookie.h @@ -50,6 +50,10 @@ extern void dapls_cb_free ( DAPL_COOKIE_BUFFER *buffer ); +extern DAT_COUNT +dapls_cb_pending ( + DAPL_COOKIE_BUFFER *buffer ); + extern DAT_RETURN dapls_rmr_cookie_alloc ( IN DAPL_COOKIE_BUFFER *buffer, diff --git a/dapl/common/dapl_ep_get_status.c b/dapl/common/dapl_ep_get_status.c index 49b4fef..a931355 100644 --- a/dapl/common/dapl_ep_get_status.c +++ b/dapl/common/dapl_ep_get_status.c @@ -98,12 +98,12 @@ dapl_ep_get_status ( if ( in_dto_idle != NULL ) { - *in_dto_idle = (dapl_os_atomic_read (&ep_ptr->recv_count)) ? DAT_FALSE : DAT_TRUE; + *in_dto_idle = (dapls_cb_pending(&ep_ptr->recv_buffer)) ? DAT_FALSE : DAT_TRUE; } if ( out_dto_idle != NULL ) { - *out_dto_idle = (dapl_os_atomic_read (&ep_ptr->req_count)) ? DAT_FALSE : DAT_TRUE; + *out_dto_idle = (dapls_cb_pending(&ep_ptr->req_buffer)) ? DAT_FALSE : DAT_TRUE; } bail: diff --git a/dapl/common/dapl_ep_post_recv.c b/dapl/common/dapl_ep_post_recv.c index c9be9ec..5f903b8 100644 --- a/dapl/common/dapl_ep_post_recv.c +++ b/dapl/common/dapl_ep_post_recv.c @@ -109,19 +109,12 @@ dapl_ep_post_recv ( } /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->recv_count); - - /* * Invoke provider specific routine to post DTO */ dat_status = dapls_ib_post_recv (ep_ptr, cookie, num_segments, local_iov); if ( dat_status != DAT_SUCCESS ) { - dapl_os_atomic_dec (&ep_ptr->recv_count); dapls_cookie_dealloc (&ep_ptr->recv_buffer, cookie); } diff --git a/dapl/common/dapl_ep_util.c b/dapl/common/dapl_ep_util.c index d11aed3..2bad346 100644 --- a/dapl/common/dapl_ep_util.c +++ b/dapl/common/dapl_ep_util.c @@ -131,9 +131,6 @@ dapl_ep_alloc ( ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; ep_ptr->cm_handle = IB_INVALID_HANDLE; - dapl_os_atomic_set (&ep_ptr->req_count, 0); - dapl_os_atomic_set (&ep_ptr->recv_count, 0); - if ( DAT_SUCCESS != dapls_cb_create ( &ep_ptr->req_buffer, ep_ptr, @@ -393,18 +390,20 @@ dapl_ep_post_send_req ( dto_type, user_cookie, &cookie ); - if ( dat_status != DAT_SUCCESS ) - { + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " dapl_post_req resource ERR:" + " dtos pending = %d, max_dtos %d, max_cb %d hd %d tl %d\n", + dapls_cb_pending(&ep_ptr->req_buffer), + ep_ptr->param.ep_attr.max_request_dtos, + ep_ptr->req_buffer.pool_size, + ep_ptr->req_buffer.head, + ep_ptr->req_buffer.tail); + goto bail; } /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->req_count); - - /* * Invoke provider specific routine to post DTO */ dat_status = dapls_ib_post_send ( ep_ptr, @@ -417,7 +416,6 @@ dapl_ep_post_send_req ( if ( dat_status != DAT_SUCCESS ) { - dapl_os_atomic_dec (&ep_ptr->req_count); dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie); } diff --git a/dapl/common/dapl_evd_util.c b/dapl/common/dapl_evd_util.c index 2c95c6d..41c128c 100644 --- a/dapl/common/dapl_evd_util.c +++ b/dapl/common/dapl_evd_util.c @@ -1029,17 +1029,11 @@ dapli_evd_cqe_to_event ( { DAPL_COOKIE_BUFFER *buffer; - if ( DAPL_DTO_TYPE_RECV == cookie->val.dto.type ) - { - dapl_os_atomic_dec (&ep_ptr->recv_count); - buffer = &ep_ptr->recv_buffer; - } + if (DAPL_DTO_TYPE_RECV == cookie->val.dto.type) + buffer = &ep_ptr->recv_buffer; else - { - dapl_os_atomic_dec (&ep_ptr->req_count); - buffer = &ep_ptr->req_buffer; - } - + buffer = &ep_ptr->req_buffer; + event_ptr->event_number = DAT_DTO_COMPLETION_EVENT; event_ptr->event_data.dto_completion_event_data.ep_handle = cookie->ep; @@ -1084,8 +1078,6 @@ dapli_evd_cqe_to_event ( case DAPL_COOKIE_TYPE_RMR: { - dapl_os_atomic_dec (&ep_ptr->req_count); - event_ptr->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_ptr->event_data.rmr_completion_event_data.rmr_handle = diff --git a/dapl/common/dapl_rmr_bind.c b/dapl/common/dapl_rmr_bind.c index 1582fb3..905ea2c 100644 --- a/dapl/common/dapl_rmr_bind.c +++ b/dapl/common/dapl_rmr_bind.c @@ -151,12 +151,6 @@ dapli_rmr_bind_fuse ( is_signaled = (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? DAT_FALSE : DAT_TRUE; - /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->req_count); - dat_status = dapls_ib_mw_bind (rmr, lmr, ep_ptr, @@ -167,7 +161,6 @@ dapli_rmr_bind_fuse ( is_signaled); if ( DAT_SUCCESS != dat_status ) { - dapl_os_atomic_dec (&ep_ptr->req_count); dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie); goto bail; } @@ -255,19 +248,12 @@ dapli_rmr_bind_unfuse ( is_signaled = (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? DAT_FALSE : DAT_TRUE; - /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->req_count); - dat_status = dapls_ib_mw_unbind (rmr, ep_ptr, cookie, is_signaled); if ( DAT_SUCCESS != dat_status ) { - dapl_os_atomic_dec (&ep_ptr->req_count); dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie); goto bail1; } -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:43:26 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:43:26 -0700 Subject: [ofa-general] [PATCH 2/7][v1.2] dapl: remove unused iov buffer allocation on the endpoint Message-ID: provider's manage iov space on stack during posting. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/common/dapl_ep_modify.c | 82 ------------------------------------------ dapl/common/dapl_ep_util.c | 46 ----------------------- dapl/include/dapl.h | 5 --- 3 files changed, 0 insertions(+), 133 deletions(-) diff --git a/dapl/common/dapl_ep_modify.c b/dapl/common/dapl_ep_modify.c index f2628af..74e3331 100644 --- a/dapl/common/dapl_ep_modify.c +++ b/dapl/common/dapl_ep_modify.c @@ -95,15 +95,11 @@ dapl_ep_modify ( DAT_BOOLEAN qp_allocated = DAT_FALSE; DAT_BOOLEAN rqst_cb_allocated = DAT_FALSE; DAT_BOOLEAN recv_cb_allocated = DAT_FALSE; - DAT_BOOLEAN rqst_iov_allocated = DAT_FALSE; - DAT_BOOLEAN recv_iov_allocated = DAT_FALSE; /* Flag indicating we've used (assigned to QP) a new one of these. */ DAT_BOOLEAN qp_used = DAT_FALSE; DAT_BOOLEAN rqst_cb_used = DAT_FALSE; DAT_BOOLEAN recv_cb_used = DAT_FALSE; - DAT_BOOLEAN rqst_iov_used = DAT_FALSE; - DAT_BOOLEAN recv_iov_used = DAT_FALSE; dat_status = dapli_ep_modify_validate_parameters ( ep_handle, ep_param_mask, @@ -171,39 +167,6 @@ dapl_ep_modify ( } recv_cb_allocated = DAT_TRUE; - alloc_ep.send_iov_num = ep_attr1.max_request_iov; - - /* - * Ensure that send_iov_num is big enough for all types - * of send requests. - */ - if (ep_attr1.max_rdma_read_iov > alloc_ep.send_iov_num) - { - alloc_ep.send_iov_num = ep_attr1.max_rdma_read_iov; - } - if (ep_attr1.max_rdma_write_iov > alloc_ep.send_iov_num) - { - alloc_ep.send_iov_num = ep_attr1.max_rdma_write_iov; - } - alloc_ep.send_iov = dapl_os_alloc (alloc_ep.send_iov_num - * sizeof (ib_data_segment_t)); - if ( alloc_ep.send_iov == NULL ) - { - dat_status = DAT_ERROR (DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - rqst_iov_allocated = DAT_TRUE; - - alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; - alloc_ep.recv_iov = dapl_os_alloc (ep_attr1.max_recv_iov - * sizeof (ib_data_segment_t)); - if ( alloc_ep.recv_iov == NULL ) - { - dat_status = DAT_ERROR (DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - recv_iov_allocated = DAT_TRUE; - dat_status = dapls_ib_qp_alloc ( ia, &alloc_ep, ep1 ); if ( dat_status != DAT_SUCCESS ) { @@ -281,20 +244,6 @@ dapl_ep_modify ( rqst_cb_used = DAT_TRUE; } - if ( ep_param_mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV - && new_ep.recv_iov_num != ep2->recv_iov_num ) - { - new_ep.recv_iov = alloc_ep.recv_iov; - recv_iov_used = DAT_TRUE; - } - - if ( ep_param_mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV - && new_ep.send_iov_num != ep2->send_iov_num ) - { - new_ep.send_iov = alloc_ep.send_iov; - recv_iov_used = DAT_TRUE; - } - /* * We need to change the QP only if there already was a QP * (leave things the way you found them!) and one of the @@ -439,37 +388,6 @@ bail: } } - if ( rqst_iov_allocated ) - { - if ( dat_status != DAT_SUCCESS || !rqst_iov_used ) - { - dapl_os_free ( alloc_ep.send_iov, - (alloc_ep.send_iov_num - * sizeof (ib_data_segment_t))); - } - else - { - dapl_os_free ( copy_of_old_ep.send_iov, - (copy_of_old_ep.send_iov_num - * sizeof (ib_data_segment_t))); - } - } - - if ( recv_iov_allocated ) - { - if ( dat_status != DAT_SUCCESS || !recv_iov_used ) - { - dapl_os_free ( alloc_ep.recv_iov, - (alloc_ep.recv_iov_num - * sizeof (ib_data_segment_t))); - } - else - { - dapl_os_free ( copy_of_old_ep.recv_iov, - (copy_of_old_ep.recv_iov_num - * sizeof (ib_data_segment_t))); - } - } return dat_status; } diff --git a/dapl/common/dapl_ep_util.c b/dapl/common/dapl_ep_util.c index 2bad346..4dd722e 100644 --- a/dapl/common/dapl_ep_util.c +++ b/dapl/common/dapl_ep_util.c @@ -151,42 +151,6 @@ dapl_ep_alloc ( goto bail; } - ep_ptr->recv_iov_num = ep_ptr->param.ep_attr.max_recv_iov; - ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_request_iov; - - /* - * Ensure that send_iov_num is big enough for all types - * of send requests. - */ - if (ep_ptr->param.ep_attr.max_rdma_read_iov > ep_ptr->send_iov_num) - { - ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_rdma_read_iov; - } - if (ep_ptr->param.ep_attr.max_rdma_write_iov > ep_ptr->send_iov_num) - { - ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_rdma_write_iov; - } - - ep_ptr->recv_iov = dapl_os_alloc ( - ep_ptr->recv_iov_num * sizeof (ib_data_segment_t)); - - if ( NULL == ep_ptr->recv_iov ) - { - dapl_ep_dealloc (ep_ptr); - ep_ptr = NULL; - goto bail; - } - - ep_ptr->send_iov = dapl_os_alloc ( - ep_ptr->send_iov_num * sizeof (ib_data_segment_t)); - - if ( NULL == ep_ptr->send_iov ) - { - dapl_ep_dealloc (ep_ptr); - ep_ptr = NULL; - goto bail; - } - dapls_io_trc_alloc (ep_ptr); bail: @@ -220,16 +184,6 @@ dapl_ep_dealloc ( dapls_cb_free ( &ep_ptr->req_buffer ); dapls_cb_free ( &ep_ptr->recv_buffer ); - if ( NULL != ep_ptr->recv_iov ) - { - dapl_os_free (ep_ptr->recv_iov, ep_ptr->recv_iov_num * sizeof (ib_data_segment_t)); - } - - if ( NULL != ep_ptr->send_iov ) - { - dapl_os_free (ep_ptr->send_iov, ep_ptr->send_iov_num * sizeof (ib_data_segment_t)); - } - if ( NULL != ep_ptr->cxn_timer ) { dapl_os_free ( ep_ptr->cxn_timer, sizeof ( DAPL_OS_TIMER ) ); diff --git a/dapl/include/dapl.h b/dapl/include/dapl.h index 45e1e8a..80c9ff3 100644 --- a/dapl/include/dapl.h +++ b/dapl/include/dapl.h @@ -448,11 +448,6 @@ struct dapl_ep DAPL_COOKIE_BUFFER req_buffer; DAPL_COOKIE_BUFFER recv_buffer; - ib_data_segment_t *recv_iov; - DAT_COUNT recv_iov_num; - - ib_data_segment_t *send_iov; - DAT_COUNT send_iov_num; #ifdef DAPL_DBG_IO_TRC int ibt_dumped; struct io_buf_track *ibt_base; -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:48:18 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:48:18 -0700 Subject: [ofa-general] [PATCH 3/7][v1.2] dapl: add opcode to string function to report opcode during failures. Message-ID: Need to use cookie opcode during failures in lieu of cqe opcode. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/common/dapl_evd_util.c | 24 +++++++++++++++++++++--- 1 files changed, 21 insertions(+), 3 deletions(-) diff --git a/dapl/common/dapl_evd_util.c b/dapl/common/dapl_evd_util.c index 41c128c..f3c05b2 100644 --- a/dapl/common/dapl_evd_util.c +++ b/dapl/common/dapl_evd_util.c @@ -42,6 +42,10 @@ #include "dapl_cookie.h" #include "dapl.h" +#include +#include +#include + STATIC _INLINE_ void dapli_evd_eh_print_cqe ( IN ib_work_completion_t *cqe); @@ -485,6 +489,19 @@ bail: return dat_status; } +STATIC _INLINE_ char * DAPL_GET_DTO_OP_STR(int op) +{ + static char *dto_ops[] = + { + "OP_SEND", + "OP_RECEIVE", + "OP_RDMA_WRITE", + "OP_RDMA_READ", + 0 + }; + return ((op < 0 || op > 3) ? "Invalid DTO OP?" : dto_ops[op]); +} + #if !defined(DAPL_GET_CQE_OP_STR) #define DAPL_GET_CQE_OP_STR(e) "Unknown CEQ OP String?" #endif @@ -1158,10 +1175,11 @@ dapli_evd_cqe_to_event ( } dapl_log(DAPL_DBG_TYPE_ERR, - "DTO completion ERR: status %d, opcode %s, vendor_err 0x%x\n", + "DTO completion ERR: status %d, op %s, vendor_err 0x%x - %s\n", DAPL_GET_CQE_STATUS(cqe_ptr), - DAPL_GET_CQE_OP_STR(cqe_ptr), - DAPL_GET_CQE_VENDOR_ERR(cqe_ptr)); + DAPL_GET_DTO_OP_STR(cookie->val.dto.type), + DAPL_GET_CQE_VENDOR_ERR(cqe_ptr), + inet_ntoa(((struct sockaddr_in *)&ep_ptr->remote_ia_address)->sin_addr)); } } -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:50:11 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:50:11 -0700 Subject: [ofa-general] [PATCH 4/7][v1.2] dapl scm: Adding socket cm provider for better scalability on large homogeneous clusters. Message-ID: Bring socket cm provider back to life with some changes: better threading support for exchanging QP information. Avoid blocking during connect to support dynamic connection model with MPI implementations. consumer control of ack timeout/retries. disconnect/reject capabilities via socket exchange. version support for wire protocol to insure compatibility with peer scm provider. Add gids to exchange. validated with Intel MPI on a 14,000+ core fabric using IB DDR. --- Makefile.am | 134 ++++++- dapl.spec.in | 10 +- dapl/openib_scm/dapl_ib_cm.c | 833 +++++++++++++++++++++++++--------------- dapl/openib_scm/dapl_ib_cq.c | 19 +- dapl/openib_scm/dapl_ib_dto.h | 212 ++++++----- dapl/openib_scm/dapl_ib_mem.c | 137 ++++---- dapl/openib_scm/dapl_ib_qp.c | 53 ++-- dapl/openib_scm/dapl_ib_util.c | 152 ++++---- dapl/openib_scm/dapl_ib_util.h | 44 ++- 9 files changed, 1016 insertions(+), 578 deletions(-) diff --git a/Makefile.am b/Makefile.am index 079ad7f..bc7926d 100644 --- a/Makefile.am +++ b/Makefile.am @@ -19,9 +19,11 @@ endif datlibdir = $(libdir) dapllibcmadir = $(libdir) +dapllibscmdir = $(libdir) datlib_LTLIBRARIES = dat/udat/libdat.la dapllibcma_LTLIBRARIES = dapl/udapl/libdaplcma.la +dapllibscm_LTLIBRARIES = dapl/udapl/libdaplscm.la dat_udat_libdat_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) \ -I$(srcdir)/dat/include/ -I$(srcdir)/dat/udat/ \ @@ -32,15 +34,22 @@ dapl_udapl_libdaplcma_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) \ -I$(srcdir)/dat/include/ -I$(srcdir)/dapl/include/ \ -I$(srcdir)/dapl/common -I$(srcdir)/dapl/udapl/linux \ -I$(srcdir)/dapl/openib_cma + +dapl_udapl_libdaplscm_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) $(XFLAGS) \ + -DOPENIB -DCQ_WAIT_OBJECT \ + -I$(srcdir)/dat/include/ -I$(srcdir)/dapl/include/ \ + -I$(srcdir)/dapl/common -I$(srcdir)/dapl/udapl/linux \ + -I$(srcdir)/dapl/openib_scm + if HAVE_LD_VERSION_SCRIPT dat_version_script = -Wl,--version-script=$(srcdir)/dat/udat/libdat.map daplcma_version_script = -Wl,--version-script=$(srcdir)/dapl/udapl/libdaplcma.map - + daplscm_version_script = -Wl,--version-script=$(srcdir)/dapl/udapl/libdaplscm.map else - dat_version_script = - daplcma_version_script = - + dat_version_script = + daplofa_version_script = + daplscm_version_script = endif # @@ -168,6 +177,115 @@ dapl_udapl_libdaplcma_la_LDFLAGS = -version-info 1:2:0 $(daplcma_version_script) -Wl,-init,dapl_init -Wl,-fini,dapl_fini \ -lpthread -libverbs -lrdmacm +# +# uDAPL OpenFabrics Socket CM version: libdaplscm.so +# +dapl_udapl_libdaplscm_la_SOURCES = dapl/udapl/dapl_init.c \ + dapl/udapl/dapl_evd_create.c \ + dapl/udapl/dapl_evd_query.c \ + dapl/udapl/dapl_cno_create.c \ + dapl/udapl/dapl_cno_modify_agent.c \ + dapl/udapl/dapl_cno_free.c \ + dapl/udapl/dapl_cno_wait.c \ + dapl/udapl/dapl_cno_query.c \ + dapl/udapl/dapl_lmr_create.c \ + dapl/udapl/dapl_evd_wait.c \ + dapl/udapl/dapl_evd_disable.c \ + dapl/udapl/dapl_evd_enable.c \ + dapl/udapl/dapl_evd_modify_cno.c \ + dapl/udapl/dapl_evd_set_unwaitable.c \ + dapl/udapl/dapl_evd_clear_unwaitable.c \ + dapl/udapl/linux/dapl_osd.c \ + dapl/common/dapl_cookie.c \ + dapl/common/dapl_cr_accept.c \ + dapl/common/dapl_cr_query.c \ + dapl/common/dapl_cr_reject.c \ + dapl/common/dapl_cr_util.c \ + dapl/common/dapl_cr_callback.c \ + dapl/common/dapl_cr_handoff.c \ + dapl/common/dapl_ep_connect.c \ + dapl/common/dapl_ep_create.c \ + dapl/common/dapl_ep_disconnect.c \ + dapl/common/dapl_ep_dup_connect.c \ + dapl/common/dapl_ep_free.c \ + dapl/common/dapl_ep_reset.c \ + dapl/common/dapl_ep_get_status.c \ + dapl/common/dapl_ep_modify.c \ + dapl/common/dapl_ep_post_rdma_read.c \ + dapl/common/dapl_ep_post_rdma_write.c \ + dapl/common/dapl_ep_post_recv.c \ + dapl/common/dapl_ep_post_send.c \ + dapl/common/dapl_ep_query.c \ + dapl/common/dapl_ep_util.c \ + dapl/common/dapl_evd_dequeue.c \ + dapl/common/dapl_evd_free.c \ + dapl/common/dapl_evd_post_se.c \ + dapl/common/dapl_evd_resize.c \ + dapl/common/dapl_evd_util.c \ + dapl/common/dapl_evd_cq_async_error_callb.c \ + dapl/common/dapl_evd_qp_async_error_callb.c \ + dapl/common/dapl_evd_un_async_error_callb.c \ + dapl/common/dapl_evd_connection_callb.c \ + dapl/common/dapl_evd_dto_callb.c \ + dapl/common/dapl_get_consumer_context.c \ + dapl/common/dapl_get_handle_type.c \ + dapl/common/dapl_hash.c \ + dapl/common/dapl_hca_util.c \ + dapl/common/dapl_ia_close.c \ + dapl/common/dapl_ia_open.c \ + dapl/common/dapl_ia_query.c \ + dapl/common/dapl_ia_util.c \ + dapl/common/dapl_llist.c \ + dapl/common/dapl_lmr_free.c \ + dapl/common/dapl_lmr_query.c \ + dapl/common/dapl_lmr_util.c \ + dapl/common/dapl_lmr_sync_rdma_read.c \ + dapl/common/dapl_lmr_sync_rdma_write.c \ + dapl/common/dapl_mr_util.c \ + dapl/common/dapl_provider.c \ + dapl/common/dapl_sp_util.c \ + dapl/common/dapl_psp_create.c \ + dapl/common/dapl_psp_create_any.c \ + dapl/common/dapl_psp_free.c \ + dapl/common/dapl_psp_query.c \ + dapl/common/dapl_pz_create.c \ + dapl/common/dapl_pz_free.c \ + dapl/common/dapl_pz_query.c \ + dapl/common/dapl_pz_util.c \ + dapl/common/dapl_rmr_create.c \ + dapl/common/dapl_rmr_free.c \ + dapl/common/dapl_rmr_bind.c \ + dapl/common/dapl_rmr_query.c \ + dapl/common/dapl_rmr_util.c \ + dapl/common/dapl_rsp_create.c \ + dapl/common/dapl_rsp_free.c \ + dapl/common/dapl_rsp_query.c \ + dapl/common/dapl_cno_util.c \ + dapl/common/dapl_set_consumer_context.c \ + dapl/common/dapl_ring_buffer_util.c \ + dapl/common/dapl_name_service.c \ + dapl/common/dapl_timer_util.c \ + dapl/common/dapl_ep_create_with_srq.c \ + dapl/common/dapl_ep_recv_query.c \ + dapl/common/dapl_ep_set_watermark.c \ + dapl/common/dapl_srq_create.c \ + dapl/common/dapl_srq_free.c \ + dapl/common/dapl_srq_query.c \ + dapl/common/dapl_srq_resize.c \ + dapl/common/dapl_srq_post_recv.c \ + dapl/common/dapl_srq_set_lw.c \ + dapl/common/dapl_srq_util.c \ + dapl/common/dapl_debug.c \ + dapl/openib_scm/dapl_ib_util.c \ + dapl/openib_scm/dapl_ib_cq.c \ + dapl/openib_scm/dapl_ib_qp.c \ + dapl/openib_scm/dapl_ib_cm.c \ + dapl/openib_scm/dapl_ib_mem.c + +dapl_udapl_libdaplscm_la_LDFLAGS = -version-info 1:2:0 $(daplscm_version_script) \ + -Wl,-init,dapl_init -Wl,-fini,dapl_fini \ + -lpthread -libverbs + libdatincludedir = $(includedir)/dat libdatinclude_HEADERS = dat/include/dat/dat.h \ @@ -230,6 +348,7 @@ EXTRA_DIST = dat/common/dat_dictionary.h \ dapl/openib_scm/dapl_ib_util.h \ dat/udat/libdat.map \ dapl/udapl/libdaplcma.map \ + dapl/udapl/libdaplscm.map \ dapl.spec.in \ $(man_MANS) \ test/dapltest/include/dapl_bpool.h \ @@ -271,9 +390,10 @@ install-exec-hook: fi; \ echo OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib0 0" ""' >> $(sysconfdir)/dat.conf; \ echo OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib1 0" ""' >> $(sysconfdir)/dat.conf; \ - echo OpenIB-cma-2 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib2 0" ""' >> $(sysconfdir)/dat.conf; \ - echo OpenIB-cma-3 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib3 0" ""' >> $(sysconfdir)/dat.conf; \ - echo OpenIB-bond u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"bond0 0" ""' >> $(sysconfdir)/dat.conf; + echo OpenIB-mthca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mthca0 1" ""' >> $(sysconfdir)/dat.conf; \ + echo OpenIB-mthca0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mthca0 2" ""' >> $(sysconfdir)/dat.conf; \ + echo OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mlx4_0 1" ""' >> $(sysconfdir)/dat.conf; \ + echo OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mlx4_0 2" ""' >> $(sysconfdir)/dat.conf; uninstall-hook: if test -e $(sysconfdir)/dat.conf; then \ diff --git a/dapl.spec.in b/dapl.spec.in index 2153e27..3e64bdc 100644 --- a/dapl.spec.in +++ b/dapl.spec.in @@ -95,10 +95,10 @@ if [ -e %{_sysconfdir}/dat.conf ]; then fi echo OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib0 0" ""' >> %{_sysconfdir}/dat.conf echo OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib1 0" ""' >> %{_sysconfdir}/dat.conf -echo OpenIB-cma-2 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib2 0" ""' >> %{_sysconfdir}/dat.conf -echo OpenIB-cma-3 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"ib3 0" ""' >> %{_sysconfdir}/dat.conf -echo OpenIB-bond u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 '"bond0 0" ""' >> %{_sysconfdir}/dat.conf - +echo OpenIB-mthca0-1 u2.0 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mthca0 1" ""' >> %{_sysconfdir}/dat.conf +echo OpenIB-mthca0-2 u2.0 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mthca0 2" ""' >> %{_sysconfdir}/dat.conf +echo OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mlx4_0 1" ""' >> %{_sysconfdir}/dat.conf +echo OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 '"mlx4_0 2" ""' >> %{_sysconfdir}/dat.conf %postun /sbin/ldconfig @@ -130,7 +130,7 @@ fi %changelog * Tue May 20 2008 Arlin Davis - 1.2.7 -- DAT/DAPL Version 1.2.7 Release 1, OFED 1.3.1 GA +- DAT/DAPL Version 1.2.7 Release 1, OFED 1.3.1 GA * Thu May 1 2008 Arlin Davis - 1.2.6 - DAT/DAPL Version 1.2.6 Release 1, OFED 1.3.1 diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index f534e8d..9e686d6 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -57,76 +57,171 @@ #include #include #include -#include +#include +#include -/* prototypes */ -static uint16_t dapli_get_lid( struct ibv_device *dev, int port ); +#include +#include +#include -static DAT_RETURN dapli_socket_connect ( DAPL_EP *ep_ptr, - DAT_IA_ADDRESS_PTR r_addr, - DAT_CONN_QUAL r_qual, - DAT_COUNT p_size, - DAT_PVOID p_data ); +#if __BYTE_ORDER == __LITTLE_ENDIAN +static inline uint64_t cpu_to_be64(uint64_t x) {return bswap_64(x);} +#elif __BYTE_ORDER == __BIG_ENDIAN +static inline uint64_t cpu_to_be64(uint64_t x) {return x;} +#endif -static DAT_RETURN dapli_socket_listen ( DAPL_IA *ia_ptr, - DAT_CONN_QUAL serviceID, - DAPL_SP *sp_ptr ); +extern int g_scm_pipe[2]; -static DAT_RETURN dapli_socket_accept( ib_cm_srvc_handle_t cm_ptr ); +static struct ib_cm_handle *dapli_cm_create(void) +{ + struct ib_cm_handle *cm_ptr; -static DAT_RETURN dapli_socket_accept_final( DAPL_EP *ep_ptr, - DAPL_CR *cr_ptr, - DAT_COUNT p_size, - DAT_PVOID p_data ); + /* Allocate CM, init lock, and initialize */ + if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) + return NULL; + + if (dapl_os_lock_init(&cm_ptr->lock)) + goto bail; -/* XXX temporary hack to get lid */ -static uint16_t dapli_get_lid(IN struct ibv_device *dev, IN int port) + (void)dapl_os_memzero(cm_ptr, sizeof(*cm_ptr)); + cm_ptr->dst.ver = htons(DSCM_VER); + cm_ptr->socket = -1; + return cm_ptr; +bail: + dapl_os_free(cm_ptr, sizeof(*cm_ptr)); + return NULL; +} + +/* mark for destroy, remove all references, schedule cleanup */ +static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) { - char path[128]; - char val[16]; - char name[256]; + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " cm_destroy: cm %p ep %p\n", cm_ptr,cm_ptr->ep); + + /* cleanup, never made it to work queue */ + if (cm_ptr->state == SCM_INIT) { + if (cm_ptr->socket >= 0) + close(cm_ptr->socket); + dapl_os_free(cm_ptr, sizeof(*cm_ptr)); + return; + } - if (sysfs_get_mnt_path(path, sizeof path)) { - fprintf(stderr, "Couldn't find sysfs mount.\n"); - return 0; + dapl_os_lock(&cm_ptr->lock); + cm_ptr->state = SCM_DESTROY; + if (cm_ptr->ep) { + cm_ptr->ep->cm_handle = IB_INVALID_HANDLE; + cm_ptr->ep->qp_handle = IB_INVALID_HANDLE; } - sprintf(name, "%s/class/infiniband/%s/ports/%d/lid", path, - ibv_get_device_name(dev), port); - if (sysfs_read_attribute_value(name, val, sizeof val)) { - fprintf(stderr, "Couldn't read LID at %s\n", name); - return 0; + /* close socket if still active */ + if (cm_ptr->socket >= 0) { + close(cm_ptr->socket); + cm_ptr->socket = -1; } - return strtol(val, NULL, 0); + dapl_os_unlock(&cm_ptr->lock); + + /* wakeup work thread */ + write(g_scm_pipe[1], "w", sizeof "w"); +} + +/* queue socket for processing CM work */ +static void dapli_cm_queue(struct ib_cm_handle *cm_ptr) +{ + /* add to work queue for cr thread processing */ + dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry); + dapl_os_lock(&cm_ptr->hca->ib_trans.lock); + dapl_llist_add_tail(&cm_ptr->hca->ib_trans.list, + (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr); + dapl_os_unlock(&cm_ptr->hca->ib_trans.lock); + + /* wakeup CM work thread */ + write(g_scm_pipe[1], "w", sizeof "w"); +} + +static uint16_t dapli_get_lid(IN struct ibv_context *ctx, IN uint8_t port) +{ + struct ibv_port_attr port_attr; + + if(ibv_query_port(ctx, port,&port_attr)) + return(0xffff); + else + return(port_attr.lid); } /* - * ACTIVE: Create socket, connect, and exchange QP information + * ACTIVE/PASSIVE: called from CR thread or consumer via ep_disconnect */ static DAT_RETURN -dapli_socket_connect ( DAPL_EP *ep_ptr, - DAT_IA_ADDRESS_PTR r_addr, - DAT_CONN_QUAL r_qual, - DAT_COUNT p_size, - DAT_PVOID p_data ) +dapli_socket_disconnect(ib_cm_handle_t cm_ptr) { - ib_cm_handle_t cm_ptr; - DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; + DAPL_EP *ep_ptr = cm_ptr->ep; + DAT_UINT32 disc_data = htonl(0xdead); + + if (ep_ptr == NULL) + return DAT_SUCCESS; + + dapl_os_lock(&cm_ptr->lock); + if ((cm_ptr->state == SCM_INIT) || + (cm_ptr->state == SCM_DISCONNECTED)) { + dapl_os_unlock(&cm_ptr->lock); + return DAT_SUCCESS; + } else { + /* send disc date, close socket, schedule destroy */ + if (cm_ptr->socket >= 0) { + write(cm_ptr->socket, &disc_data, sizeof(disc_data)); + close(cm_ptr->socket); + cm_ptr->socket = -1; + } + cm_ptr->state = SCM_DISCONNECTED; + write(g_scm_pipe[1], "w", sizeof "w"); + } + dapl_os_unlock(&cm_ptr->lock); + + + if (ep_ptr->cr_ptr) { + dapls_cr_callback(cm_ptr, + IB_CME_DISCONNECTED, + NULL, + ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr); + } else { + dapl_evd_connection_callback(ep_ptr->cm_handle, + IB_CME_DISCONNECTED, + NULL, + ep_ptr); + } + + /* remove reference from endpoint */ + ep_ptr->cm_handle = NULL; + + /* schedule destroy */ + + + return DAT_SUCCESS; +} + + +/* + * ACTIVE: Create socket, connect, defer exchange QP information to CR thread + * to avoid blocking. + */ +DAT_RETURN +dapli_socket_connect(DAPL_EP *ep_ptr, + DAT_IA_ADDRESS_PTR r_addr, + DAT_CONN_QUAL r_qual, + DAT_COUNT p_size, + DAT_PVOID p_data) +{ + ib_cm_handle_t cm_ptr; int len, opt = 1; struct iovec iovec[2]; - short rtu_data = htons(0x0E0F); - - dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d\n", r_qual); + DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; + + dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d p_size=%d\n", + r_qual,p_size); - /* - * Allocate CM and initialize - */ - if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL ) { + cm_ptr = dapli_cm_create(); + if (cm_ptr == NULL) return DAT_INSUFFICIENT_RESOURCES; - } - - (void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) ); - cm_ptr->socket = -1; /* create, connect, sockopt, and exchange QP information */ if ((cm_ptr->socket = socket(AF_INET,SOCK_STREAM,0)) < 0 ) { @@ -136,197 +231,252 @@ dapli_socket_connect ( DAPL_EP *ep_ptr, ((struct sockaddr_in*)r_addr)->sin_port = htons(r_qual); - if ( connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0 ) { + if (connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " connect: %s on r_qual %d\n", strerror(errno), (unsigned int)r_qual); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); + dapli_cm_destroy(cm_ptr); return DAT_INVALID_ADDRESS; } setsockopt(cm_ptr->socket,IPPROTO_TCP,TCP_NODELAY,&opt,sizeof(opt)); - + + dapl_dbg_log(DAPL_DBG_TYPE_EP, " socket connected!\n"); + + /* Send QP info, IA address, and private data */ - cm_ptr->dst.qpn = ep_ptr->qp_handle->qp_num; - cm_ptr->dst.port = ia_ptr->hca_ptr->port_num; - cm_ptr->dst.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, - ia_ptr->hca_ptr->port_num ); + cm_ptr->dst.qpn = htonl(ep_ptr->qp_handle->qp_num); + cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); + cm_ptr->dst.lid = + htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num)); + if (cm_ptr->dst.lid == 0xffff) + goto bail; + + /* in network order */ + if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num, + 0, + &cm_ptr->dst.gid)) + goto bail; + cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; - cm_ptr->dst.p_size = p_size; + cm_ptr->dst.p_size = htonl(p_size); iovec[0].iov_base = &cm_ptr->dst; iovec[0].iov_len = sizeof(ib_qp_cm_t); - if ( p_size ) { + if (p_size) { iovec[1].iov_base = p_data; iovec[1].iov_len = p_size; } - len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) ); - if ( len != (p_size + sizeof(ib_qp_cm_t)) ) { + + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket connected, write QP and private data\n"); + len = writev(cm_ptr->socket, iovec, (p_size ? 2:1)); + if (len != (p_size + sizeof(ib_qp_cm_t))) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " connect write: ERR %s, wcnt=%d\n", strerror(errno), len); goto bail; } - dapl_dbg_log(DAPL_DBG_TYPE_EP, + dapl_dbg_log(DAPL_DBG_TYPE_CM, " connect: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", - cm_ptr->dst.port, cm_ptr->dst.lid, - cm_ptr->dst.qpn, cm_ptr->dst.p_size ); + ntohs(cm_ptr->dst.port), ntohs(cm_ptr->dst.lid), + ntohl(cm_ptr->dst.qpn), ntohl(cm_ptr->dst.p_size)); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " connect SRC GID subnet %016llx id %016llx\n", + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.subnet_prefix), + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.interface_id)); + + /* queue up to work thread to avoid blocking consumer */ + cm_ptr->state = SCM_CONN_PENDING; + cm_ptr->hca = ia_ptr->hca_ptr; + cm_ptr->ep = ep_ptr; + dapli_cm_queue(cm_ptr); + return DAT_SUCCESS; +bail: + /* close socket, free cm structure */ + dapli_cm_destroy(cm_ptr); + return DAT_INTERNAL_ERROR; +} + + +/* + * ACTIVE: exchange QP information, called from CR thread + */ +void +dapli_socket_connect_rtu(ib_cm_handle_t cm_ptr) +{ + DAPL_EP *ep_ptr = cm_ptr->ep; + int len; + struct iovec iovec[2]; + short rtu_data = htons(0x0E0F); + ib_cm_events_t event = IB_CME_DESTINATION_REJECT; /* read DST information into cm_ptr, overwrite SRC info */ - len = readv( cm_ptr->socket, iovec, 1 ); - if ( len != sizeof(ib_qp_cm_t) ) { + dapl_dbg_log(DAPL_DBG_TYPE_EP," connect_rtu: recv peer QP data\n"); + + iovec[0].iov_base = &cm_ptr->dst; + iovec[0].iov_len = sizeof(ib_qp_cm_t); + len = readv(cm_ptr->socket, iovec, 1); + if (len != sizeof(ib_qp_cm_t) || ntohs(cm_ptr->dst.ver) != DSCM_VER) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " connect read: ERR %s, rcnt=%d\n", - strerror(errno), len); + " connect_rtu read: ERR %s, rcnt=%d, ver=%d\n", + strerror(errno), len, cm_ptr->dst.ver); + goto bail; + } + /* check for consumer reject */ + if (cm_ptr->dst.rej) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " connect_rtu read: PEER REJ reason=0x%x\n", + ntohs(cm_ptr->dst.rej)); + event = IB_CME_DESTINATION_REJECT_PRIVATE_DATA; goto bail; } + + /* convert peer response values to host order */ + cm_ptr->dst.port = ntohs(cm_ptr->dst.port); + cm_ptr->dst.lid = ntohs(cm_ptr->dst.lid); + cm_ptr->dst.qpn = ntohl(cm_ptr->dst.qpn); + cm_ptr->dst.p_size = ntohl(cm_ptr->dst.p_size); + + /* save remote address information */ + dapl_os_memcpy( &ep_ptr->remote_ia_address, + &cm_ptr->dst.ia_address, + sizeof(ep_ptr->remote_ia_address)); + dapl_dbg_log(DAPL_DBG_TYPE_EP, - " connect: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", + " connect_rtu: DST %s port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", + inet_ntoa(((struct sockaddr_in *)&cm_ptr->dst.ia_address)->sin_addr), cm_ptr->dst.port, cm_ptr->dst.lid, - cm_ptr->dst.qpn, cm_ptr->dst.p_size ); + cm_ptr->dst.qpn, cm_ptr->dst.p_size); /* validate private data size before reading */ - if ( cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE ) { + if (cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " connect read: psize (%d) wrong\n", + " connect_rtu read: psize (%d) wrong\n", cm_ptr->dst.p_size ); goto bail; } /* read private data into cm_handle if any present */ - if ( cm_ptr->dst.p_size ) { + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket connected, read private data\n"); + if (cm_ptr->dst.p_size) { iovec[0].iov_base = cm_ptr->p_data; iovec[0].iov_len = cm_ptr->dst.p_size; - len = readv( cm_ptr->socket, iovec, 1 ); - if ( len != cm_ptr->dst.p_size ) { + len = readv(cm_ptr->socket, iovec, 1); + if (len != cm_ptr->dst.p_size) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " connect read pdata: ERR %s, rcnt=%d\n", + " connect_rtu read pdata: ERR %s, rcnt=%d\n", strerror(errno), len); goto bail; } } /* modify QP to RTR and then to RTS with remote info */ - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTR, &cm_ptr->dst) != DAT_SUCCESS) goto bail; - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTS, &cm_ptr->dst) != DAT_SUCCESS) goto bail; ep_ptr->qp_state = IB_QP_STATE_RTS; + dapl_dbg_log(DAPL_DBG_TYPE_EP," connect_rtu: send RTU\n"); + /* complete handshake after final QP state change */ - write(cm_ptr->socket, &rtu_data, sizeof(rtu_data) ); + write(cm_ptr->socket, &rtu_data, sizeof(rtu_data)); /* init cm_handle and post the event with private data */ ep_ptr->cm_handle = cm_ptr; - dapl_dbg_log( DAPL_DBG_TYPE_EP," ACTIVE: connected!\n" ); - dapl_evd_connection_callback( ep_ptr->cm_handle, - IB_CME_CONNECTED, - cm_ptr->p_data, - ep_ptr ); - return DAT_SUCCESS; - + cm_ptr->state = SCM_CONNECTED; + dapl_dbg_log(DAPL_DBG_TYPE_EP," ACTIVE: connected!\n"); + dapl_evd_connection_callback(cm_ptr, + IB_CME_CONNECTED, + cm_ptr->p_data, + ep_ptr); + return; bail: /* close socket, free cm structure and post error event */ - if ( cm_ptr->socket >= 0 ) - close(cm_ptr->socket); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); - dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */ - - dapl_evd_connection_callback( ep_ptr->cm_handle, - IB_CME_LOCAL_FAILURE, - NULL, - ep_ptr ); - return DAT_INTERNAL_ERROR; + dapli_cm_destroy(cm_ptr); + dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ + dapl_evd_connection_callback(NULL, event, NULL, ep_ptr); } - /* * PASSIVE: Create socket, listen, accept, exchange QP information */ -static DAT_RETURN -dapli_socket_listen ( DAPL_IA *ia_ptr, - DAT_CONN_QUAL serviceID, - DAPL_SP *sp_ptr ) +DAT_RETURN +dapli_socket_listen(DAPL_IA *ia_ptr, + DAT_CONN_QUAL serviceID, + DAPL_SP *sp_ptr ) { struct sockaddr_in addr; ib_cm_srvc_handle_t cm_ptr = NULL; int opt = 1; DAT_RETURN dat_status = DAT_SUCCESS; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " listen(ia_ptr %p ServiceID %d sp_ptr %p)\n", - ia_ptr, serviceID, sp_ptr); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " listen(ia_ptr %p ServiceID %d sp_ptr %p)\n", + ia_ptr, serviceID, sp_ptr); - /* Allocate CM and initialize */ - if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) + cm_ptr = dapli_cm_create(); + if (cm_ptr == NULL) return DAT_INSUFFICIENT_RESOURCES; - (void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) ); - - cm_ptr->socket = cm_ptr->l_socket = -1; cm_ptr->sp = sp_ptr; - cm_ptr->hca_ptr = ia_ptr->hca_ptr; + cm_ptr->hca = ia_ptr->hca_ptr; /* bind, listen, set sockopt, accept, exchange data */ - if ((cm_ptr->l_socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) { + if ((cm_ptr->socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) { dapl_dbg_log (DAPL_DBG_TYPE_ERR, "socket for listen returned %d\n", errno); dat_status = DAT_INSUFFICIENT_RESOURCES; goto bail; } - setsockopt(cm_ptr->l_socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt)); + setsockopt(cm_ptr->socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt)); addr.sin_port = htons(serviceID); addr.sin_family = AF_INET; addr.sin_addr.s_addr = INADDR_ANY; - if (( bind( cm_ptr->l_socket,(struct sockaddr*)&addr, sizeof(addr) ) < 0) || - (listen( cm_ptr->l_socket, 128 ) < 0) ) { - + if ((bind(cm_ptr->socket,(struct sockaddr*)&addr, sizeof(addr)) < 0) || + (listen(cm_ptr->socket, 128) < 0)) { dapl_dbg_log( DAPL_DBG_TYPE_CM, " listen: ERROR %s on conn_qual 0x%x\n", strerror(errno),serviceID); - - if ( errno == EADDRINUSE ) + if (errno == EADDRINUSE) dat_status = DAT_CONN_QUAL_IN_USE; else dat_status = DAT_CONN_QUAL_UNAVAILABLE; - goto bail; } /* set cm_handle for this service point, save listen socket */ sp_ptr->cm_srvc_handle = cm_ptr; - /* add to SP->CR thread list */ - dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry); - dapl_os_lock( &cm_ptr->hca_ptr->ib_trans.lock ); - dapl_llist_add_tail(&cm_ptr->hca_ptr->ib_trans.list, - (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr); - dapl_os_unlock(&cm_ptr->hca_ptr->ib_trans.lock); + /* queue up listen socket to process inbound CR's */ + cm_ptr->state = SCM_LISTEN; + dapli_cm_queue(cm_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " listen: qual 0x%x cr %p s_fd %d\n", + ntohs(serviceID), cm_ptr, cm_ptr->socket ); - dapl_dbg_log( DAPL_DBG_TYPE_CM, - " listen: qual 0x%x cr %p s_fd %d\n", - ntohs(serviceID), cm_ptr, cm_ptr->l_socket ); - return dat_status; bail: dapl_dbg_log( DAPL_DBG_TYPE_CM, " listen: ERROR on conn_qual 0x%x\n",serviceID); - if ( cm_ptr->l_socket >= 0 ) - close( cm_ptr->l_socket ); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); + dapli_cm_destroy(cm_ptr); return dat_status; } /* - * PASSIVE: send local QP information, private data, and wait for - * active side to respond with QP RTS/RTR status + * PASSIVE: accept socket, receive peer QP information, private data, post cr_event */ -static DAT_RETURN +DAT_RETURN dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) { ib_cm_handle_t acm_ptr; @@ -334,6 +484,8 @@ dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) int len; DAT_RETURN dat_status = DAT_SUCCESS; + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket_accept\n"); + /* Allocate accept CM and initialize */ if ((acm_ptr = dapl_os_alloc(sizeof(*acm_ptr))) == NULL) return DAT_INSUFFICIENT_RESOURCES; @@ -342,155 +494,221 @@ dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) acm_ptr->socket = -1; acm_ptr->sp = cm_ptr->sp; - acm_ptr->hca_ptr = cm_ptr->hca_ptr; + acm_ptr->hca = cm_ptr->hca; len = sizeof(acm_ptr->dst.ia_address); - acm_ptr->socket = accept(cm_ptr->l_socket, + acm_ptr->socket = accept(cm_ptr->socket, (struct sockaddr*)&acm_ptr->dst.ia_address, - (socklen_t*)&len ); + (socklen_t*)&len); - if ( acm_ptr->socket < 0 ) { + if (acm_ptr->socket < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " accept: ERR %s on FD %d l_cr %p\n", - strerror(errno),cm_ptr->l_socket,cm_ptr); + strerror(errno),cm_ptr->socket,cm_ptr); dat_status = DAT_INTERNAL_ERROR; goto bail; } + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket accepted, read QP data\n"); + /* read in DST QP info, IA address. check for private data */ - len = read( acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t) ); - if ( len != sizeof(ib_qp_cm_t) ) { + len = read(acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t)); + if (len != sizeof(ib_qp_cm_t) || + ntohs(acm_ptr->dst.ver) != DSCM_VER) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept read: ERR %s, rcnt=%d\n", - strerror(errno), len); + " accept read: ERR %s, rcnt=%d, ver=%d\n", + strerror(errno), len, acm_ptr->dst.ver); dat_status = DAT_INTERNAL_ERROR; goto bail; - } + + /* convert accepted values to host order */ + acm_ptr->dst.port = ntohs(acm_ptr->dst.port); + acm_ptr->dst.lid = ntohs(acm_ptr->dst.lid); + acm_ptr->dst.qpn = ntohl(acm_ptr->dst.qpn); + acm_ptr->dst.p_size = ntohl(acm_ptr->dst.p_size); + dapl_dbg_log(DAPL_DBG_TYPE_EP, - " accept: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", - acm_ptr->dst.port, acm_ptr->dst.lid, - acm_ptr->dst.qpn, acm_ptr->dst.p_size ); + " accept: DST %s port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", + inet_ntoa(((struct sockaddr_in *)&acm_ptr->dst.ia_address)->sin_addr), + acm_ptr->dst.port, acm_ptr->dst.lid, + acm_ptr->dst.qpn, acm_ptr->dst.p_size); /* validate private data size before reading */ - if ( acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE ) { + if (acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept read: psize (%d) wrong\n", - acm_ptr->dst.p_size ); + " accept read: psize (%d) wrong\n", + acm_ptr->dst.p_size); dat_status = DAT_INTERNAL_ERROR; goto bail; } + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket accepted, read private data\n"); + /* read private data into cm_handle if any present */ - if ( acm_ptr->dst.p_size ) { + if (acm_ptr->dst.p_size) { len = read( acm_ptr->socket, - acm_ptr->p_data, acm_ptr->dst.p_size ); - if ( len != acm_ptr->dst.p_size ) { + acm_ptr->p_data, acm_ptr->dst.p_size); + if (len != acm_ptr->dst.p_size) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept read pdata: ERR %s, rcnt=%d\n", - strerror(errno), len ); + " accept read pdata: ERR %s, rcnt=%d\n", + strerror(errno), len); dat_status = DAT_INTERNAL_ERROR; goto bail; } - dapl_dbg_log(DAPL_DBG_TYPE_EP, - " accept: psize=%d read\n", - acm_ptr->dst.p_size); + dapl_dbg_log(DAPL_DBG_TYPE_EP," accept: psize=%d read\n",len); p_data = acm_ptr->p_data; } - /* trigger CR event and return SUCCESS */ - dapls_cr_callback( acm_ptr, - IB_CME_CONNECTION_REQUEST_PENDING, - p_data, - acm_ptr->sp ); + acm_ptr->state = SCM_ACCEPTING; + /* trigger CR event and return SUCCESS */ + dapls_cr_callback(acm_ptr, + IB_CME_CONNECTION_REQUEST_PENDING, + p_data, + acm_ptr->sp ); return DAT_SUCCESS; - bail: - if ( acm_ptr->socket >=0 ) - close( acm_ptr->socket ); - dapl_os_free( acm_ptr, sizeof( *acm_ptr ) ); + dapli_cm_destroy(acm_ptr); return DAT_INTERNAL_ERROR; } - -static DAT_RETURN -dapli_socket_accept_final( DAPL_EP *ep_ptr, - DAPL_CR *cr_ptr, - DAT_COUNT p_size, - DAT_PVOID p_data ) +/* + * PASSIVE: consumer accept, send local QP information, private data, + * queue on work thread to receive RTU information to avoid blocking + * user thread. + */ +DAT_RETURN +dapli_socket_accept_usr(DAPL_EP *ep_ptr, + DAPL_CR *cr_ptr, + DAT_COUNT p_size, + DAT_PVOID p_data) { DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; - ib_cm_handle_t cm_ptr = cr_ptr->ib_cm_handle; - ib_qp_cm_t qp_cm; + ib_cm_handle_t cm_ptr = cr_ptr->ib_cm_handle; struct iovec iovec[2]; int len; - short rtu_data = 0; - if (p_size > IB_MAX_REP_PDATA_SIZE) + if (p_size > IB_MAX_REP_PDATA_SIZE) return DAT_LENGTH_ERROR; /* must have a accepted socket */ - if ( cm_ptr->socket < 0 ) + if (cm_ptr->socket < 0) return DAT_INTERNAL_ERROR; + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " accept_usr: remote port=0x%x lid=0x%x" + " qpn=0x%x psize=%d\n", + cm_ptr->dst.port, cm_ptr->dst.lid, + cm_ptr->dst.qpn, cm_ptr->dst.p_size); + /* modify QP to RTR and then to RTS with remote info already read */ - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTR, &cm_ptr->dst) != DAT_SUCCESS) goto bail; - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTS, &cm_ptr->dst) != DAT_SUCCESS) goto bail; ep_ptr->qp_state = IB_QP_STATE_RTS; - /* Send QP info, IA address, and private data */ - qp_cm.qpn = ep_ptr->qp_handle->qp_num; - qp_cm.port = ia_ptr->hca_ptr->port_num; - qp_cm.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, - ia_ptr->hca_ptr->port_num ); - qp_cm.ia_address = ia_ptr->hca_ptr->hca_address; - qp_cm.p_size = p_size; - iovec[0].iov_base = &qp_cm; + /* save remote address information */ + dapl_os_memcpy( &ep_ptr->remote_ia_address, + &cm_ptr->dst.ia_address, + sizeof(ep_ptr->remote_ia_address)); + + /* send our QP info, IA address, and private data */ + cm_ptr->dst.qpn = htonl(ep_ptr->qp_handle->qp_num); + cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); + cm_ptr->dst.lid = htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num)); + if (cm_ptr->dst.lid == 0xffff) + goto bail; + + /* in network order */ + if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num, + 0, + &cm_ptr->dst.gid)) + goto bail; + + cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; + cm_ptr->dst.p_size = htonl(p_size); + iovec[0].iov_base = &cm_ptr->dst; iovec[0].iov_len = sizeof(ib_qp_cm_t); if (p_size) { iovec[1].iov_base = p_data; iovec[1].iov_len = p_size; } - len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) ); + len = writev(cm_ptr->socket, iovec, (p_size ? 2:1)); if (len != (p_size + sizeof(ib_qp_cm_t))) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept_final: ERR %s, wcnt=%d\n", + " accept_rtu: ERR %s, wcnt=%d\n", strerror(errno), len); goto bail; } - dapl_dbg_log(DAPL_DBG_TYPE_EP, - " accept_final: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", - qp_cm.port, qp_cm.lid, qp_cm.qpn, qp_cm.p_size ); - + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " accept_usr: local port=0x%x lid=0x%x" + " qpn=0x%x psize=%d\n", + ntohs(cm_ptr->dst.port), ntohs(cm_ptr->dst.lid), + ntohl(cm_ptr->dst.qpn), ntohl(cm_ptr->dst.p_size)); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " accept_usr SRC GID subnet %016llx id %016llx\n", + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.subnet_prefix), + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.interface_id)); + + /* save state and reference to EP, queue for RTU data */ + cm_ptr->ep = ep_ptr; + cm_ptr->hca = ia_ptr->hca_ptr; + cm_ptr->state = SCM_ACCEPTED; + + /* restore remote address information for query */ + dapl_os_memcpy( &cm_ptr->dst.ia_address, + &ep_ptr->remote_ia_address, + sizeof(cm_ptr->dst.ia_address)); + + dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: accepted!\n" ); + dapli_cm_queue(cm_ptr); + return DAT_SUCCESS; +bail: + dapl_dbg_log(DAPL_DBG_TYPE_ERR," accept_rtu: ERR !QP_RTR_RTS \n"); + dapli_cm_destroy(cm_ptr); + dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ + return DAT_INTERNAL_ERROR; +} + +/* + * PASSIVE: read RTU from active peer, post CONN event + */ +void +dapli_socket_accept_rtu(ib_cm_handle_t cm_ptr) +{ + int len; + short rtu_data = 0; + /* complete handshake after final QP state change */ - len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data) ); - if ( len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f ) { + len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data)); + if (len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept_final: ERR %s, rcnt=%d rdata=%x\n", - strerror(errno), len, ntohs(rtu_data) ); + " accept_rtu: ERR %s, rcnt=%d rdata=%x\n", + strerror(errno), len, ntohs(rtu_data)); goto bail; } + /* save state and reference to EP, queue for disc event */ + cm_ptr->state = SCM_CONNECTED; + /* final data exchange if remote QP state is good to go */ dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: connected!\n" ); - dapls_cr_callback ( cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp ); - return DAT_SUCCESS; - + dapls_cr_callback(cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp); + return; bail: - dapl_dbg_log( DAPL_DBG_TYPE_ERR," accept_final: ERR !QP_RTR_RTS \n"); - if ( cm_ptr >= 0 ) - close( cm_ptr->socket ); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); - dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */ - - return DAT_INTERNAL_ERROR; + dapls_ib_reinit_ep(cm_ptr->ep); /* reset QP state */ + dapli_cm_destroy(cm_ptr); + dapls_cr_callback(cm_ptr, IB_CME_DESTINATION_REJECT, NULL, cm_ptr->sp); } @@ -528,18 +746,13 @@ dapls_ib_connect ( dapl_dbg_log ( DAPL_DBG_TYPE_EP, " connect(ep_handle %p ....)\n", ep_handle); - /* - * Sanity check - */ - if ( NULL == ep_handle ) - return DAT_SUCCESS; ep_ptr = (DAPL_EP*)ep_handle; qp_ptr = ep_ptr->qp_handle; - return (dapli_socket_connect( ep_ptr, remote_ia_address, - remote_conn_qual, - private_data_size, private_data )); + return (dapli_socket_connect(ep_ptr, remote_ia_address, + remote_conn_qual, + private_data_size, private_data)); } /* @@ -556,12 +769,11 @@ dapls_ib_connect ( * * Returns: * DAT_SUCCESS - * */ DAT_RETURN -dapls_ib_disconnect ( +dapls_ib_disconnect( IN DAPL_EP *ep_ptr, - IN DAT_CLOSE_FLAGS close_flags ) + IN DAT_CLOSE_FLAGS close_flags) { ib_cm_handle_t cm_ptr = ep_ptr->cm_handle; @@ -569,28 +781,13 @@ dapls_ib_disconnect ( "dapls_ib_disconnect(ep_handle %p ....)\n", ep_ptr); - if ( cm_ptr->socket >= 0 ) { - close( cm_ptr->socket ); - cm_ptr->socket = -1; - } - /* reinit to modify QP state */ dapls_ib_reinit_ep(ep_ptr); - if ( ep_ptr->cr_ptr ) { - dapls_cr_callback ( ep_ptr->cm_handle, - IB_CME_DISCONNECTED, - NULL, - ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr ); - } else { - dapl_evd_connection_callback ( ep_ptr->cm_handle, - IB_CME_DISCONNECTED, - NULL, - ep_ptr ); - ep_ptr->cm_handle = NULL; - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); - } - return DAT_SUCCESS; + if (cm_ptr == NULL) + return DAT_SUCCESS; + else + return(dapli_socket_disconnect(cm_ptr)); } /* @@ -679,13 +876,14 @@ dapls_ib_remove_conn_listener ( ia_ptr, sp_ptr, cm_ptr ); /* close accepted socket, free cm_srvc_handle and return */ - if ( cm_ptr != NULL ) { - if ( cm_ptr->l_socket >= 0 ) { - close( cm_ptr->l_socket ); + if (cm_ptr != NULL) { + if (cm_ptr->socket >= 0) { + close(cm_ptr->socket ); cm_ptr->socket = -1; } /* cr_thread will free */ sp_ptr->cm_srvc_handle = NULL; + write(g_scm_pipe[1], "w", sizeof "w"); } return DAT_SUCCESS; } @@ -720,23 +918,22 @@ dapls_ib_accept_connection ( DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - "dapls_ib_accept_connection(cr %p ep %p prd %p,%d)\n", - cr_handle, ep_handle, p_data, p_size ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + "dapls_ib_accept_connection(cr %p ep %p prd %p,%d)\n", + cr_handle, ep_handle, p_data, p_size ); - cr_ptr = (DAPL_CR *) cr_handle; - ep_ptr = (DAPL_EP *) ep_handle; + cr_ptr = (DAPL_CR *)cr_handle; + ep_ptr = (DAPL_EP *)ep_handle; /* allocate and attach a QP if necessary */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) { + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { DAT_RETURN status; - status = dapls_ib_qp_alloc( ep_ptr->header.owner_ia, - ep_ptr, ep_ptr ); - if ( status != DAT_SUCCESS ) + status = dapls_ib_qp_alloc(ep_ptr->header.owner_ia, + ep_ptr, ep_ptr); + if (status != DAT_SUCCESS) return status; } - - return ( dapli_socket_accept_final(ep_ptr, cr_ptr, p_size, p_data) ); + return(dapli_socket_accept_usr(ep_ptr, cr_ptr, p_size, p_data)); } @@ -759,19 +956,29 @@ dapls_ib_accept_connection ( DAT_RETURN dapls_ib_reject_connection ( IN ib_cm_handle_t ib_cm_handle, - IN int reject_reason ) + IN int reject_reason) { ib_cm_srvc_handle_t cm_ptr = ib_cm_handle; + struct iovec iovec; dapl_dbg_log (DAPL_DBG_TYPE_EP, "dapls_ib_reject_connection(cm_handle %p reason %x)\n", - ib_cm_handle, reject_reason ); - - /* just close the socket and return */ - if ( cm_ptr->socket > 0 ) { - close( cm_ptr->socket ); + ib_cm_handle, reject_reason); + + /* write reject data to indicate reject */ + if (cm_ptr->socket >= 0) { + cm_ptr->dst.rej = (uint16_t)reject_reason; + cm_ptr->dst.rej = htons(cm_ptr->dst.rej); + iovec.iov_base = &cm_ptr->dst; + iovec.iov_len = sizeof(ib_qp_cm_t); + writev(cm_ptr->socket, &iovec, 1); + close(cm_ptr->socket); cm_ptr->socket = -1; } + + /* cr_thread will destroy CR */ + cm_ptr->state = SCM_REJECTED; + write(g_scm_pipe[1], "w", sizeof "w"); return DAT_SUCCESS; } @@ -991,24 +1198,25 @@ dapls_ib_get_cm_event ( return ib_cm_event; } -/* async CR processing thread to avoid blocking applications */ +/* outbound/inbound CR processing thread to avoid blocking applications */ +#define SCM_MAX_CONN 8192 void cr_thread(void *arg) { struct dapl_hca *hca_ptr = arg; - ib_cm_srvc_handle_t cr, next_cr; - int max_fd; - fd_set rfd,rfds; - struct timeval to; + ib_cm_handle_t cr, next_cr; + int ret,idx; + char rbuf[2]; + struct pollfd ufds[SCM_MAX_CONN]; dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cr_thread: ENTER hca %p\n",hca_ptr); dapl_os_lock( &hca_ptr->ib_trans.lock ); hca_ptr->ib_trans.cr_state = IB_THREAD_RUN; while (hca_ptr->ib_trans.cr_state == IB_THREAD_RUN) { - - FD_ZERO( &rfds ); - max_fd = -1; - + idx=0; + ufds[idx].fd = g_scm_pipe[0]; /* wakeup and process work */ + ufds[idx].events = POLLIN; + if (!dapl_llist_is_empty(&hca_ptr->ib_trans.list)) next_cr = dapl_llist_peek_head (&hca_ptr->ib_trans.list); else @@ -1016,51 +1224,70 @@ void cr_thread(void *arg) while (next_cr) { cr = next_cr; - dapl_dbg_log (DAPL_DBG_TYPE_CM," thread: cm_ptr %p\n", cr ); - if (cr->l_socket == -1 || + if ((cr->socket == -1) || hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) { - dapl_dbg_log(DAPL_DBG_TYPE_CM," thread: Freeing %p\n", cr); + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: Free %p\n", cr); next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list, (DAPL_LLIST_ENTRY*)&cr->entry ); dapl_llist_remove_entry(&hca_ptr->ib_trans.list, (DAPL_LLIST_ENTRY*)&cr->entry); - dapl_os_free( cr, sizeof(*cr) ); + dapl_os_free(cr, sizeof(*cr)); continue; } - - FD_SET( cr->l_socket, &rfds ); /* add to select set */ - if ( cr->l_socket > max_fd ) - max_fd = cr->l_socket; - - /* individual select poll to check for work */ - FD_ZERO(&rfd); - FD_SET(cr->l_socket, &rfd); - dapl_os_unlock(&hca_ptr->ib_trans.lock); - to.tv_sec = 0; - to.tv_usec = 0; - if ( select(cr->l_socket + 1,&rfd, NULL, NULL, &to) < 0) { - dapl_dbg_log (DAPL_DBG_TYPE_CM, - " thread: ERR %s on cr %p sk %d\n", - strerror(errno), cr, cr->l_socket); - close(cr->l_socket); - cr->l_socket = -1; - } else if ( FD_ISSET(cr->l_socket, &rfd) && - dapli_socket_accept(cr)) { - close(cr->l_socket); - cr->l_socket = -1; + + if (idx==SCM_MAX_CONN-1) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "SCM ERR: cm_thread exceeded FD_SETSIZE %d\n",idx+1); + continue; } - dapl_os_lock( &hca_ptr->ib_trans.lock ); + + /* Add to ufds for poll, check for immediate work */ + ufds[++idx].fd = cr->socket; /* add listen or cr */ + ufds[idx].events = POLLIN; + + /* check socket for event, accept in or connect out */ + dapl_dbg_log(DAPL_DBG_TYPE_CM," poll cr=%p, fd=%d,%d\n", + cr, cr->socket, ufds[idx].fd); + dapl_os_unlock(&hca_ptr->ib_trans.lock); + ret = poll(&ufds[idx],1,1); + dapl_dbg_log(DAPL_DBG_TYPE_CM," poll wakeup ret=%d cr->st=%d ev=%d fd=%d\n", + ret,cr->state,ufds[idx].revents,ufds[idx].fd); + + /* data on listen, qp exchange, and on disconnect request */ + if ((ret == 1) && ufds[idx].revents == POLLIN) { + if (cr->socket > 0) { + if (cr->state == SCM_LISTEN) + dapli_socket_accept(cr); + else if (cr->state == SCM_ACCEPTED) + dapli_socket_accept_rtu(cr); + else if (cr->state == SCM_CONN_PENDING) + dapli_socket_connect_rtu(cr); + else if (cr->state == SCM_CONNECTED) + dapli_socket_disconnect(cr); + } + } else if (ret != 0) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " cr_thread(cr=%p) st=%d poll ERR= %s\n", + cr,cr->state,strerror(errno)); + /* POLLUP or poll error case, issue event if connected */ + if (cr->state == SCM_CONNECTED) + dapli_socket_disconnect(cr); + } + dapl_os_lock(&hca_ptr->ib_trans.lock); next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list, - (DAPL_LLIST_ENTRY*)&cr->entry ); + (DAPL_LLIST_ENTRY*)&cr->entry); } - dapl_os_unlock( &hca_ptr->ib_trans.lock ); - to.tv_sec = 0; - to.tv_usec = 100000; /* wakeup and check destroy */ - select(max_fd + 1, &rfds, NULL, NULL, &to); - dapl_os_lock( &hca_ptr->ib_trans.lock ); + dapl_os_unlock(&hca_ptr->ib_trans.lock); + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: sleep, %d\n", idx+1); + poll(ufds,idx+1,-1); /* infinite, all sockets and pipe */ + /* if pipe used to wakeup, consume */ + if (ufds[0].revents == POLLIN) + read(g_scm_pipe[0], rbuf, 2); + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: wakeup\n"); + dapl_os_lock(&hca_ptr->ib_trans.lock); } - dapl_os_unlock( &hca_ptr->ib_trans.lock ); + dapl_os_unlock(&hca_ptr->ib_trans.lock); hca_ptr->ib_trans.cr_state = IB_THREAD_EXIT; dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cr_thread(hca %p) exit\n",hca_ptr); } diff --git a/dapl/openib_scm/dapl_ib_cq.c b/dapl/openib_scm/dapl_ib_cq.c index 7ac7037..56b729e 100644 --- a/dapl/openib_scm/dapl_ib_cq.c +++ b/dapl/openib_scm/dapl_ib_cq.c @@ -97,7 +97,7 @@ void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr) while (hca_ptr->ib_trans.cq_state != IB_THREAD_EXIT) { struct timespec sleep, remain; sleep.tv_sec = 0; - sleep.tv_nsec = 200000000; /* 200 ms */ + sleep.tv_nsec = 2000000; /* 2 ms */ dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " cq_thread_destroy: waiting for cq_thread\n"); nanosleep (&sleep, &remain); @@ -422,12 +422,21 @@ DAT_RETURN dapls_ib_cq_free ( IN DAPL_IA *ia_ptr, IN DAPL_EVD *evd_ptr) { - if ( evd_ptr->ib_cq_handle != IB_INVALID_HANDLE ) { - /* copy all entries on CQ to EVD before destroying */ - dapls_evd_copy_cq(evd_ptr); + DAT_EVENT event; + + if (evd_ptr->ib_cq_handle != IB_INVALID_HANDLE) { + /* pull off CQ and EVD entries and toss */ + while (dapls_ib_completion_poll == DAT_SUCCESS); + while (dapl_evd_dequeue(evd_ptr,&event) != DAT_QUEUE_EMPTY); +#if 1 + ibv_destroy_cq(evd_ptr->ib_cq_handle); + evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; + return DAT_SUCCESS; +#else if (ibv_destroy_cq(evd_ptr->ib_cq_handle)) return(dapl_convert_errno(errno,"destroy_cq")); evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; +#endif } return DAT_SUCCESS; } @@ -600,7 +609,7 @@ dapls_ib_wait_object_wait ( status = ETIMEDOUT; dapl_dbg_log (DAPL_DBG_TYPE_CM, - " cq_object_wait: RET evd %p ibv_cq %p ibv_ctx %p %s\n", + " cq_object_wait: RET evd %p ibv_cq %p %s\n", evd_ptr, ibv_cq,strerror(errno)); return(dapl_convert_errno(status,"cq_wait_object_wait")); diff --git a/dapl/openib_scm/dapl_ib_dto.h b/dapl/openib_scm/dapl_ib_dto.h index bea3e4d..b15f347 100644 --- a/dapl/openib_scm/dapl_ib_dto.h +++ b/dapl/openib_scm/dapl_ib_dto.h @@ -35,7 +35,7 @@ * * Description: * - * The uDAPL openib provider - DTO operations and CQE macros + * The OpenIB SCM (socket CM) provider - DTO operations and CQE macros * ************************************************************************ **** * Source Control System Information @@ -61,26 +61,25 @@ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p); */ STATIC _INLINE_ DAT_RETURN dapls_ib_post_recv ( - IN DAPL_EP *ep_ptr, + IN DAPL_EP *ep_ptr, IN DAPL_COOKIE *cookie, - IN DAT_COUNT segments, + IN DAT_COUNT segments, IN DAT_LMR_TRIPLET *local_iov ) { - ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; - ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; - struct ibv_recv_wr wr; - struct ibv_recv_wr *bad_wr; - DAT_COUNT i, total_len; - int ret; + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; + ib_data_segment_t *ds_array_p; + struct ibv_recv_wr wr; + struct ibv_recv_wr *bad_wr; + DAT_COUNT i, total_len; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_rcv: ep %p cookie %p segs %d l_iov %p\n", - ep_ptr, cookie, segments, local_iov); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_rcv: ep %p cookie %p segs %d l_iov %p\n", + ep_ptr, cookie, segments, local_iov); - if ( segments <= DEFAULT_DS_ENTRIES ) + if (segments <= DEFAULT_DS_ENTRIES) ds_array_p = ds_array; else - ds_array_start_p = ds_array_p = + ds_array_p = dapl_os_alloc(segments * sizeof(ib_data_segment_t)); if (NULL == ds_array_p) @@ -93,18 +92,18 @@ dapls_ib_post_recv ( wr.wr_id = (uint64_t)(uintptr_t)cookie; wr.sg_list = ds_array_p; - for (i = 0; i < segments; i++ ) { - if ( !local_iov[i].segment_length ) + for (i = 0; i < segments; i++) { + if (!local_iov[i].segment_length) continue; - ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; + ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; ds_array_p->length = local_iov[i].segment_length; - ds_array_p->lkey = local_iov[i].lmr_context; + ds_array_p->lkey = local_iov[i].lmr_context; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " post_rcv: l_key 0x%x va %p len %d\n", - ds_array_p->lkey, ds_array_p->addr, - ds_array_p->length ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_rcv: l_key 0x%x va %p len %d\n", + ds_array_p->lkey, ds_array_p->addr, + ds_array_p->length ); total_len += ds_array_p->length; wr.num_sge++; @@ -114,18 +113,12 @@ dapls_ib_post_recv ( if (cookie != NULL) cookie->val.dto.size = total_len; - ret = ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr); + if (ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr)) + return( dapl_convert_errno(errno,"ibv_recv") ); - if (ds_array_start_p != NULL) - dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); - - if (ret) - return( dapl_convert_errno(EFAULT,"ibv_recv") ); - return DAT_SUCCESS; } - /* * dapls_ib_post_send * @@ -133,35 +126,36 @@ dapls_ib_post_recv ( */ STATIC _INLINE_ DAT_RETURN dapls_ib_post_send ( - IN DAPL_EP *ep_ptr, - IN ib_send_op_type_t op_type, - IN DAPL_COOKIE *cookie, - IN DAT_COUNT segments, - IN DAT_LMR_TRIPLET *local_iov, - IN const DAT_RMR_TRIPLET *remote_iov, - IN DAT_COMPLETION_FLAGS completion_flags) + IN DAPL_EP *ep_ptr, + IN ib_send_op_type_t op_type, + IN DAPL_COOKIE *cookie, + IN DAT_COUNT segments, + IN DAT_LMR_TRIPLET *local_iov, + IN const DAT_RMR_TRIPLET *remote_iov, + IN DAT_COMPLETION_FLAGS completion_flags) { - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_snd: ep %p op %d ck %p sgs %d l_iov %p r_iov %p f %d\n", - ep_ptr, op_type, cookie, segments, local_iov, - remote_iov, completion_flags); - - ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; - ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; - struct ibv_send_wr wr; - struct ibv_send_wr *bad_wr; - ib_hca_transport_t *ibt_ptr = &ep_ptr->header.owner_ia->hca_ptr->ib_trans; - DAT_COUNT i, total_len; - int ret; + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: ep %p op %d ck %p sgs", + "%d l_iov %p r_iov %p f %d\n", + ep_ptr, op_type, cookie, segments, local_iov, + remote_iov, completion_flags); + + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; + ib_data_segment_t *ds_array_p; + struct ibv_send_wr wr; + struct ibv_send_wr *bad_wr; + ib_hca_transport_t *ibt_ptr = + &ep_ptr->header.owner_ia->hca_ptr->ib_trans; + DAT_COUNT i, total_len; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_snd: ep %p cookie %p segs %d l_iov %p\n", - ep_ptr, cookie, segments, local_iov); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: ep %p cookie %p segs %d l_iov %p\n", + ep_ptr, cookie, segments, local_iov); - if( segments <= DEFAULT_DS_ENTRIES ) + if(segments <= DEFAULT_DS_ENTRIES) ds_array_p = ds_array; else - ds_array_start_p = ds_array_p = + ds_array_p = dapl_os_alloc(segments * sizeof(ib_data_segment_t)); if (NULL == ds_array_p) @@ -180,14 +174,14 @@ dapls_ib_post_send ( if ( !local_iov[i].segment_length ) continue; - ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; + ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; ds_array_p->length = local_iov[i].segment_length; - ds_array_p->lkey = local_iov[i].lmr_context; + ds_array_p->lkey = local_iov[i].lmr_context; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " post_snd: lkey 0x%x va %p len %d \n", - ds_array_p->lkey, ds_array_p->addr, - ds_array_p->length ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: lkey 0x%x va %p len %d\n", + ds_array_p->lkey, ds_array_p->addr, + ds_array_p->length ); total_len += ds_array_p->length; wr.num_sge++; @@ -196,20 +190,21 @@ dapls_ib_post_send ( if (cookie != NULL) cookie->val.dto.size = total_len; - + if ((op_type == OP_RDMA_WRITE) || (op_type == OP_RDMA_READ)) { wr.wr.rdma.remote_addr = remote_iov->target_address; wr.wr.rdma.rkey = remote_iov->rmr_context; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " post_snd_rdma: rkey 0x%x va %#016Lx\n", - wr.wr.rdma.rkey, wr.wr.rdma.remote_addr ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd_rdma: rkey 0x%x va %#016Lx\n", + wr.wr.rdma.rkey, wr.wr.rdma.remote_addr); } + /* inline data for send or write ops */ - if ((total_len <= ibt_ptr->max_inline_send ) && + if ((total_len <= ibt_ptr->max_inline_send) && ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE))) wr.send_flags |= IBV_SEND_INLINE; - + /* set completion flags in work request */ wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG & completion_flags) ? 0 : IBV_SEND_SIGNALED; @@ -218,24 +213,19 @@ dapls_ib_post_send ( wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG & completion_flags) ? IBV_SEND_SOLICITED : 0; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", - wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", + wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); - ret = ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr); + if (ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr)) + return( dapl_convert_errno(errno,"ibv_recv") ); - if (ds_array_start_p != NULL) - dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); - - if (ret) - return( dapl_convert_errno(EFAULT,"ibv_send") ); - - dapl_dbg_log (DAPL_DBG_TYPE_EP," post_snd: returned\n"); + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); return DAT_SUCCESS; } STATIC _INLINE_ DAT_RETURN -dapls_ib_optional_prv_dat ( +dapls_ib_optional_prv_dat( IN DAPL_CR *cr_ptr, IN const void *event_data, OUT DAPL_CR **cr_pp) @@ -243,34 +233,68 @@ dapls_ib_optional_prv_dat ( return DAT_SUCCESS; } +/* map Work Completions to DAPL WR operations */ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) { - switch (cqe_p->opcode) { + switch (cqe_p->opcode) { case IBV_WC_SEND: - return (OP_SEND); + return (OP_SEND); case IBV_WC_RDMA_WRITE: - return (OP_RDMA_WRITE); + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) + return (OP_RDMA_WRITE_IMM); + else + return (OP_RDMA_WRITE); case IBV_WC_RDMA_READ: - return (OP_RDMA_READ); + return (OP_RDMA_READ); case IBV_WC_COMP_SWAP: - return (OP_COMP_AND_SWAP); + return (OP_COMP_AND_SWAP); case IBV_WC_FETCH_ADD: - return (OP_FETCH_AND_ADD); + return (OP_FETCH_AND_ADD); case IBV_WC_BIND_MW: - return (OP_BIND_MW); + return (OP_BIND_MW); case IBV_WC_RECV: - return (OP_RECEIVE); + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) + return (OP_RECEIVE_IMM); + else + return (OP_RECEIVE); case IBV_WC_RECV_RDMA_WITH_IMM: - return (OP_RECEIVE_IMM); + return (OP_RECEIVE_IMM); default: - return (OP_INVALID); - } + return (OP_INVALID); + } +} + +#define DAPL_GET_CQE_OPTYPE(cqe_p) dapls_cqe_opcode(cqe_p) +#define DAPL_GET_CQE_WRID(cqe_p) ((ib_work_completion_t*)cqe_p)->wr_id +#define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t*)cqe_p)->status +#define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t*)cqe_p)->byte_len +#define DAPL_GET_CQE_IMMED_DATA(cqe_p) ((ib_work_completion_t*)cqe_p)->imm_data +#define DAPL_GET_CQE_VENDOR_ERR(cqe_p) ((ib_work_completion_t*)cqe_p)->vendor_err + +STATIC _INLINE_ char * dapls_dto_op_str(int op) +{ + static char *optable[] = + { + "OP_RDMA_WRITE", + "OP_RDMA_WRITE_IMM", + "OP_SEND", + "OP_SEND_IMM", + "OP_RDMA_READ", + "OP_COMP_AND_SWAP", + "OP_FETCH_AND_ADD", + "OP_RECEIVE", + "OP_RECEIVE_IMM", + "OP_BIND_MW", + 0 + }; + return ((op < 0 || op > 9) ? "Invalid CQE OP?" : optable[op]); } -#define DAPL_GET_CQE_OPTYPE(cqe_p) dapls_cqe_opcode(cqe_p) -#define DAPL_GET_CQE_WRID(cqe_p) ((ib_work_completion_t*)cqe_p)->wr_id -#define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t*)cqe_p)->status -#define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t*)cqe_p)->byte_len -#define DAPL_GET_CQE_IMMED_DATA(cqe_p) ((ib_work_completion_t*)cqe_p)->imm_data +static _INLINE_ char * +dapls_cqe_op_str(IN ib_work_completion_t *cqe_ptr) +{ + return dapls_dto_op_str(DAPL_GET_CQE_OPTYPE(cqe_ptr)); +} +#define DAPL_GET_CQE_OP_STR(cqe) dapls_cqe_op_str(cqe) #endif /* _DAPL_IB_DTO_H_ */ diff --git a/dapl/openib_scm/dapl_ib_mem.c b/dapl/openib_scm/dapl_ib_mem.c index de36c0f..6a5e4a2 100644 --- a/dapl/openib_scm/dapl_ib_mem.c +++ b/dapl/openib_scm/dapl_ib_mem.c @@ -1,4 +1,6 @@ /* + * Copyright (c) 2005-2007 Intel Corporation. All rights reserved. + * * This Software is licensed under one of the following licenses: * * 1) under the terms of the "Common Public License 1.0" a copy of which is @@ -25,12 +27,11 @@ /********************************************************************** * - * MODULE: dapl_det_mem.c + * MODULE: dapl_ib_mem.c * - * PURPOSE: Intel DET APIs: Memory windows, registration, - * and protection domain + * PURPOSE: Memory windows, registration, and protection domain * - * $Id: $ + * $Id:$ * **********************************************************************/ @@ -61,8 +62,7 @@ * */ STATIC _INLINE_ int -dapls_convert_privileges ( - IN DAT_MEM_PRIV_FLAGS privileges) +dapls_convert_privileges(IN DAT_MEM_PRIV_FLAGS privileges) { int access = 0; @@ -101,16 +101,15 @@ dapls_convert_privileges ( * */ DAT_RETURN -dapls_ib_pd_alloc ( - IN DAPL_IA *ia_ptr, - IN DAPL_PZ *pz ) +dapls_ib_pd_alloc(IN DAPL_IA *ia_ptr, IN DAPL_PZ *pz) { /* get a protection domain */ pz->pd_handle = ibv_alloc_pd(ia_ptr->hca_ptr->ib_hca_handle); if (!pz->pd_handle) return(dapl_convert_errno(ENOMEM,"alloc_pd")); - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " pd_alloc: pd_handle=%p\n", + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " pd_alloc: pd_handle=%p\n", pz->pd_handle ); return DAT_SUCCESS; @@ -134,13 +133,18 @@ dapls_ib_pd_alloc ( * */ DAT_RETURN -dapls_ib_pd_free ( - IN DAPL_PZ *pz ) +dapls_ib_pd_free(IN DAPL_PZ *pz ) { if (pz->pd_handle != IB_INVALID_HANDLE) { +#if 1 + ibv_dealloc_pd(pz->pd_handle); + pz->pd_handle = IB_INVALID_HANDLE; + return DAT_SUCCESS; +#else if (ibv_dealloc_pd(pz->pd_handle)) return(dapl_convert_errno(errno,"dealloc_pd")); pz->pd_handle = IB_INVALID_HANDLE; +#endif } return DAT_SUCCESS; } @@ -165,25 +169,24 @@ dapls_ib_pd_free ( * */ DAT_RETURN -dapls_ib_mr_register ( - IN DAPL_IA *ia_ptr, - IN DAPL_LMR *lmr, - IN DAT_PVOID virt_addr, - IN DAT_VLEN length, - IN DAT_MEM_PRIV_FLAGS privileges) +dapls_ib_mr_register(IN DAPL_IA *ia_ptr, + IN DAPL_LMR *lmr, + IN DAT_PVOID virt_addr, + IN DAT_VLEN length, + IN DAT_MEM_PRIV_FLAGS privileges) { ib_pd_handle_t ib_pd_handle; ib_pd_handle = ((DAPL_PZ *)lmr->param.pz_handle)->pd_handle; - dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, - " mr_register: ia=%p, lmr=%p va=%p ln=%d pv=0x%x\n", - ia_ptr, lmr, virt_addr, length, privileges ); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " mr_register: ia=%p, lmr=%p va=%p ln=%d pv=0x%x\n", + ia_ptr, lmr, virt_addr, length, privileges ); /* TODO: shared memory */ if (lmr->param.mem_type == DAT_MEM_TYPE_SHARED_VIRTUAL) { - dapl_dbg_log( DAPL_DBG_TYPE_ERR, - " mr_register_shared: NOT IMPLEMENTED\n"); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mr_register_shared: NOT IMPLEMENTED\n"); return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } @@ -200,16 +203,16 @@ dapls_ib_mr_register ( lmr->param.lmr_context = lmr->mr_handle->lkey; lmr->param.rmr_context = lmr->mr_handle->rkey; lmr->param.registered_size = length; - lmr->param.registered_address = (DAT_VADDR)(uintptr_t) virt_addr; + lmr->param.registered_address = (DAT_VADDR)(uintptr_t)virt_addr; - dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, - " mr_register: mr=%p h %x pd %p ctx %p ,lkey=0x%x, rkey=0x%x priv=%x\n", - lmr->mr_handle, lmr->mr_handle->handle, - lmr->mr_handle->pd, - lmr->mr_handle->context, - lmr->mr_handle->lkey, - lmr->mr_handle->rkey, - length, dapls_convert_privileges(privileges) ); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " mr_register: mr=%p addr=%p h %x pd %p ctx %p " + "lkey=0x%x rkey=0x%x priv=%x\n", + lmr->mr_handle, lmr->mr_handle->addr, + lmr->mr_handle->handle, + lmr->mr_handle->pd, lmr->mr_handle->context, + lmr->mr_handle->lkey, lmr->mr_handle->rkey, + length, dapls_convert_privileges(privileges)); return DAT_SUCCESS; } @@ -231,8 +234,7 @@ dapls_ib_mr_register ( * */ DAT_RETURN -dapls_ib_mr_deregister ( - IN DAPL_LMR *lmr ) +dapls_ib_mr_deregister(IN DAPL_LMR *lmr) { if (lmr->mr_handle != IB_INVALID_HANDLE) { if (ibv_dereg_mr(lmr->mr_handle)) @@ -263,13 +265,14 @@ dapls_ib_mr_deregister ( * */ DAT_RETURN -dapls_ib_mr_register_shared ( - IN DAPL_IA *ia_ptr, - IN DAPL_LMR *lmr, - IN DAT_MEM_PRIV_FLAGS privileges ) +dapls_ib_mr_register_shared(IN DAPL_IA *ia_ptr, + IN DAPL_LMR *lmr, + IN DAT_MEM_PRIV_FLAGS privileges) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mr_register_shared: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mr_register_shared: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* @@ -289,12 +292,13 @@ dapls_ib_mr_register_shared ( * */ DAT_RETURN -dapls_ib_mw_alloc ( - IN DAPL_RMR *rmr ) +dapls_ib_mw_alloc (IN DAPL_RMR *rmr) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_alloc: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_alloc: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* @@ -314,11 +318,12 @@ dapls_ib_mw_alloc ( * */ DAT_RETURN -dapls_ib_mw_free ( - IN DAPL_RMR *rmr ) +dapls_ib_mw_free(IN DAPL_RMR *rmr) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_free: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_free: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* @@ -339,17 +344,18 @@ dapls_ib_mw_free ( * */ DAT_RETURN -dapls_ib_mw_bind ( - IN DAPL_RMR *rmr, - IN DAPL_LMR *lmr, - IN DAPL_EP *ep, - IN DAPL_COOKIE *cookie, - IN DAT_VADDR virtual_address, - IN DAT_VLEN length, - IN DAT_MEM_PRIV_FLAGS mem_priv, - IN DAT_BOOLEAN is_signaled) +dapls_ib_mw_bind(IN DAPL_RMR *rmr, + IN DAPL_LMR *lmr, + IN DAPL_EP *ep, + IN DAPL_COOKIE *cookie, + IN DAT_VADDR virtual_address, + IN DAT_VLEN length, + IN DAT_MEM_PRIV_FLAGS mem_priv, + IN DAT_BOOLEAN is_signaled) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_bind: NOT IMPLEMENTED\n"); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_bind: NOT IMPLEMENTED\n"); + return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } @@ -372,14 +378,15 @@ dapls_ib_mw_bind ( * */ DAT_RETURN -dapls_ib_mw_unbind ( - IN DAPL_RMR *rmr, - IN DAPL_EP *ep, - IN DAPL_COOKIE *cookie, - IN DAT_BOOLEAN is_signaled ) +dapls_ib_mw_unbind(IN DAPL_RMR *rmr, + IN DAPL_EP *ep, + IN DAPL_COOKIE *cookie, + IN DAT_BOOLEAN is_signaled ) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_unbind: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_unbind: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index 3a1e3c8..1eba2bd 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -110,15 +110,23 @@ dapls_ib_qp_alloc ( /* Setup attributes and create qp */ dapl_os_memzero((void*)&qp_create, sizeof(qp_create)); qp_create.send_cq = req_cq; - qp_create.recv_cq = rcv_cq; qp_create.cap.max_send_wr = attr->max_request_dtos; - qp_create.cap.max_recv_wr = attr->max_recv_dtos; qp_create.cap.max_send_sge = attr->max_request_iov; - qp_create.cap.max_recv_sge = attr->max_recv_iov; qp_create.cap.max_inline_data = ia_ptr->hca_ptr->ib_trans.max_inline_send; qp_create.qp_type = IBV_QPT_RC; qp_create.qp_context = (void*)ep_ptr; + /* ibv assumes rcv_cq is never NULL, set to req_cq */ + if (rcv_cq == NULL) { + qp_create.recv_cq = req_cq; + qp_create.cap.max_recv_wr = 0; + qp_create.cap.max_recv_sge = 0; + } else { + qp_create.recv_cq = rcv_cq; + qp_create.cap.max_recv_wr = attr->max_recv_dtos; + qp_create.cap.max_recv_sge = attr->max_recv_iov; + } + ep_ptr->qp_handle = ibv_create_qp( ib_pd_handle, &qp_create); if (!ep_ptr->qp_handle) return(dapl_convert_errno(ENOMEM, "create_qp")); @@ -298,9 +306,10 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IN ib_qp_state_t qp_state, IN ib_qp_cm_t *qp_cm ) { - struct ibv_qp_attr qp_attr; + struct ibv_qp_attr qp_attr; enum ibv_qp_attr_mask mask = IBV_QP_STATE; - DAPL_EP *ep_ptr = (DAPL_EP*)qp_handle->qp_context; + DAPL_EP *ep_ptr = (DAPL_EP*)qp_handle->qp_context; + DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; dapl_os_memzero((void*)&qp_attr, sizeof(qp_attr)); qp_attr.qp_state = qp_state; @@ -315,14 +324,16 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER; + qp_attr.qp_state = IBV_QPS_RTR; - qp_attr.path_mtu = IBV_MTU_1024; + qp_attr.path_mtu = IBV_MTU_2048; qp_attr.dest_qp_num = qp_cm->qpn; qp_attr.rq_psn = 1; qp_attr.max_dest_rd_atomic = ep_ptr->param.ep_attr.max_rdma_read_out; - qp_attr.min_rnr_timer = 12; + qp_attr.min_rnr_timer = ia_ptr->hca_ptr->ib_trans.rnr_timer; qp_attr.ah_attr.is_global = 0; + qp_attr.ah_attr.grh.dgid = qp_cm->gid; qp_attr.ah_attr.dlid = qp_cm->lid; qp_attr.ah_attr.sl = 0; qp_attr.ah_attr.src_path_bits = 0; @@ -343,30 +354,25 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC; + qp_attr.qp_state = IBV_QPS_RTS; - qp_attr.timeout = 14; - qp_attr.retry_cnt = 7; - qp_attr.rnr_retry = 7; + qp_attr.timeout = ia_ptr->hca_ptr->ib_trans.ack_timer; + qp_attr.retry_cnt = ia_ptr->hca_ptr->ib_trans.ack_retry; + qp_attr.rnr_retry = ia_ptr->hca_ptr->ib_trans.rnr_retry; qp_attr.sq_psn = 1; qp_attr.max_rd_atomic = ep_ptr->param.ep_attr.max_rdma_read_out; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " modify_qp_rts: psn %x rd_atomic %d\n", - qp_attr.sq_psn, qp_attr.max_rd_atomic ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " modify_qp_rts: psn %x rd_atomic %d ack %d " + " retry %d rnr_retry %d\n", + qp_attr.sq_psn, qp_attr.max_rd_atomic, + qp_attr.timeout, qp_attr.retry_cnt, + qp_attr.rnr_retry ); break; } case IBV_QPS_INIT: { - DAPL_IA *ia_ptr; - DAPL_EP *ep_ptr; - /* need to find way back to port num */ - ep_ptr = (DAPL_EP*)qp_handle->qp_context; - if (ep_ptr) - ia_ptr = ep_ptr->header.owner_ia; - else - break; - mask |= IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS; @@ -377,7 +383,8 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ | - IBV_ACCESS_REMOTE_ATOMIC; + IBV_ACCESS_REMOTE_ATOMIC | + IBV_ACCESS_MW_BIND; dapl_dbg_log (DAPL_DBG_TYPE_EP, " modify_qp_init: pi %x port %x acc %x\n", diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index 5e34b47..a9941f5 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -62,6 +62,7 @@ static const char rcsid[] = "$Id: $"; #include int g_dapl_loopback_connection = 0; +int g_scm_pipe[2]; /* just get IP address for hostname */ DAT_RETURN getipaddr( char *addr, int addr_len) @@ -70,14 +71,14 @@ DAT_RETURN getipaddr( char *addr, int addr_len) struct hostent *h_ptr; struct utsname ourname; - if ( uname( &ourname ) < 0 ) + if (uname( &ourname ) < 0) return DAT_INTERNAL_ERROR; - h_ptr = gethostbyname( ourname.nodename ); - if ( h_ptr == NULL ) + h_ptr = gethostbyname(ourname.nodename); + if (h_ptr == NULL) return DAT_INTERNAL_ERROR; - if ( h_ptr->h_addrtype == AF_INET ) { + if (h_ptr->h_addrtype == AF_INET) { int i; struct in_addr **alist = (struct in_addr **)h_ptr->h_addr_list; @@ -87,18 +88,17 @@ DAT_RETURN getipaddr( char *addr, int addr_len) /* Walk the list of addresses for host */ for (i=0; alist[i] != NULL; i++) { - - /* first non-loopback address */ - if ( *(uint32_t*)alist[i] != htonl(0x7f000001) ) { - dapl_os_memcpy( &ipv4_addr->sin_addr, - h_ptr->h_addr_list[i], - 4 ); + /* first non-loopback address */ + if (*(uint32_t*)alist[i] != htonl(0x7f000001)) { + dapl_os_memcpy(&ipv4_addr->sin_addr, + h_ptr->h_addr_list[i], + 4); break; } } /* if no acceptable address found */ if (*(uint32_t*)&ipv4_addr->sin_addr == 0) - return DAT_INVALID_ADDRESS; + return DAT_INVALID_ADDRESS; } else return DAT_INVALID_ADDRESS; @@ -122,6 +122,10 @@ DAT_RETURN getipaddr( char *addr, int addr_len) */ int32_t dapls_ib_init (void) { + /* create pipe for waking up thread */ + if (pipe(g_scm_pipe)) + return 1; + return 0; } @@ -156,7 +160,7 @@ DAT_RETURN dapls_ib_open_hca ( int i; DAT_RETURN dat_status = DAT_SUCCESS; - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " open_hca: %s - %p\n", hca_name, hca_ptr ); /* Get list of all IB devices, find match, open */ @@ -170,65 +174,83 @@ DAT_RETURN dapls_ib_open_hca ( for (i = 0; dev_list[i]; ++i) { hca_ptr->ib_trans.ib_dev = dev_list[i]; - if (!strcmp(ibv_get_device_name(hca_ptr->ib_trans.ib_dev),hca_name)) + if (!strcmp(ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + hca_name)) goto found; } - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: IB device %s not found\n", - hca_name); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: device %s not found\n", + hca_name); goto err; found: - dapl_dbg_log (DAPL_DBG_TYPE_UTIL," open_hca: Found dev %s %016llx\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), - (unsigned long long)bswap_64(ibv_get_device_guid(hca_ptr->ib_trans.ib_dev))); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL," open_hca: Found dev %s %016llx\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + (unsigned long long) + bswap_64(ibv_get_device_guid(hca_ptr->ib_trans.ib_dev))); hca_ptr->ib_hca_handle = ibv_open_device(hca_ptr->ib_trans.ib_dev); if (!hca_ptr->ib_hca_handle) { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: IB dev open failed for %s\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev) ); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: dev open failed for %s, err=%s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + strerror(errno)); goto err; } - /* set inline max with enviroment or default */ + /* set RC tunables via enviroment or default */ hca_ptr->ib_trans.max_inline_send = - dapl_os_get_env_val ( "DAPL_MAX_INLINE", INLINE_SEND_DEFAULT ); + dapl_os_get_env_val("DAPL_MAX_INLINE", INLINE_SEND_DEFAULT); + hca_ptr->ib_trans.ack_retry = + dapl_os_get_env_val("DAPL_ACK_RETRY", SCM_ACK_RETRY); + hca_ptr->ib_trans.ack_timer = + dapl_os_get_env_val("DAPL_ACK_TIMER", SCM_ACK_TIMER); + hca_ptr->ib_trans.rnr_retry = + dapl_os_get_env_val("DAPL_RNR_RETRY", SCM_RNR_RETRY); + hca_ptr->ib_trans.rnr_timer = + dapl_os_get_env_val("DAPL_RNR_TIMER", SCM_RNR_TIMER); /* initialize cq_lock */ dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.cq_lock); - if (dat_status != DAT_SUCCESS) - { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: failed to init cq_lock\n"); + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: failed to init cq_lock\n"); goto bail; } /* EVD events without direct CQ channels, non-blocking */ hca_ptr->ib_trans.ib_cq = ibv_create_comp_channel(hca_ptr->ib_hca_handle); + if (hca_ptr->ib_trans.ib_cq == NULL) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: ibv_create_comp_channel ERR %s\n", + strerror(errno)); + goto bail; + } + opts = fcntl(hca_ptr->ib_trans.ib_cq->fd, F_GETFL); /* uCQ */ if (opts < 0 || fcntl(hca_ptr->ib_trans.ib_cq->fd, F_SETFL, opts | O_NONBLOCK) < 0) { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: ERR with CQ FD\n" ); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: fcntl on ib_cq->fd %d ERR %d %s\n", + hca_ptr->ib_trans.ib_cq->fd, opts, + strerror(errno)); goto bail; } if (dapli_cq_thread_init(hca_ptr)) { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: cq_thread_init failed for %s\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev) ); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: cq_thread_init failed for %s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev)); goto bail; } /* initialize cr_list lock */ dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.lock); - if (dat_status != DAT_SUCCESS) - { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: failed to init lock\n"); + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: failed to init cr_list lock\n"); goto bail; } @@ -240,10 +262,9 @@ found: dat_status = dapl_os_thread_create(cr_thread, (void*)hca_ptr, &hca_ptr->ib_trans.thread ); - if (dat_status != DAT_SUCCESS) - { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: failed to create thread\n"); + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: failed to create thread\n"); goto bail; } @@ -251,7 +272,7 @@ found: while (hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) { struct timespec sleep, remain; sleep.tv_sec = 0; - sleep.tv_nsec = 20000000; /* 20 ms */ + sleep.tv_nsec = 2000000; /* 2 ms */ dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " open_hca: waiting for cr_thread\n"); nanosleep (&sleep, &remain); @@ -259,16 +280,15 @@ found: /* get the IP address of the device */ dat_status = getipaddr((char*)&hca_ptr->hca_address, - sizeof(DAT_SOCK_ADDR6) ); - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, - " open_hca: %s, port %d, %s %d.%d.%d.%d\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), hca_ptr->port_num, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_family == AF_INET ? "AF_INET":"AF_INET6", - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 0 & 0xff, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 8 & 0xff, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 16 & 0xff, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff ); - + sizeof(DAT_SOCK_ADDR6)); + + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " open_hca: devname %s, port %d, hostname_IP %s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + hca_ptr->port_num, + inet_ntoa(((struct sockaddr_in *) + &hca_ptr->hca_address)->sin_addr)); + ibv_free_device_list(dev_list); return dat_status; @@ -308,15 +328,15 @@ DAT_RETURN dapls_ib_close_hca ( IN DAPL_HCA *hca_ptr ) return(dapl_convert_errno(errno,"ib_close_device")); hca_ptr->ib_hca_handle = IB_INVALID_HANDLE; } - dapl_os_lock_destroy(&hca_ptr->ib_trans.cq_lock); /* destroy cr_thread and lock */ hca_ptr->ib_trans.cr_state = IB_THREAD_CANCEL; + write(g_scm_pipe[1], "w", sizeof "w"); while (hca_ptr->ib_trans.cr_state != IB_THREAD_EXIT) { struct timespec sleep, remain; sleep.tv_sec = 0; - sleep.tv_nsec = 20000000; /* 20 ms */ + sleep.tv_nsec = 2000000; /* 2 ms */ dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " close_hca: waiting for cr_thread\n"); nanosleep (&sleep, &remain); @@ -378,14 +398,11 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->vendor_name[DAT_NAME_MAX_LENGTH - 1] = '\0'; ia_attr->ia_address_ptr = (DAT_IA_ADDRESS_PTR)&hca_ptr->hca_address; - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, - " query_hca: %s %s %d.%d.%d.%d\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_family == AF_INET ? "AF_INET":"AF_INET6", - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 0 & 0xff, - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 8 & 0xff, - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 16 & 0xff, - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 24 & 0xff ); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " query_hca: %s %s \n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + inet_ntoa(((struct sockaddr_in *) + &hca_ptr->hca_address)->sin_addr)); ia_attr->hardware_version_major = dev_attr.hw_ver; /* ia_attr->hardware_version_minor = dev_attr.fw_ver; */ @@ -408,11 +425,14 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->max_pzs = dev_attr.max_pd; ia_attr->max_mtu_size = port_attr.max_msg_sz; ia_attr->max_rdma_size = port_attr.max_msg_sz; + ia_attr->max_iov_segments_per_rdma_read = dev_attr.max_sge; + ia_attr->max_iov_segments_per_rdma_write = dev_attr.max_sge; ia_attr->num_transport_attr = 0; ia_attr->transport_attr = NULL; ia_attr->num_vendor_attr = 0; ia_attr->vendor_attr = NULL; - + hca_ptr->ib_trans.ack_timer = DAPL_MAX(dev_attr.local_ca_ack_delay, + hca_ptr->ib_trans.ack_timer); dapl_dbg_log (DAPL_DBG_TYPE_UTIL, " query_hca: (%x.%x) ep %d ep_q %d evd %d evd_q %d\n", ia_attr->hardware_version_major, @@ -420,11 +440,10 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->max_eps, ia_attr->max_dto_per_ep, ia_attr->max_evds, ia_attr->max_evd_qlen ); dapl_dbg_log (DAPL_DBG_TYPE_UTIL, - " query_hca: msg %llu rdma %llu iov %d lmr %d rmr %d\n", + " query_hca: msg %llu rdma %llu iov %d lmr %d rmr %d ack_time %d\n", ia_attr->max_mtu_size, ia_attr->max_rdma_size, ia_attr->max_iov_segments_per_dto, ia_attr->max_lmrs, - ia_attr->max_rmrs ); - + ia_attr->max_rmrs,hca_ptr->ib_trans.ack_timer ); } if (ep_attr != NULL) { @@ -443,7 +462,6 @@ DAT_RETURN dapls_ib_query_hca ( ep_attr->max_recv_dtos, ep_attr->max_recv_iov, ep_attr->max_rdma_read_in, ep_attr->max_rdma_read_out); } - return DAT_SUCCESS; } diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index 0d928df..81a1752 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -71,15 +71,18 @@ typedef ib_hca_handle_t dapl_ibal_ca_t; /* CM mappings, user CM not complete use SOCKETS */ -/* destination info to exchange until real IB CM shows up */ +/* destination info to exchange, define wire protocol version */ +#define DSCM_VER 2 typedef struct _ib_qp_cm { - uint32_t qpn; + uint16_t ver; + uint16_t rej; uint16_t lid; uint16_t port; - int p_size; + uint32_t qpn; + uint32_t p_size; DAT_SOCK_ADDR6 ia_address; - + union ibv_gid gid; } ib_qp_cm_t; /* @@ -94,20 +97,34 @@ struct ib_llist_entry struct dapl_llist_entry *list_head; }; +typedef enum scm_state +{ + SCM_INIT, + SCM_LISTEN, + SCM_CONN_PENDING, + SCM_ACCEPTING, + SCM_ACCEPTED, + SCM_REJECTED, + SCM_CONNECTED, + SCM_DISCONNECTED, + SCM_DESTROY +} SCM_STATE; + struct ib_cm_handle { struct ib_llist_entry entry; + DAPL_OS_LOCK lock; + SCM_STATE state; int socket; - int l_socket; - struct dapl_hca *hca_ptr; - DAT_HANDLE cr; + struct dapl_hca *hca; DAT_HANDLE sp; + struct dapl_ep *ep; ib_qp_cm_t dst; unsigned char p_data[256]; }; typedef struct ib_cm_handle *ib_cm_handle_t; -typedef ib_cm_handle_t ib_cm_srvc_handle_t; +typedef ib_cm_handle_t ib_cm_srvc_handle_t; DAT_RETURN getipaddr(char *addr, int addr_len); @@ -163,6 +180,12 @@ typedef struct ibv_comp_channel *ib_wait_obj_handle_t; /* inline send rdma threshold */ #define INLINE_SEND_DEFAULT 128 +/* RC timer - retry count defaults */ +#define SCM_ACK_TIMER 15 /* 5 bits, 4.096us*2^ack_timer. 15 == 134ms */ +#define SCM_ACK_RETRY 7 /* 3 bits, 7 * 134ms = 940ms */ +#define SCM_RNR_TIMER 28 /* 5 bits, 28 == 163ms, 31 == 491ms */ +#define SCM_RNR_RETRY 7 /* 3 bits, 7 == infinite */ + /* CM private data areas */ #define IB_MAX_REQ_PDATA_SIZE 92 #define IB_MAX_REP_PDATA_SIZE 196 @@ -268,7 +291,10 @@ typedef struct _ib_hca_transport ib_async_cq_handler_t async_cq_error; ib_async_dto_handler_t async_cq; ib_async_qp_handler_t async_qp_error; - + uint8_t ack_timer; + uint8_t ack_retry; + uint8_t rnr_timer; + uint8_t rnr_retry; } ib_hca_transport_t; /* provider specfic fields for shared memory support */ -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:51:13 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:51:13 -0700 Subject: [ofa-general] [PATCH 5/7][v1.2] dapl scm: support global routing and set mtu based on active_mtu Message-ID: Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_scm/dapl_ib_qp.c | 42 +++++++++++++++++++++------------------ dapl/openib_scm/dapl_ib_util.c | 7 ++++++ dapl/openib_scm/dapl_ib_util.h | 11 +++++++++- 3 files changed, 40 insertions(+), 20 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index 1eba2bd..8577131 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -25,9 +25,9 @@ /********************************************************************** * - * MODULE: dapl_det_qp.c + * MODULE: dapl_ib_qp.c * - * PURPOSE: QP routines for access to DET Verbs + * PURPOSE: QP routines for access to ofa rdma verbs * * $Id: $ **********************************************************************/ @@ -285,17 +285,12 @@ void dapls_ib_reinit_ep ( IN DAPL_EP *ep_ptr) { - if ( ep_ptr->qp_handle != IB_INVALID_HANDLE ) { /* move to RESET state and then to INIT */ dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_RESET, 0); dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_INIT, 0); ep_ptr->qp_state = IB_QP_STATE_INIT; } - - /* TODO: When IB-CM is implement then handle timewait before - * allowing re-use of this QP - */ } /* @@ -325,19 +320,28 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER; - qp_attr.qp_state = IBV_QPS_RTR; - qp_attr.path_mtu = IBV_MTU_2048; - qp_attr.dest_qp_num = qp_cm->qpn; - qp_attr.rq_psn = 1; - qp_attr.max_dest_rd_atomic = + qp_attr.qp_state = IBV_QPS_RTR; + qp_attr.dest_qp_num = qp_cm->qpn; + qp_attr.rq_psn = 1; + qp_attr.path_mtu = + ia_ptr->hca_ptr->ib_trans.mtu; + qp_attr.max_dest_rd_atomic = ep_ptr->param.ep_attr.max_rdma_read_out; - qp_attr.min_rnr_timer = ia_ptr->hca_ptr->ib_trans.rnr_timer; - qp_attr.ah_attr.is_global = 0; - qp_attr.ah_attr.grh.dgid = qp_cm->gid; - qp_attr.ah_attr.dlid = qp_cm->lid; - qp_attr.ah_attr.sl = 0; - qp_attr.ah_attr.src_path_bits = 0; - qp_attr.ah_attr.port_num = qp_cm->port; + qp_attr.min_rnr_timer = + ia_ptr->hca_ptr->ib_trans.rnr_timer; + qp_attr.ah_attr.dlid = qp_cm->lid; + /* global routing */ + if (ia_ptr->hca_ptr->ib_trans.global) { + qp_attr.ah_attr.is_global = 1; + qp_attr.ah_attr.grh.dgid = qp_cm->gid; + qp_attr.ah_attr.grh.hop_limit = + ia_ptr->hca_ptr->ib_trans.hop_limit; + qp_attr.ah_attr.grh.traffic_class = + ia_ptr->hca_ptr->ib_trans.tclass; + } + qp_attr.ah_attr.sl = 0; + qp_attr.ah_attr.src_path_bits = 0; + qp_attr.ah_attr.port_num = qp_cm->port; dapl_dbg_log (DAPL_DBG_TYPE_EP, " modify_qp_rtr: qpn %x lid %x " diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index a9941f5..ecf6f6a 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -210,6 +210,12 @@ found: dapl_os_get_env_val("DAPL_RNR_RETRY", SCM_RNR_RETRY); hca_ptr->ib_trans.rnr_timer = dapl_os_get_env_val("DAPL_RNR_TIMER", SCM_RNR_TIMER); + hca_ptr->ib_trans.global = + dapl_os_get_env_val("DAPL_GLOBAL_ROUTING", SCM_GLOBAL); + hca_ptr->ib_trans.hop_limit = + dapl_os_get_env_val("DAPL_HOP_LIMIT", SCM_HOP_LIMIT); + hca_ptr->ib_trans.tclass = + dapl_os_get_env_val("DAPL_TCLASS", SCM_TCLASS); /* initialize cq_lock */ dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.cq_lock); @@ -433,6 +439,7 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->vendor_attr = NULL; hca_ptr->ib_trans.ack_timer = DAPL_MAX(dev_attr.local_ca_ack_delay, hca_ptr->ib_trans.ack_timer); + hca_ptr->ib_trans.mtu = port_attr.active_mtu; dapl_dbg_log (DAPL_DBG_TYPE_UTIL, " query_hca: (%x.%x) ep %d ep_q %d evd %d evd_q %d\n", ia_attr->hardware_version_major, diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index 81a1752..37c5dbb 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -186,6 +186,11 @@ typedef struct ibv_comp_channel *ib_wait_obj_handle_t; #define SCM_RNR_TIMER 28 /* 5 bits, 28 == 163ms, 31 == 491ms */ #define SCM_RNR_RETRY 7 /* 3 bits, 7 == infinite */ +/* Global routing defaults */ +#define SCM_GLOBAL 0 /* global routing is disabled */ +#define SCM_HOP_LIMIT 0xff +#define SCM_TCLASS 0 + /* CM private data areas */ #define IB_MAX_REQ_PDATA_SIZE 92 #define IB_MAX_REP_PDATA_SIZE 196 @@ -193,7 +198,7 @@ typedef struct ibv_comp_channel *ib_wait_obj_handle_t; #define IB_MAX_DREQ_PDATA_SIZE 220 #define IB_MAX_DREP_PDATA_SIZE 224 -/* DTO OPs, ordered for DAPL ENUM definitions ???*/ +/* DTO OPs, ordered for DAPL ENUM definitions */ #define OP_RDMA_WRITE IBV_WR_RDMA_WRITE #define OP_RDMA_WRITE_IMM IBV_WR_RDMA_WRITE_WITH_IMM #define OP_SEND IBV_WR_SEND @@ -295,6 +300,10 @@ typedef struct _ib_hca_transport uint8_t ack_retry; uint8_t rnr_timer; uint8_t rnr_retry; + uint8_t global; + uint8_t hop_limit; + uint8_t tclass; + uint8_t mtu; } ib_hca_transport_t; /* provider specfic fields for shared memory support */ -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:52:13 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:52:13 -0700 Subject: [ofa-general] [PATCH 6/7][v1.2] dapl scm: add stdout logging for uname and gethostbyname errors during open. Message-ID: Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_scm/dapl_ib_util.c | 11 +++++++++-- 1 files changed, 9 insertions(+), 2 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index ecf6f6a..96116c8 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -71,12 +71,19 @@ DAT_RETURN getipaddr( char *addr, int addr_len) struct hostent *h_ptr; struct utsname ourname; - if (uname( &ourname ) < 0) + if (uname(&ourname) < 0) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: uname err=%s\n", strerror(errno)); return DAT_INTERNAL_ERROR; + } h_ptr = gethostbyname(ourname.nodename); - if (h_ptr == NULL) + if (h_ptr == NULL) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: gethostbyname err=%s\n", + strerror(errno)); return DAT_INTERNAL_ERROR; + } if (h_ptr->h_addrtype == AF_INET) { int i; -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:53:27 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:53:27 -0700 Subject: [ofa-general] [PATCH 7/7][v1.2] dapl scm: fix ibv_destroy_cq busy error condition during dat_evd_free. Message-ID: Problem surfaced while running Intel MPI 3.1 and mpich2-test suite. dapli_destroy_cm was incorrectly removing reference to qp_handle in endpoint and destroy_cq and destroy_pd code was ignoring verbs errors. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_scm/dapl_ib_cm.c | 4 +--- dapl/openib_scm/dapl_ib_cq.c | 13 ++++--------- dapl/openib_scm/dapl_ib_mem.c | 8 +------- 3 files changed, 6 insertions(+), 19 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index 9e686d6..f78ebe6 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -108,10 +108,8 @@ static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) dapl_os_lock(&cm_ptr->lock); cm_ptr->state = SCM_DESTROY; - if (cm_ptr->ep) { + if (cm_ptr->ep) cm_ptr->ep->cm_handle = IB_INVALID_HANDLE; - cm_ptr->ep->qp_handle = IB_INVALID_HANDLE; - } /* close socket if still active */ if (cm_ptr->socket >= 0) { diff --git a/dapl/openib_scm/dapl_ib_cq.c b/dapl/openib_scm/dapl_ib_cq.c index 56b729e..7d6bd4f 100644 --- a/dapl/openib_scm/dapl_ib_cq.c +++ b/dapl/openib_scm/dapl_ib_cq.c @@ -423,20 +423,15 @@ DAT_RETURN dapls_ib_cq_free ( IN DAPL_EVD *evd_ptr) { DAT_EVENT event; + ib_work_completion_t wc; if (evd_ptr->ib_cq_handle != IB_INVALID_HANDLE) { /* pull off CQ and EVD entries and toss */ - while (dapls_ib_completion_poll == DAT_SUCCESS); - while (dapl_evd_dequeue(evd_ptr,&event) != DAT_QUEUE_EMPTY); -#if 1 - ibv_destroy_cq(evd_ptr->ib_cq_handle); - evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; - return DAT_SUCCESS; -#else + while (ibv_poll_cq(evd_ptr->ib_cq_handle, 1, &wc) == 1); + while (dapl_evd_dequeue(evd_ptr, &event) == DAT_SUCCESS); if (ibv_destroy_cq(evd_ptr->ib_cq_handle)) - return(dapl_convert_errno(errno,"destroy_cq")); + return(dapl_convert_errno(errno,"ibv_destroy_cq")); evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; -#endif } return DAT_SUCCESS; } diff --git a/dapl/openib_scm/dapl_ib_mem.c b/dapl/openib_scm/dapl_ib_mem.c index 6a5e4a2..0faccc8 100644 --- a/dapl/openib_scm/dapl_ib_mem.c +++ b/dapl/openib_scm/dapl_ib_mem.c @@ -136,15 +136,9 @@ DAT_RETURN dapls_ib_pd_free(IN DAPL_PZ *pz ) { if (pz->pd_handle != IB_INVALID_HANDLE) { -#if 1 - ibv_dealloc_pd(pz->pd_handle); - pz->pd_handle = IB_INVALID_HANDLE; - return DAT_SUCCESS; -#else if (ibv_dealloc_pd(pz->pd_handle)) - return(dapl_convert_errno(errno,"dealloc_pd")); + return(dapl_convert_errno(errno,"ibv_dealloc_pd")); pz->pd_handle = IB_INVALID_HANDLE; -#endif } return DAT_SUCCESS; } -- 1.5.2.5 From jsquyres at cisco.com Fri Jun 20 11:53:48 2008 From: jsquyres at cisco.com (Jeff Squyres) Date: Fri, 20 Jun 2008 14:53:48 -0400 Subject: [ofa-general] maximum IB/iWARP max_inline_data size Message-ID: If you ibv_create_qp() with max_inline_data==0: - Mellanox ConnectX reports a number that is a formula based on several values, such as max_send_sge and max_recv_sge. But its actual max inline value (I think Pasha told me) is 1k. - Chelsio T3 always reports a fixed value back -- its actual max_inline_data value (64). - Preliminary tests with NetEffect NE020 seem to show same behavior as T3 (always report back 64). Our bias in OMPI is to figure such values dynamically at run-time. With T3/NE020, it seems to be easy. Note that if you choose a max_inline_data value too large, ibv_create_qp will simply fail. So how can we figure out what the max_inline_data value is for Mellanox adapters at run-time? (I'd prefer not to repeatedly trying to create a QP with increasing values of max_inline_data until it fails). Thanks! -- Jeff Squyres Cisco Systems From arlin.r.davis at intel.com Fri Jun 20 11:54:41 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:54:41 -0700 Subject: [ofa-general] [PATCH 1/7][v2.0] dapl: endpoint pending request count is wrong Message-ID: The code assumes every cookie allocated during posting of requests gets completed. This incorrect assumption results in wrong pending count. Remove request_pending field and replace with direct call, dapl_cb_pending, to provide accurate data to consumer. Add debug print if consumer overruns request queue. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/common/dapl_cookie.c | 28 ++++++++++++++++++++++++++++ dapl/common/dapl_cookie.h | 4 ++++ dapl/common/dapl_ep_get_status.c | 4 ++-- dapl/common/dapl_ep_post_recv.c | 7 ------- dapl/common/dapl_ep_util.c | 22 ++++++++++------------ dapl/common/dapl_evd_util.c | 16 ++++------------ dapl/common/dapl_rmr_bind.c | 14 -------------- 7 files changed, 48 insertions(+), 47 deletions(-) diff --git a/dapl/common/dapl_cookie.c b/dapl/common/dapl_cookie.c index 7998d7f..1462fe9 100644 --- a/dapl/common/dapl_cookie.c +++ b/dapl/common/dapl_cookie.c @@ -260,6 +260,34 @@ dapls_cb_put ( return DAT_SUCCESS; } +/* + * dapls_cb_pending + * + * snapshot of active entries on cookie ring buffer + * + * Input: + * buffer pointer to DAPL_COOKIE_BUFFER + * + * Returns: + * DAT_COUNT number of active/pending cookies + * + */ +extern DAT_COUNT +dapls_cb_pending ( + DAPL_COOKIE_BUFFER *buffer ) +{ + DAT_COUNT head, tail; + + head = dapl_os_atomic_read(&buffer->head); + tail = dapl_os_atomic_read(&buffer->tail); + + if (head == tail) + return 0; + else if (head > tail) + return (head - tail); + else + return ((buffer->pool_size - tail) + head); +} /* * dapls_rmr_cookie_alloc diff --git a/dapl/common/dapl_cookie.h b/dapl/common/dapl_cookie.h index 20fbec6..f953b28 100644 --- a/dapl/common/dapl_cookie.h +++ b/dapl/common/dapl_cookie.h @@ -50,6 +50,10 @@ extern void dapls_cb_free ( DAPL_COOKIE_BUFFER *buffer ); +extern DAT_COUNT +dapls_cb_pending ( + DAPL_COOKIE_BUFFER *buffer ); + extern DAT_RETURN dapls_rmr_cookie_alloc ( IN DAPL_COOKIE_BUFFER *buffer, diff --git a/dapl/common/dapl_ep_get_status.c b/dapl/common/dapl_ep_get_status.c index d55b512..853afff 100644 --- a/dapl/common/dapl_ep_get_status.c +++ b/dapl/common/dapl_ep_get_status.c @@ -98,12 +98,12 @@ dapl_ep_get_status ( if ( in_dto_idle != NULL ) { - *in_dto_idle = (dapl_os_atomic_read (&ep_ptr->recv_count)) ? DAT_FALSE : DAT_TRUE; + *in_dto_idle = (dapls_cb_pending(&ep_ptr->recv_buffer)) ? DAT_FALSE : DAT_TRUE; } if ( out_dto_idle != NULL ) { - *out_dto_idle = (dapl_os_atomic_read (&ep_ptr->req_count)) ? DAT_FALSE : DAT_TRUE; + *out_dto_idle = (dapls_cb_pending(&ep_ptr->req_buffer)) ? DAT_FALSE : DAT_TRUE; } bail: diff --git a/dapl/common/dapl_ep_post_recv.c b/dapl/common/dapl_ep_post_recv.c index 2e52411..12724cd 100644 --- a/dapl/common/dapl_ep_post_recv.c +++ b/dapl/common/dapl_ep_post_recv.c @@ -109,19 +109,12 @@ dapl_ep_post_recv ( } /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->recv_count); - - /* * Invoke provider specific routine to post DTO */ dat_status = dapls_ib_post_recv (ep_ptr, cookie, num_segments, local_iov); if ( dat_status != DAT_SUCCESS ) { - dapl_os_atomic_dec (&ep_ptr->recv_count); dapls_cookie_dealloc (&ep_ptr->recv_buffer, cookie); } diff --git a/dapl/common/dapl_ep_util.c b/dapl/common/dapl_ep_util.c index e29e287..f3f88bc 100644 --- a/dapl/common/dapl_ep_util.c +++ b/dapl/common/dapl_ep_util.c @@ -161,9 +161,6 @@ dapl_ep_alloc ( ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; ep_ptr->cm_handle = IB_INVALID_HANDLE; - dapl_os_atomic_set (&ep_ptr->req_count, 0); - dapl_os_atomic_set (&ep_ptr->recv_count, 0); - if ( DAT_SUCCESS != dapls_cb_create ( &ep_ptr->req_buffer, ep_ptr, @@ -431,18 +428,20 @@ dapl_ep_post_send_req ( dto_type, user_cookie, &cookie ); - if ( dat_status != DAT_SUCCESS ) - { + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " dapl_post_req resource ERR:" + " dtos pending = %d, max_dtos %d, max_cb %d hd %d tl %d\n", + dapls_cb_pending(&ep_ptr->req_buffer), + ep_ptr->param.ep_attr.max_request_dtos, + ep_ptr->req_buffer.pool_size, + ep_ptr->req_buffer.head, + ep_ptr->req_buffer.tail); + goto bail; } /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->req_count); - - /* * Invoke provider specific routine to post DTO */ dat_status = dapls_ib_post_send ( ep_ptr, @@ -455,7 +454,6 @@ dapl_ep_post_send_req ( if ( dat_status != DAT_SUCCESS ) { - dapl_os_atomic_dec (&ep_ptr->req_count); dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie); } diff --git a/dapl/common/dapl_evd_util.c b/dapl/common/dapl_evd_util.c index 293759f..87a0194 100755 --- a/dapl/common/dapl_evd_util.c +++ b/dapl/common/dapl_evd_util.c @@ -1099,17 +1099,11 @@ dapli_evd_cqe_to_event ( } #endif - if ( DAPL_DTO_TYPE_RECV == cookie->val.dto.type ) - { - dapl_os_atomic_dec (&ep_ptr->recv_count); - buffer = &ep_ptr->recv_buffer; - } + if (DAPL_DTO_TYPE_RECV == cookie->val.dto.type) + buffer = &ep_ptr->recv_buffer; else - { - dapl_os_atomic_dec (&ep_ptr->req_count); - buffer = &ep_ptr->req_buffer; - } - + buffer = &ep_ptr->req_buffer; + event_ptr->event_number = DAT_DTO_COMPLETION_EVENT; event_ptr->event_data.dto_completion_event_data.ep_handle = cookie->ep; @@ -1136,8 +1130,6 @@ dapli_evd_cqe_to_event ( case DAPL_COOKIE_TYPE_RMR: { - dapl_os_atomic_dec (&ep_ptr->req_count); - event_ptr->event_number = DAT_RMR_BIND_COMPLETION_EVENT; event_ptr->event_data.rmr_completion_event_data.rmr_handle = diff --git a/dapl/common/dapl_rmr_bind.c b/dapl/common/dapl_rmr_bind.c index 01d06c3..12a98f6 100755 --- a/dapl/common/dapl_rmr_bind.c +++ b/dapl/common/dapl_rmr_bind.c @@ -151,12 +151,6 @@ dapli_rmr_bind_fuse ( is_signaled = (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? DAT_FALSE : DAT_TRUE; - /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->req_count); - dat_status = dapls_ib_mw_bind (rmr, lmr, ep_ptr, @@ -167,7 +161,6 @@ dapli_rmr_bind_fuse ( is_signaled); if ( DAT_SUCCESS != dat_status ) { - dapl_os_atomic_dec (&ep_ptr->req_count); dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie); goto bail; } @@ -255,19 +248,12 @@ dapli_rmr_bind_unfuse ( is_signaled = (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? DAT_FALSE : DAT_TRUE; - /* - * Take reference before posting to avoid race conditions with - * completions - */ - dapl_os_atomic_inc (&ep_ptr->req_count); - dat_status = dapls_ib_mw_unbind (rmr, ep_ptr, cookie, is_signaled); if ( DAT_SUCCESS != dat_status ) { - dapl_os_atomic_dec (&ep_ptr->req_count); dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie); goto bail1; } -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:56:09 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:56:09 -0700 Subject: [ofa-general] [PATCH 2/7][v2.0] dapl: remove unused iov buffer allocation on the endpoint Message-ID: provider's manage iov space on stack during posting. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/common/dapl_ep_modify.c | 82 ------------------------------------------ dapl/common/dapl_ep_util.c | 46 ----------------------- dapl/include/dapl.h | 5 --- 3 files changed, 0 insertions(+), 133 deletions(-) diff --git a/dapl/common/dapl_ep_modify.c b/dapl/common/dapl_ep_modify.c index e006b1d..05aa0ad 100644 --- a/dapl/common/dapl_ep_modify.c +++ b/dapl/common/dapl_ep_modify.c @@ -95,15 +95,11 @@ dapl_ep_modify ( DAT_BOOLEAN qp_allocated = DAT_FALSE; DAT_BOOLEAN rqst_cb_allocated = DAT_FALSE; DAT_BOOLEAN recv_cb_allocated = DAT_FALSE; - DAT_BOOLEAN rqst_iov_allocated = DAT_FALSE; - DAT_BOOLEAN recv_iov_allocated = DAT_FALSE; /* Flag indicating we've used (assigned to QP) a new one of these. */ DAT_BOOLEAN qp_used = DAT_FALSE; DAT_BOOLEAN rqst_cb_used = DAT_FALSE; DAT_BOOLEAN recv_cb_used = DAT_FALSE; - DAT_BOOLEAN rqst_iov_used = DAT_FALSE; - DAT_BOOLEAN recv_iov_used = DAT_FALSE; dat_status = dapli_ep_modify_validate_parameters ( ep_handle, ep_param_mask, @@ -171,39 +167,6 @@ dapl_ep_modify ( } recv_cb_allocated = DAT_TRUE; - alloc_ep.send_iov_num = ep_attr1.max_request_iov; - - /* - * Ensure that send_iov_num is big enough for all types - * of send requests. - */ - if (ep_attr1.max_rdma_read_iov > alloc_ep.send_iov_num) - { - alloc_ep.send_iov_num = ep_attr1.max_rdma_read_iov; - } - if (ep_attr1.max_rdma_write_iov > alloc_ep.send_iov_num) - { - alloc_ep.send_iov_num = ep_attr1.max_rdma_write_iov; - } - alloc_ep.send_iov = dapl_os_alloc (alloc_ep.send_iov_num - * sizeof (ib_data_segment_t)); - if ( alloc_ep.send_iov == NULL ) - { - dat_status = DAT_ERROR (DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - rqst_iov_allocated = DAT_TRUE; - - alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; - alloc_ep.recv_iov = dapl_os_alloc (ep_attr1.max_recv_iov - * sizeof (ib_data_segment_t)); - if ( alloc_ep.recv_iov == NULL ) - { - dat_status = DAT_ERROR (DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - recv_iov_allocated = DAT_TRUE; - dat_status = dapls_ib_qp_alloc ( ia, &alloc_ep, ep1 ); if ( dat_status != DAT_SUCCESS ) { @@ -281,20 +244,6 @@ dapl_ep_modify ( rqst_cb_used = DAT_TRUE; } - if ( ep_param_mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV - && new_ep.recv_iov_num != ep2->recv_iov_num ) - { - new_ep.recv_iov = alloc_ep.recv_iov; - recv_iov_used = DAT_TRUE; - } - - if ( ep_param_mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV - && new_ep.send_iov_num != ep2->send_iov_num ) - { - new_ep.send_iov = alloc_ep.send_iov; - recv_iov_used = DAT_TRUE; - } - /* * We need to change the QP only if there already was a QP * (leave things the way you found them!) and one of the @@ -439,37 +388,6 @@ bail: } } - if ( rqst_iov_allocated ) - { - if ( dat_status != DAT_SUCCESS || !rqst_iov_used ) - { - dapl_os_free ( alloc_ep.send_iov, - (alloc_ep.send_iov_num - * sizeof (ib_data_segment_t))); - } - else - { - dapl_os_free ( copy_of_old_ep.send_iov, - (copy_of_old_ep.send_iov_num - * sizeof (ib_data_segment_t))); - } - } - - if ( recv_iov_allocated ) - { - if ( dat_status != DAT_SUCCESS || !recv_iov_used ) - { - dapl_os_free ( alloc_ep.recv_iov, - (alloc_ep.recv_iov_num - * sizeof (ib_data_segment_t))); - } - else - { - dapl_os_free ( copy_of_old_ep.recv_iov, - (copy_of_old_ep.recv_iov_num - * sizeof (ib_data_segment_t))); - } - } return dat_status; } diff --git a/dapl/common/dapl_ep_util.c b/dapl/common/dapl_ep_util.c index f3f88bc..782e09a 100644 --- a/dapl/common/dapl_ep_util.c +++ b/dapl/common/dapl_ep_util.c @@ -181,42 +181,6 @@ dapl_ep_alloc ( goto bail; } - ep_ptr->recv_iov_num = ep_ptr->param.ep_attr.max_recv_iov; - ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_request_iov; - - /* - * Ensure that send_iov_num is big enough for all types - * of send requests. - */ - if (ep_ptr->param.ep_attr.max_rdma_read_iov > ep_ptr->send_iov_num) - { - ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_rdma_read_iov; - } - if (ep_ptr->param.ep_attr.max_rdma_write_iov > ep_ptr->send_iov_num) - { - ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_rdma_write_iov; - } - - ep_ptr->recv_iov = dapl_os_alloc ( - ep_ptr->recv_iov_num * sizeof (ib_data_segment_t)); - - if ( NULL == ep_ptr->recv_iov ) - { - dapl_ep_dealloc (ep_ptr); - ep_ptr = NULL; - goto bail; - } - - ep_ptr->send_iov = dapl_os_alloc ( - ep_ptr->send_iov_num * sizeof (ib_data_segment_t)); - - if ( NULL == ep_ptr->send_iov ) - { - dapl_ep_dealloc (ep_ptr); - ep_ptr = NULL; - goto bail; - } - dapls_io_trc_alloc (ep_ptr); bail: @@ -250,16 +214,6 @@ dapl_ep_dealloc ( dapls_cb_free ( &ep_ptr->req_buffer ); dapls_cb_free ( &ep_ptr->recv_buffer ); - if ( NULL != ep_ptr->recv_iov ) - { - dapl_os_free (ep_ptr->recv_iov, ep_ptr->recv_iov_num * sizeof (ib_data_segment_t)); - } - - if ( NULL != ep_ptr->send_iov ) - { - dapl_os_free (ep_ptr->send_iov, ep_ptr->send_iov_num * sizeof (ib_data_segment_t)); - } - if ( NULL != ep_ptr->cxn_timer ) { dapl_os_free ( ep_ptr->cxn_timer, sizeof ( DAPL_OS_TIMER ) ); diff --git a/dapl/include/dapl.h b/dapl/include/dapl.h index 42db81e..f0f2095 100755 --- a/dapl/include/dapl.h +++ b/dapl/include/dapl.h @@ -465,11 +465,6 @@ struct dapl_ep DAPL_COOKIE_BUFFER req_buffer; DAPL_COOKIE_BUFFER recv_buffer; - ib_data_segment_t *recv_iov; - DAT_COUNT recv_iov_num; - - ib_data_segment_t *send_iov; - DAT_COUNT send_iov_num; #ifdef DAPL_DBG_IO_TRC int ibt_dumped; struct io_buf_track *ibt_base; -- 1.5.2.5 From hrosenstock at xsigo.com Fri Jun 20 11:57:39 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 11:57:39 -0700 Subject: [ofa-general] [PATCH] opensm/osm_sa_mcmember_record.c: Fix some off by 1 issues related to max_multicast_lid_ho Message-ID: <1213988259.13056.203.camel@hrosenstock-ws.xsigo.com> opensm/osm_sa_mcmember_record.c: Fix some off by 1 issues related to max_multicast_lid_ho Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index c0405f6..46c87c7 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -153,7 +153,7 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) OSM_LOG_ENTER(sa->p_log); if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO - && cl_ntoh16(requested_mlid) < p_subn->max_multicast_lid_ho + && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho && cl_qmap_get(&p_subn->mgrp_mlid_tbl, requested_mlid) == cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { @@ -172,7 +172,7 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) } max_num_mlids = - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; /* track all used mlids in the array (by mlid index) */ used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); From arlin.r.davis at intel.com Fri Jun 20 11:57:20 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:57:20 -0700 Subject: [ofa-general] [PATCH 3/7][v2.0] dapl: add opcode to string function to report opcode during failures. Message-ID: Need to use cookie opcode during failures in lieu of cqe opcode. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/common/dapl_evd_util.c | 24 +++++++++++++++++++++--- 1 files changed, 21 insertions(+), 3 deletions(-) mode change 100755 => 100644 dapl/common/dapl_evd_util.c diff --git a/dapl/common/dapl_evd_util.c b/dapl/common/dapl_evd_util.c old mode 100755 new mode 100644 index 87a0194..ea3e27c --- a/dapl/common/dapl_evd_util.c +++ b/dapl/common/dapl_evd_util.c @@ -45,6 +45,10 @@ #include "dapl_cookie.h" #include "dapl.h" +#include +#include +#include + STATIC _INLINE_ void dapli_evd_eh_print_cqe ( IN ib_work_completion_t *cqe); @@ -543,6 +547,19 @@ bail: return dat_status; } +STATIC _INLINE_ char * DAPL_GET_DTO_OP_STR(int op) +{ + static char *dto_ops[] = + { + "OP_SEND", + "OP_RECEIVE", + "OP_RDMA_WRITE", + "OP_RDMA_READ", + 0 + }; + return ((op < 0 || op > 3) ? "Invalid DTO OP?" : dto_ops[op]); +} + #if !defined(DAPL_GET_CQE_OP_STR) #define DAPL_GET_CQE_OP_STR(e) "Unknown CEQ OP String?" #endif @@ -1210,10 +1227,11 @@ dapli_evd_cqe_to_event ( } dapl_log(DAPL_DBG_TYPE_ERR, - "DTO completion ERR: status %d, opcode %s, vendor_err 0x%x\n", + "DTO completion ERR: status %d, op %s, vendor_err 0x%x - %s\n", DAPL_GET_CQE_STATUS(cqe_ptr), - DAPL_GET_CQE_OP_STR(cqe_ptr), - DAPL_GET_CQE_VENDOR_ERR(cqe_ptr)); + DAPL_GET_DTO_OP_STR(cookie->val.dto.type), + DAPL_GET_CQE_VENDOR_ERR(cqe_ptr), + inet_ntoa(((struct sockaddr_in *)&ep_ptr->remote_ia_address)->sin_addr)); } } -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 11:58:11 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:58:11 -0700 Subject: [ofa-general] [PATCH 4/7][v2.0] dapl scm: Adding socket cm provider for better scalability on large homogeneous clusters. Message-ID: Bring socket cm provider back to life with some changes: better threading support for exchanging QP information. Avoid blocking during connect to support dynamic connection model with MPI implementations. consumer control of ack timeout/retries. disconnect/reject capabilities via socket exchange. version support for wire protocol to insure compatibility with peer scm provider. Add gids to exchange. validated with Intel MPI on a 14,000+ core fabric using IB DDR. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- Makefile.am | 152 ++++++- dapl.spec.in | 10 +- dapl/openib_scm/dapl_ib_cm.c | 858 +++++++++++++++++++++------------- dapl/openib_scm/dapl_ib_cq.c | 19 +- dapl/openib_scm/dapl_ib_dto.h | 397 ++++++++++++---- dapl/openib_scm/dapl_ib_extensions.c | 313 +++++++++++++ dapl/openib_scm/dapl_ib_mem.c | 157 ++++--- dapl/openib_scm/dapl_ib_qp.c | 53 ++- dapl/openib_scm/dapl_ib_util.c | 196 +++++--- dapl/openib_scm/dapl_ib_util.h | 46 ++- dapl/udapl/libdaploscm.map | 7 + 11 files changed, 1619 insertions(+), 589 deletions(-) create mode 100755 dapl/openib_scm/dapl_ib_extensions.c create mode 100755 dapl/udapl/libdaploscm.map diff --git a/Makefile.am b/Makefile.am index bb75dea..db3ab57 100755 --- a/Makefile.am +++ b/Makefile.am @@ -13,10 +13,12 @@ endif if EXT_TYPE_IB XFLAGS = -DDAT_EXTENSIONS -XPROGRAMS = dapl/openib_cma/dapl_ib_extensions.c +XPROGRAMS_CMA = dapl/openib_cma/dapl_ib_extensions.c +XPROGRAMS_SCM = dapl/openib_scm/dapl_ib_extensions.c else XFLAGS = -XPROGRAMS = +XPROGRAMS_CMA = +XPROGRAMS_SCM = endif if DEBUG @@ -27,9 +29,11 @@ endif datlibdir = $(libdir) dapllibofadir = $(libdir) +daplliboscmdir = $(libdir) datlib_LTLIBRARIES = dat/udat/libdat2.la dapllibofa_LTLIBRARIES = dapl/udapl/libdaplofa.la +daplliboscm_LTLIBRARIES = dapl/udapl/libdaploscm.la dat_udat_libdat2_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) $(XFLAGS) \ -I$(srcdir)/dat/include/ -I$(srcdir)/dat/udat/ \ @@ -41,14 +45,20 @@ dapl_udapl_libdaplofa_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) $(X -I$(srcdir)/dapl/common -I$(srcdir)/dapl/udapl/linux \ -I$(srcdir)/dapl/openib_cma +dapl_udapl_libdaploscm_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) $(XFLAGS) \ + -DOPENIB -DCQ_WAIT_OBJECT \ + -I$(srcdir)/dat/include/ -I$(srcdir)/dapl/include/ \ + -I$(srcdir)/dapl/common -I$(srcdir)/dapl/udapl/linux \ + -I$(srcdir)/dapl/openib_scm + if HAVE_LD_VERSION_SCRIPT dat_version_script = -Wl,--version-script=$(srcdir)/dat/udat/libdat2.map daplofa_version_script = -Wl,--version-script=$(srcdir)/dapl/udapl/libdaplofa.map - + daploscm_version_script = -Wl,--version-script=$(srcdir)/dapl/udapl/libdaploscm.map else dat_version_script = daplofa_version_script = - + daploscm_version_script = endif # @@ -173,12 +183,125 @@ dapl_udapl_libdaplofa_la_SOURCES = dapl/udapl/dapl_init.c \ dapl/openib_cma/dapl_ib_cq.c \ dapl/openib_cma/dapl_ib_qp.c \ dapl/openib_cma/dapl_ib_cm.c \ - dapl/openib_cma/dapl_ib_mem.c $(XPROGRAMS) + dapl/openib_cma/dapl_ib_mem.c $(XPROGRAMS_CMA) dapl_udapl_libdaplofa_la_LDFLAGS = -version-info 2:0:0 $(daplofa_version_script) \ -Wl,-init,dapl_init -Wl,-fini,dapl_fini \ -lpthread -libverbs -lrdmacm +# +# uDAPL OpenFabrics Socket CM version: libdaplscm.so +# +dapl_udapl_libdaploscm_la_SOURCES = dapl/udapl/dapl_init.c \ + dapl/udapl/dapl_evd_create.c \ + dapl/udapl/dapl_evd_query.c \ + dapl/udapl/dapl_cno_create.c \ + dapl/udapl/dapl_cno_modify_agent.c \ + dapl/udapl/dapl_cno_free.c \ + dapl/udapl/dapl_cno_wait.c \ + dapl/udapl/dapl_cno_query.c \ + dapl/udapl/dapl_lmr_create.c \ + dapl/udapl/dapl_evd_wait.c \ + dapl/udapl/dapl_evd_disable.c \ + dapl/udapl/dapl_evd_enable.c \ + dapl/udapl/dapl_evd_modify_cno.c \ + dapl/udapl/dapl_evd_set_unwaitable.c \ + dapl/udapl/dapl_evd_clear_unwaitable.c \ + dapl/udapl/linux/dapl_osd.c \ + dapl/common/dapl_cookie.c \ + dapl/common/dapl_cr_accept.c \ + dapl/common/dapl_cr_query.c \ + dapl/common/dapl_cr_reject.c \ + dapl/common/dapl_cr_util.c \ + dapl/common/dapl_cr_callback.c \ + dapl/common/dapl_cr_handoff.c \ + dapl/common/dapl_ep_connect.c \ + dapl/common/dapl_ep_create.c \ + dapl/common/dapl_ep_disconnect.c \ + dapl/common/dapl_ep_dup_connect.c \ + dapl/common/dapl_ep_free.c \ + dapl/common/dapl_ep_reset.c \ + dapl/common/dapl_ep_get_status.c \ + dapl/common/dapl_ep_modify.c \ + dapl/common/dapl_ep_post_rdma_read.c \ + dapl/common/dapl_ep_post_rdma_write.c \ + dapl/common/dapl_ep_post_recv.c \ + dapl/common/dapl_ep_post_send.c \ + dapl/common/dapl_ep_query.c \ + dapl/common/dapl_ep_util.c \ + dapl/common/dapl_evd_dequeue.c \ + dapl/common/dapl_evd_free.c \ + dapl/common/dapl_evd_post_se.c \ + dapl/common/dapl_evd_resize.c \ + dapl/common/dapl_evd_util.c \ + dapl/common/dapl_evd_cq_async_error_callb.c \ + dapl/common/dapl_evd_qp_async_error_callb.c \ + dapl/common/dapl_evd_un_async_error_callb.c \ + dapl/common/dapl_evd_connection_callb.c \ + dapl/common/dapl_evd_dto_callb.c \ + dapl/common/dapl_get_consumer_context.c \ + dapl/common/dapl_get_handle_type.c \ + dapl/common/dapl_hash.c \ + dapl/common/dapl_hca_util.c \ + dapl/common/dapl_ia_close.c \ + dapl/common/dapl_ia_open.c \ + dapl/common/dapl_ia_query.c \ + dapl/common/dapl_ia_util.c \ + dapl/common/dapl_llist.c \ + dapl/common/dapl_lmr_free.c \ + dapl/common/dapl_lmr_query.c \ + dapl/common/dapl_lmr_util.c \ + dapl/common/dapl_lmr_sync_rdma_read.c \ + dapl/common/dapl_lmr_sync_rdma_write.c \ + dapl/common/dapl_mr_util.c \ + dapl/common/dapl_provider.c \ + dapl/common/dapl_sp_util.c \ + dapl/common/dapl_psp_create.c \ + dapl/common/dapl_psp_create_any.c \ + dapl/common/dapl_psp_free.c \ + dapl/common/dapl_psp_query.c \ + dapl/common/dapl_pz_create.c \ + dapl/common/dapl_pz_free.c \ + dapl/common/dapl_pz_query.c \ + dapl/common/dapl_pz_util.c \ + dapl/common/dapl_rmr_create.c \ + dapl/common/dapl_rmr_free.c \ + dapl/common/dapl_rmr_bind.c \ + dapl/common/dapl_rmr_query.c \ + dapl/common/dapl_rmr_util.c \ + dapl/common/dapl_rsp_create.c \ + dapl/common/dapl_rsp_free.c \ + dapl/common/dapl_rsp_query.c \ + dapl/common/dapl_cno_util.c \ + dapl/common/dapl_set_consumer_context.c \ + dapl/common/dapl_ring_buffer_util.c \ + dapl/common/dapl_name_service.c \ + dapl/common/dapl_timer_util.c \ + dapl/common/dapl_ep_create_with_srq.c \ + dapl/common/dapl_ep_recv_query.c \ + dapl/common/dapl_ep_set_watermark.c \ + dapl/common/dapl_srq_create.c \ + dapl/common/dapl_srq_free.c \ + dapl/common/dapl_srq_query.c \ + dapl/common/dapl_srq_resize.c \ + dapl/common/dapl_srq_post_recv.c \ + dapl/common/dapl_srq_set_lw.c \ + dapl/common/dapl_srq_util.c \ + dapl/common/dapl_debug.c \ + dapl/common/dapl_ia_ha.c \ + dapl/common/dapl_csp.c \ + dapl/common/dapl_ep_post_send_invalidate.c \ + dapl/common/dapl_ep_post_rdma_read_to_rmr.c \ + dapl/openib_scm/dapl_ib_util.c \ + dapl/openib_scm/dapl_ib_cq.c \ + dapl/openib_scm/dapl_ib_qp.c \ + dapl/openib_scm/dapl_ib_cm.c \ + dapl/openib_scm/dapl_ib_mem.c $(XPROGRAMS_SCM) + +dapl_udapl_libdaploscm_la_LDFLAGS = -version-info 2:0:0 $(daploscm_version_script) \ + -Wl,-init,dapl_init -Wl,-fini,dapl_fini \ + -lpthread -libverbs + libdatincludedir = $(includedir)/dat2 libdatinclude_HEADERS = dat/include/dat2/dat.h \ @@ -238,8 +361,12 @@ EXTRA_DIST = dat/common/dat_dictionary.h \ dapl/include/dapl_vendor.h \ dapl/openib_cma/dapl_ib_dto.h \ dapl/openib_cma/dapl_ib_util.h \ + dapl/openib_scm/dapl_ib_dto.h \ + dapl/openib_scm/dapl_ib_util.h \ dat/udat/libdat2.map \ + doc/dat.conf \ dapl/udapl/libdaplofa.map \ + dapl/udapl/libdaploscm.map \ dapl.spec.in \ $(man_MANS) \ test/dapltest/include/dapl_bpool.h \ @@ -272,7 +399,7 @@ EXTRA_DIST = dat/common/dat_dictionary.h \ dist-hook: dapl.spec cp dapl.spec $(distdir) - + install-exec-hook: if test -e $(sysconfdir)/dat.conf; then \ sed -e '/ofa-v2-.* u2/d' < $(sysconfdir)/dat.conf > /tmp/$$$$ofadapl; \ @@ -280,15 +407,18 @@ install-exec-hook: fi; \ echo ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib0 0" ""' >> $(sysconfdir)/dat.conf; \ echo ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib1 0" ""' >> $(sysconfdir)/dat.conf; \ - echo ofa-v2-ib2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib2 0" ""' >> $(sysconfdir)/dat.conf; \ - echo ofa-v2-ib3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib3 0" ""' >> $(sysconfdir)/dat.conf; \ - echo ofa-v2-bond u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"bond0 0" ""' >> $(sysconfdir)/dat.conf; + echo ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mthca0 1" ""' >> $(sysconfdir)/dat.conf; \ + echo ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mthca0 2" ""' >> $(sysconfdir)/dat.conf; \ + echo ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mlx4_0 1" ""' >> $(sysconfdir)/dat.conf; \ + echo ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mlx4_0 2" ""' >> $(sysconfdir)/dat.conf; + echo ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"ipath0 1" ""' >> $(sysconfdir)/dat.conf; \ + echo ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"ipath0 2" ""' >> $(sysconfdir)/dat.conf; + echo ofa-v2-ehca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"ehca0 1" ""' >> $(sysconfdir)/dat.conf; uninstall-hook: if test -e $(sysconfdir)/dat.conf; then \ sed -e '/ofa-v2-.* u2/d' < $(sysconfdir)/dat.conf > /tmp/$$$$ofadapl; \ cp /tmp/$$$$ofadapl $(sysconfdir)/dat.conf; \ - fi; + fi; SUBDIRS = . test/dtest test/dapltest - diff --git a/dapl.spec.in b/dapl.spec.in index de28e48..b73967a 100644 --- a/dapl.spec.in +++ b/dapl.spec.in @@ -95,9 +95,13 @@ if [ -e %{_sysconfdir}/dat.conf ]; then fi echo ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib0 0" ""' >> %{_sysconfdir}/dat.conf echo ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib1 0" ""' >> %{_sysconfdir}/dat.conf -echo ofa-v2-ib2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib2 0" ""' >> %{_sysconfdir}/dat.conf -echo ofa-v2-ib3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"ib3 0" ""' >> %{_sysconfdir}/dat.conf -echo ofa-v2-bond u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 '"bond0 0" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mthca0 1" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mthca0 2" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mlx4_0 1" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"mlx4_0 2" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"ipath0 1" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"ipath0 2" ""' >> %{_sysconfdir}/dat.conf +echo ofa-v2-ehca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 '"ehca0 1" ""' >> %{_sysconfdir}/dat.conf %postun /sbin/ldconfig diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index 94e3959..9b30625 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -57,76 +57,171 @@ #include #include #include -#include +#include +#include -/* prototypes */ -static uint16_t dapli_get_lid( struct ibv_device *dev, int port ); +#include +#include +#include -static DAT_RETURN dapli_socket_connect ( DAPL_EP *ep_ptr, - DAT_IA_ADDRESS_PTR r_addr, - DAT_CONN_QUAL r_qual, - DAT_COUNT p_size, - DAT_PVOID p_data ); +#if __BYTE_ORDER == __LITTLE_ENDIAN +static inline uint64_t cpu_to_be64(uint64_t x) {return bswap_64(x);} +#elif __BYTE_ORDER == __BIG_ENDIAN +static inline uint64_t cpu_to_be64(uint64_t x) {return x;} +#endif -static DAT_RETURN dapli_socket_listen ( DAPL_IA *ia_ptr, - DAT_CONN_QUAL serviceID, - DAPL_SP *sp_ptr ); +extern int g_scm_pipe[2]; -static DAT_RETURN dapli_socket_accept( ib_cm_srvc_handle_t cm_ptr ); +static struct ib_cm_handle *dapli_cm_create(void) +{ + struct ib_cm_handle *cm_ptr; -static DAT_RETURN dapli_socket_accept_final( DAPL_EP *ep_ptr, - DAPL_CR *cr_ptr, - DAT_COUNT p_size, - DAT_PVOID p_data ); + /* Allocate CM, init lock, and initialize */ + if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) + return NULL; + + if (dapl_os_lock_init(&cm_ptr->lock)) + goto bail; -/* XXX temporary hack to get lid */ -static uint16_t dapli_get_lid(IN struct ibv_device *dev, IN int port) + (void)dapl_os_memzero(cm_ptr, sizeof(*cm_ptr)); + cm_ptr->dst.ver = htons(DSCM_VER); + cm_ptr->socket = -1; + return cm_ptr; +bail: + dapl_os_free(cm_ptr, sizeof(*cm_ptr)); + return NULL; +} + +/* mark for destroy, remove all references, schedule cleanup */ +static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) { - char path[128]; - char val[16]; - char name[256]; + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " cm_destroy: cm %p ep %p\n", cm_ptr,cm_ptr->ep); + + /* cleanup, never made it to work queue */ + if (cm_ptr->state == SCM_INIT) { + if (cm_ptr->socket >= 0) + close(cm_ptr->socket); + dapl_os_free(cm_ptr, sizeof(*cm_ptr)); + return; + } - if (sysfs_get_mnt_path(path, sizeof path)) { - fprintf(stderr, "Couldn't find sysfs mount.\n"); - return 0; + dapl_os_lock(&cm_ptr->lock); + cm_ptr->state = SCM_DESTROY; + if (cm_ptr->ep) { + cm_ptr->ep->cm_handle = IB_INVALID_HANDLE; + cm_ptr->ep->qp_handle = IB_INVALID_HANDLE; } - sprintf(name, "%s/class/infiniband/%s/ports/%d/lid", path, - ibv_get_device_name(dev), port); - if (sysfs_read_attribute_value(name, val, sizeof val)) { - fprintf(stderr, "Couldn't read LID at %s\n", name); - return 0; + /* close socket if still active */ + if (cm_ptr->socket >= 0) { + close(cm_ptr->socket); + cm_ptr->socket = -1; } - return strtol(val, NULL, 0); + dapl_os_unlock(&cm_ptr->lock); + + /* wakeup work thread */ + write(g_scm_pipe[1], "w", sizeof "w"); +} + +/* queue socket for processing CM work */ +static void dapli_cm_queue(struct ib_cm_handle *cm_ptr) +{ + /* add to work queue for cr thread processing */ + dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry); + dapl_os_lock(&cm_ptr->hca->ib_trans.lock); + dapl_llist_add_tail(&cm_ptr->hca->ib_trans.list, + (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr); + dapl_os_unlock(&cm_ptr->hca->ib_trans.lock); + + /* wakeup CM work thread */ + write(g_scm_pipe[1], "w", sizeof "w"); +} + +static uint16_t dapli_get_lid(IN struct ibv_context *ctx, IN uint8_t port) +{ + struct ibv_port_attr port_attr; + + if(ibv_query_port(ctx, port,&port_attr)) + return(0xffff); + else + return(port_attr.lid); } /* - * ACTIVE: Create socket, connect, and exchange QP information + * ACTIVE/PASSIVE: called from CR thread or consumer via ep_disconnect */ static DAT_RETURN -dapli_socket_connect ( DAPL_EP *ep_ptr, - DAT_IA_ADDRESS_PTR r_addr, - DAT_CONN_QUAL r_qual, - DAT_COUNT p_size, - DAT_PVOID p_data ) +dapli_socket_disconnect(dp_ib_cm_handle_t cm_ptr) { - ib_cm_handle_t cm_ptr; - DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; + DAPL_EP *ep_ptr = cm_ptr->ep; + DAT_UINT32 disc_data = htonl(0xdead); + + if (ep_ptr == NULL) + return DAT_SUCCESS; + + dapl_os_lock(&cm_ptr->lock); + if ((cm_ptr->state == SCM_INIT) || + (cm_ptr->state == SCM_DISCONNECTED)) { + dapl_os_unlock(&cm_ptr->lock); + return DAT_SUCCESS; + } else { + /* send disc date, close socket, schedule destroy */ + if (cm_ptr->socket >= 0) { + write(cm_ptr->socket, &disc_data, sizeof(disc_data)); + close(cm_ptr->socket); + cm_ptr->socket = -1; + } + cm_ptr->state = SCM_DISCONNECTED; + write(g_scm_pipe[1], "w", sizeof "w"); + } + dapl_os_unlock(&cm_ptr->lock); + + + if (ep_ptr->cr_ptr) { + dapls_cr_callback(cm_ptr, + IB_CME_DISCONNECTED, + NULL, + ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr); + } else { + dapl_evd_connection_callback(ep_ptr->cm_handle, + IB_CME_DISCONNECTED, + NULL, + ep_ptr); + } + + /* remove reference from endpoint */ + ep_ptr->cm_handle = NULL; + + /* schedule destroy */ + + + return DAT_SUCCESS; +} + + +/* + * ACTIVE: Create socket, connect, defer exchange QP information to CR thread + * to avoid blocking. + */ +DAT_RETURN +dapli_socket_connect(DAPL_EP *ep_ptr, + DAT_IA_ADDRESS_PTR r_addr, + DAT_CONN_QUAL r_qual, + DAT_COUNT p_size, + DAT_PVOID p_data) +{ + dp_ib_cm_handle_t cm_ptr; int len, opt = 1; struct iovec iovec[2]; - short rtu_data = htons(0x0E0F); - - dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d\n", r_qual); + DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; + + dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d p_size=%d\n", + r_qual,p_size); - /* - * Allocate CM and initialize - */ - if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL ) { + cm_ptr = dapli_cm_create(); + if (cm_ptr == NULL) return DAT_INSUFFICIENT_RESOURCES; - } - - (void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) ); - cm_ptr->socket = -1; /* create, connect, sockopt, and exchange QP information */ if ((cm_ptr->socket = socket(AF_INET,SOCK_STREAM,0)) < 0 ) { @@ -136,204 +231,261 @@ dapli_socket_connect ( DAPL_EP *ep_ptr, ((struct sockaddr_in*)r_addr)->sin_port = htons(r_qual); - if ( connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0 ) { + if (connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " connect: %s on r_qual %d\n", strerror(errno), (unsigned int)r_qual); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); + dapli_cm_destroy(cm_ptr); return DAT_INVALID_ADDRESS; } setsockopt(cm_ptr->socket,IPPROTO_TCP,TCP_NODELAY,&opt,sizeof(opt)); - + + dapl_dbg_log(DAPL_DBG_TYPE_EP, " socket connected!\n"); + + /* Send QP info, IA address, and private data */ - cm_ptr->dst.qpn = ep_ptr->qp_handle->qp_num; - cm_ptr->dst.port = ia_ptr->hca_ptr->port_num; - cm_ptr->dst.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, - ia_ptr->hca_ptr->port_num ); + cm_ptr->dst.qpn = htonl(ep_ptr->qp_handle->qp_num); + cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); + cm_ptr->dst.lid = + htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num)); + if (cm_ptr->dst.lid == 0xffff) + goto bail; + + /* in network order */ + if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num, + 0, + &cm_ptr->dst.gid)) + goto bail; + cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; - cm_ptr->dst.p_size = p_size; + cm_ptr->dst.p_size = htonl(p_size); iovec[0].iov_base = &cm_ptr->dst; iovec[0].iov_len = sizeof(ib_qp_cm_t); - if ( p_size ) { + if (p_size) { iovec[1].iov_base = p_data; iovec[1].iov_len = p_size; } - len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) ); - if ( len != (p_size + sizeof(ib_qp_cm_t)) ) { + + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket connected, write QP and private data\n"); + len = writev(cm_ptr->socket, iovec, (p_size ? 2:1)); + if (len != (p_size + sizeof(ib_qp_cm_t))) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " connect write: ERR %s, wcnt=%d\n", strerror(errno), len); goto bail; } - dapl_dbg_log(DAPL_DBG_TYPE_EP, + dapl_dbg_log(DAPL_DBG_TYPE_CM, " connect: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", - cm_ptr->dst.port, cm_ptr->dst.lid, - cm_ptr->dst.qpn, cm_ptr->dst.p_size ); + ntohs(cm_ptr->dst.port), ntohs(cm_ptr->dst.lid), + ntohl(cm_ptr->dst.qpn), ntohl(cm_ptr->dst.p_size)); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " connect SRC GID subnet %016llx id %016llx\n", + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.subnet_prefix), + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.interface_id)); + + /* queue up to work thread to avoid blocking consumer */ + cm_ptr->state = SCM_CONN_PENDING; + cm_ptr->hca = ia_ptr->hca_ptr; + cm_ptr->ep = ep_ptr; + dapli_cm_queue(cm_ptr); + return DAT_SUCCESS; +bail: + /* close socket, free cm structure */ + dapli_cm_destroy(cm_ptr); + return DAT_INTERNAL_ERROR; +} + + +/* + * ACTIVE: exchange QP information, called from CR thread + */ +void +dapli_socket_connect_rtu(dp_ib_cm_handle_t cm_ptr) +{ + DAPL_EP *ep_ptr = cm_ptr->ep; + int len; + struct iovec iovec[2]; + short rtu_data = htons(0x0E0F); + ib_cm_events_t event = IB_CME_DESTINATION_REJECT; /* read DST information into cm_ptr, overwrite SRC info */ - len = readv( cm_ptr->socket, iovec, 1 ); - if ( len != sizeof(ib_qp_cm_t) ) { + dapl_dbg_log(DAPL_DBG_TYPE_EP," connect_rtu: recv peer QP data\n"); + + iovec[0].iov_base = &cm_ptr->dst; + iovec[0].iov_len = sizeof(ib_qp_cm_t); + len = readv(cm_ptr->socket, iovec, 1); + if (len != sizeof(ib_qp_cm_t) || ntohs(cm_ptr->dst.ver) != DSCM_VER) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " connect read: ERR %s, rcnt=%d\n", - strerror(errno), len); + " connect_rtu read: ERR %s, rcnt=%d, ver=%d\n", + strerror(errno), len, cm_ptr->dst.ver); + goto bail; + } + /* check for consumer reject */ + if (cm_ptr->dst.rej) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " connect_rtu read: PEER REJ reason=0x%x\n", + ntohs(cm_ptr->dst.rej)); + event = IB_CME_DESTINATION_REJECT_PRIVATE_DATA; goto bail; } + + /* convert peer response values to host order */ + cm_ptr->dst.port = ntohs(cm_ptr->dst.port); + cm_ptr->dst.lid = ntohs(cm_ptr->dst.lid); + cm_ptr->dst.qpn = ntohl(cm_ptr->dst.qpn); + cm_ptr->dst.p_size = ntohl(cm_ptr->dst.p_size); + + /* save remote address information */ + dapl_os_memcpy( &ep_ptr->remote_ia_address, + &cm_ptr->dst.ia_address, + sizeof(ep_ptr->remote_ia_address)); + dapl_dbg_log(DAPL_DBG_TYPE_EP, - " connect: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", + " connect_rtu: DST %s port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", + inet_ntoa(((struct sockaddr_in *)&cm_ptr->dst.ia_address)->sin_addr), cm_ptr->dst.port, cm_ptr->dst.lid, - cm_ptr->dst.qpn, cm_ptr->dst.p_size ); + cm_ptr->dst.qpn, cm_ptr->dst.p_size); /* validate private data size before reading */ - if ( cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE ) { + if (cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " connect read: psize (%d) wrong\n", + " connect_rtu read: psize (%d) wrong\n", cm_ptr->dst.p_size ); goto bail; } /* read private data into cm_handle if any present */ - if ( cm_ptr->dst.p_size ) { + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket connected, read private data\n"); + if (cm_ptr->dst.p_size) { iovec[0].iov_base = cm_ptr->p_data; iovec[0].iov_len = cm_ptr->dst.p_size; - len = readv( cm_ptr->socket, iovec, 1 ); - if ( len != cm_ptr->dst.p_size ) { + len = readv(cm_ptr->socket, iovec, 1); + if (len != cm_ptr->dst.p_size) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " connect read pdata: ERR %s, rcnt=%d\n", + " connect_rtu read pdata: ERR %s, rcnt=%d\n", strerror(errno), len); goto bail; } } /* modify QP to RTR and then to RTS with remote info */ - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTR, &cm_ptr->dst) != DAT_SUCCESS) goto bail; - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTS, &cm_ptr->dst) != DAT_SUCCESS) goto bail; ep_ptr->qp_state = IB_QP_STATE_RTS; + dapl_dbg_log(DAPL_DBG_TYPE_EP," connect_rtu: send RTU\n"); + /* complete handshake after final QP state change */ - write(cm_ptr->socket, &rtu_data, sizeof(rtu_data) ); + write(cm_ptr->socket, &rtu_data, sizeof(rtu_data)); /* init cm_handle and post the event with private data */ ep_ptr->cm_handle = cm_ptr; - dapl_dbg_log( DAPL_DBG_TYPE_EP," ACTIVE: connected!\n" ); - dapl_evd_connection_callback( ep_ptr->cm_handle, - IB_CME_CONNECTED, - cm_ptr->p_data, - ep_ptr ); - return DAT_SUCCESS; - + cm_ptr->state = SCM_CONNECTED; + dapl_dbg_log(DAPL_DBG_TYPE_EP," ACTIVE: connected!\n"); + dapl_evd_connection_callback(cm_ptr, + IB_CME_CONNECTED, + cm_ptr->p_data, + ep_ptr); + return; bail: /* close socket, free cm structure and post error event */ - if ( cm_ptr->socket >= 0 ) - close(cm_ptr->socket); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); - dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */ - - dapl_evd_connection_callback( ep_ptr->cm_handle, - IB_CME_LOCAL_FAILURE, - NULL, - ep_ptr ); - return DAT_INTERNAL_ERROR; + dapli_cm_destroy(cm_ptr); + dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ + dapl_evd_connection_callback(NULL, event, NULL, ep_ptr); } - /* * PASSIVE: Create socket, listen, accept, exchange QP information */ -static DAT_RETURN -dapli_socket_listen ( DAPL_IA *ia_ptr, - DAT_CONN_QUAL serviceID, - DAPL_SP *sp_ptr ) +DAT_RETURN +dapli_socket_listen(DAPL_IA *ia_ptr, + DAT_CONN_QUAL serviceID, + DAPL_SP *sp_ptr ) { struct sockaddr_in addr; ib_cm_srvc_handle_t cm_ptr = NULL; int opt = 1; DAT_RETURN dat_status = DAT_SUCCESS; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " listen(ia_ptr %p ServiceID %d sp_ptr %p)\n", - ia_ptr, serviceID, sp_ptr); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " listen(ia_ptr %p ServiceID %d sp_ptr %p)\n", + ia_ptr, serviceID, sp_ptr); - /* Allocate CM and initialize */ - if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) + cm_ptr = dapli_cm_create(); + if (cm_ptr == NULL) return DAT_INSUFFICIENT_RESOURCES; - (void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) ); - - cm_ptr->socket = cm_ptr->l_socket = -1; cm_ptr->sp = sp_ptr; - cm_ptr->hca_ptr = ia_ptr->hca_ptr; + cm_ptr->hca = ia_ptr->hca_ptr; /* bind, listen, set sockopt, accept, exchange data */ - if ((cm_ptr->l_socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) { + if ((cm_ptr->socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) { dapl_dbg_log (DAPL_DBG_TYPE_ERR, "socket for listen returned %d\n", errno); dat_status = DAT_INSUFFICIENT_RESOURCES; goto bail; } - setsockopt(cm_ptr->l_socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt)); + setsockopt(cm_ptr->socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt)); addr.sin_port = htons(serviceID); addr.sin_family = AF_INET; addr.sin_addr.s_addr = INADDR_ANY; - if (( bind( cm_ptr->l_socket,(struct sockaddr*)&addr, sizeof(addr) ) < 0) || - (listen( cm_ptr->l_socket, 128 ) < 0) ) { - + if ((bind(cm_ptr->socket,(struct sockaddr*)&addr, sizeof(addr)) < 0) || + (listen(cm_ptr->socket, 128) < 0)) { dapl_dbg_log( DAPL_DBG_TYPE_CM, " listen: ERROR %s on conn_qual 0x%x\n", strerror(errno),serviceID); - - if ( errno == EADDRINUSE ) + if (errno == EADDRINUSE) dat_status = DAT_CONN_QUAL_IN_USE; else dat_status = DAT_CONN_QUAL_UNAVAILABLE; - goto bail; } /* set cm_handle for this service point, save listen socket */ sp_ptr->cm_srvc_handle = cm_ptr; - /* add to SP->CR thread list */ - dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry); - dapl_os_lock( &cm_ptr->hca_ptr->ib_trans.lock ); - dapl_llist_add_tail(&cm_ptr->hca_ptr->ib_trans.list, - (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr); - dapl_os_unlock(&cm_ptr->hca_ptr->ib_trans.lock); + /* queue up listen socket to process inbound CR's */ + cm_ptr->state = SCM_LISTEN; + dapli_cm_queue(cm_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " listen: qual 0x%x cr %p s_fd %d\n", + ntohs(serviceID), cm_ptr, cm_ptr->socket ); - dapl_dbg_log( DAPL_DBG_TYPE_CM, - " listen: qual 0x%x cr %p s_fd %d\n", - ntohs(serviceID), cm_ptr, cm_ptr->l_socket ); - return dat_status; bail: dapl_dbg_log( DAPL_DBG_TYPE_CM, " listen: ERROR on conn_qual 0x%x\n",serviceID); - if ( cm_ptr->l_socket >= 0 ) - close( cm_ptr->l_socket ); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); + dapli_cm_destroy(cm_ptr); return dat_status; } /* - * PASSIVE: send local QP information, private data, and wait for - * active side to respond with QP RTS/RTR status + * PASSIVE: accept socket, receive peer QP information, private data, post cr_event */ -static DAT_RETURN +DAT_RETURN dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) { - ib_cm_handle_t acm_ptr; + dp_ib_cm_handle_t acm_ptr; void *p_data = NULL; int len; DAT_RETURN dat_status = DAT_SUCCESS; + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket_accept\n"); + /* Allocate accept CM and initialize */ if ((acm_ptr = dapl_os_alloc(sizeof(*acm_ptr))) == NULL) return DAT_INSUFFICIENT_RESOURCES; @@ -342,155 +494,221 @@ dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) acm_ptr->socket = -1; acm_ptr->sp = cm_ptr->sp; - acm_ptr->hca_ptr = cm_ptr->hca_ptr; + acm_ptr->hca = cm_ptr->hca; len = sizeof(acm_ptr->dst.ia_address); - acm_ptr->socket = accept(cm_ptr->l_socket, + acm_ptr->socket = accept(cm_ptr->socket, (struct sockaddr*)&acm_ptr->dst.ia_address, - (socklen_t*)&len ); + (socklen_t*)&len); - if ( acm_ptr->socket < 0 ) { + if (acm_ptr->socket < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " accept: ERR %s on FD %d l_cr %p\n", - strerror(errno),cm_ptr->l_socket,cm_ptr); + strerror(errno),cm_ptr->socket,cm_ptr); dat_status = DAT_INTERNAL_ERROR; goto bail; } + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket accepted, read QP data\n"); + /* read in DST QP info, IA address. check for private data */ - len = read( acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t) ); - if ( len != sizeof(ib_qp_cm_t) ) { + len = read(acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t)); + if (len != sizeof(ib_qp_cm_t) || + ntohs(acm_ptr->dst.ver) != DSCM_VER) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept read: ERR %s, rcnt=%d\n", - strerror(errno), len); + " accept read: ERR %s, rcnt=%d, ver=%d\n", + strerror(errno), len, acm_ptr->dst.ver); dat_status = DAT_INTERNAL_ERROR; goto bail; - } + + /* convert accepted values to host order */ + acm_ptr->dst.port = ntohs(acm_ptr->dst.port); + acm_ptr->dst.lid = ntohs(acm_ptr->dst.lid); + acm_ptr->dst.qpn = ntohl(acm_ptr->dst.qpn); + acm_ptr->dst.p_size = ntohl(acm_ptr->dst.p_size); + dapl_dbg_log(DAPL_DBG_TYPE_EP, - " accept: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", - acm_ptr->dst.port, acm_ptr->dst.lid, - acm_ptr->dst.qpn, acm_ptr->dst.p_size ); + " accept: DST %s port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", + inet_ntoa(((struct sockaddr_in *)&acm_ptr->dst.ia_address)->sin_addr), + acm_ptr->dst.port, acm_ptr->dst.lid, + acm_ptr->dst.qpn, acm_ptr->dst.p_size); /* validate private data size before reading */ - if ( acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE ) { + if (acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept read: psize (%d) wrong\n", - acm_ptr->dst.p_size ); + " accept read: psize (%d) wrong\n", + acm_ptr->dst.p_size); dat_status = DAT_INTERNAL_ERROR; goto bail; } + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket accepted, read private data\n"); + /* read private data into cm_handle if any present */ - if ( acm_ptr->dst.p_size ) { + if (acm_ptr->dst.p_size) { len = read( acm_ptr->socket, - acm_ptr->p_data, acm_ptr->dst.p_size ); - if ( len != acm_ptr->dst.p_size ) { + acm_ptr->p_data, acm_ptr->dst.p_size); + if (len != acm_ptr->dst.p_size) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept read pdata: ERR %s, rcnt=%d\n", - strerror(errno), len ); + " accept read pdata: ERR %s, rcnt=%d\n", + strerror(errno), len); dat_status = DAT_INTERNAL_ERROR; goto bail; } - dapl_dbg_log(DAPL_DBG_TYPE_EP, - " accept: psize=%d read\n", - acm_ptr->dst.p_size); + dapl_dbg_log(DAPL_DBG_TYPE_EP," accept: psize=%d read\n",len); p_data = acm_ptr->p_data; } - /* trigger CR event and return SUCCESS */ - dapls_cr_callback( acm_ptr, - IB_CME_CONNECTION_REQUEST_PENDING, - p_data, - acm_ptr->sp ); + acm_ptr->state = SCM_ACCEPTING; + /* trigger CR event and return SUCCESS */ + dapls_cr_callback(acm_ptr, + IB_CME_CONNECTION_REQUEST_PENDING, + p_data, + acm_ptr->sp ); return DAT_SUCCESS; - bail: - if ( acm_ptr->socket >=0 ) - close( acm_ptr->socket ); - dapl_os_free( acm_ptr, sizeof( *acm_ptr ) ); + dapli_cm_destroy(acm_ptr); return DAT_INTERNAL_ERROR; } - -static DAT_RETURN -dapli_socket_accept_final( DAPL_EP *ep_ptr, - DAPL_CR *cr_ptr, - DAT_COUNT p_size, - DAT_PVOID p_data ) +/* + * PASSIVE: consumer accept, send local QP information, private data, + * queue on work thread to receive RTU information to avoid blocking + * user thread. + */ +DAT_RETURN +dapli_socket_accept_usr(DAPL_EP *ep_ptr, + DAPL_CR *cr_ptr, + DAT_COUNT p_size, + DAT_PVOID p_data) { DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; - ib_cm_handle_t cm_ptr = cr_ptr->ib_cm_handle; - ib_qp_cm_t qp_cm; + dp_ib_cm_handle_t cm_ptr = cr_ptr->ib_cm_handle; struct iovec iovec[2]; int len; - short rtu_data = 0; - if (p_size > IB_MAX_REP_PDATA_SIZE) + if (p_size > IB_MAX_REP_PDATA_SIZE) return DAT_LENGTH_ERROR; /* must have a accepted socket */ - if ( cm_ptr->socket < 0 ) + if (cm_ptr->socket < 0) return DAT_INTERNAL_ERROR; + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " accept_usr: remote port=0x%x lid=0x%x" + " qpn=0x%x psize=%d\n", + cm_ptr->dst.port, cm_ptr->dst.lid, + cm_ptr->dst.qpn, cm_ptr->dst.p_size); + /* modify QP to RTR and then to RTS with remote info already read */ - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTR, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTR, &cm_ptr->dst) != DAT_SUCCESS) goto bail; - if ( dapls_modify_qp_state( ep_ptr->qp_handle, - IBV_QPS_RTS, &cm_ptr->dst ) != DAT_SUCCESS ) + if (dapls_modify_qp_state(ep_ptr->qp_handle, + IBV_QPS_RTS, &cm_ptr->dst) != DAT_SUCCESS) goto bail; ep_ptr->qp_state = IB_QP_STATE_RTS; - /* Send QP info, IA address, and private data */ - qp_cm.qpn = ep_ptr->qp_handle->qp_num; - qp_cm.port = ia_ptr->hca_ptr->port_num; - qp_cm.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, - ia_ptr->hca_ptr->port_num ); - qp_cm.ia_address = ia_ptr->hca_ptr->hca_address; - qp_cm.p_size = p_size; - iovec[0].iov_base = &qp_cm; + /* save remote address information */ + dapl_os_memcpy( &ep_ptr->remote_ia_address, + &cm_ptr->dst.ia_address, + sizeof(ep_ptr->remote_ia_address)); + + /* send our QP info, IA address, and private data */ + cm_ptr->dst.qpn = htonl(ep_ptr->qp_handle->qp_num); + cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); + cm_ptr->dst.lid = htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num)); + if (cm_ptr->dst.lid == 0xffff) + goto bail; + + /* in network order */ + if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, + (uint8_t)ia_ptr->hca_ptr->port_num, + 0, + &cm_ptr->dst.gid)) + goto bail; + + cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; + cm_ptr->dst.p_size = htonl(p_size); + iovec[0].iov_base = &cm_ptr->dst; iovec[0].iov_len = sizeof(ib_qp_cm_t); if (p_size) { iovec[1].iov_base = p_data; iovec[1].iov_len = p_size; } - len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) ); + len = writev(cm_ptr->socket, iovec, (p_size ? 2:1)); if (len != (p_size + sizeof(ib_qp_cm_t))) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept_final: ERR %s, wcnt=%d\n", + " accept_rtu: ERR %s, wcnt=%d\n", strerror(errno), len); goto bail; } - dapl_dbg_log(DAPL_DBG_TYPE_EP, - " accept_final: SRC port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", - qp_cm.port, qp_cm.lid, qp_cm.qpn, qp_cm.p_size ); - + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " accept_usr: local port=0x%x lid=0x%x" + " qpn=0x%x psize=%d\n", + ntohs(cm_ptr->dst.port), ntohs(cm_ptr->dst.lid), + ntohl(cm_ptr->dst.qpn), ntohl(cm_ptr->dst.p_size)); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " accept_usr SRC GID subnet %016llx id %016llx\n", + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.subnet_prefix), + (unsigned long long) + cpu_to_be64(cm_ptr->dst.gid.global.interface_id)); + + /* save state and reference to EP, queue for RTU data */ + cm_ptr->ep = ep_ptr; + cm_ptr->hca = ia_ptr->hca_ptr; + cm_ptr->state = SCM_ACCEPTED; + + /* restore remote address information for query */ + dapl_os_memcpy( &cm_ptr->dst.ia_address, + &ep_ptr->remote_ia_address, + sizeof(cm_ptr->dst.ia_address)); + + dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: accepted!\n" ); + dapli_cm_queue(cm_ptr); + return DAT_SUCCESS; +bail: + dapl_dbg_log(DAPL_DBG_TYPE_ERR," accept_rtu: ERR !QP_RTR_RTS \n"); + dapli_cm_destroy(cm_ptr); + dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ + return DAT_INTERNAL_ERROR; +} + +/* + * PASSIVE: read RTU from active peer, post CONN event + */ +void +dapli_socket_accept_rtu(dp_ib_cm_handle_t cm_ptr) +{ + int len; + short rtu_data = 0; + /* complete handshake after final QP state change */ - len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data) ); - if ( len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f ) { + len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data)); + if (len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " accept_final: ERR %s, rcnt=%d rdata=%x\n", - strerror(errno), len, ntohs(rtu_data) ); + " accept_rtu: ERR %s, rcnt=%d rdata=%x\n", + strerror(errno), len, ntohs(rtu_data)); goto bail; } + /* save state and reference to EP, queue for disc event */ + cm_ptr->state = SCM_CONNECTED; + /* final data exchange if remote QP state is good to go */ dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: connected!\n" ); - dapls_cr_callback ( cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp ); - return DAT_SUCCESS; - + dapls_cr_callback(cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp); + return; bail: - dapl_dbg_log( DAPL_DBG_TYPE_ERR," accept_final: ERR !QP_RTR_RTS \n"); - if ( cm_ptr >= 0 ) - close( cm_ptr->socket ); - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); - dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */ - - return DAT_INTERNAL_ERROR; + dapls_ib_reinit_ep(cm_ptr->ep); /* reset QP state */ + dapli_cm_destroy(cm_ptr); + dapls_cr_callback(cm_ptr, IB_CME_DESTINATION_REJECT, NULL, cm_ptr->sp); } @@ -528,18 +746,13 @@ dapls_ib_connect ( dapl_dbg_log ( DAPL_DBG_TYPE_EP, " connect(ep_handle %p ....)\n", ep_handle); - /* - * Sanity check - */ - if ( NULL == ep_handle ) - return DAT_SUCCESS; ep_ptr = (DAPL_EP*)ep_handle; qp_ptr = ep_ptr->qp_handle; - return (dapli_socket_connect( ep_ptr, remote_ia_address, - remote_conn_qual, - private_data_size, private_data )); + return (dapli_socket_connect(ep_ptr, remote_ia_address, + remote_conn_qual, + private_data_size, private_data)); } /* @@ -556,41 +769,25 @@ dapls_ib_connect ( * * Returns: * DAT_SUCCESS - * */ DAT_RETURN -dapls_ib_disconnect ( +dapls_ib_disconnect( IN DAPL_EP *ep_ptr, - IN DAT_CLOSE_FLAGS close_flags ) + IN DAT_CLOSE_FLAGS close_flags) { - ib_cm_handle_t cm_ptr = ep_ptr->cm_handle; + dp_ib_cm_handle_t cm_ptr = ep_ptr->cm_handle; dapl_dbg_log (DAPL_DBG_TYPE_EP, "dapls_ib_disconnect(ep_handle %p ....)\n", ep_ptr); - if ( cm_ptr->socket >= 0 ) { - close( cm_ptr->socket ); - cm_ptr->socket = -1; - } - /* reinit to modify QP state */ dapls_ib_reinit_ep(ep_ptr); - if ( ep_ptr->cr_ptr ) { - dapls_cr_callback ( ep_ptr->cm_handle, - IB_CME_DISCONNECTED, - NULL, - ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr ); - } else { - dapl_evd_connection_callback ( ep_ptr->cm_handle, - IB_CME_DISCONNECTED, - NULL, - ep_ptr ); - ep_ptr->cm_handle = NULL; - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); - } - return DAT_SUCCESS; + if (cm_ptr == NULL) + return DAT_SUCCESS; + else + return(dapli_socket_disconnect(cm_ptr)); } /* @@ -679,13 +876,14 @@ dapls_ib_remove_conn_listener ( ia_ptr, sp_ptr, cm_ptr ); /* close accepted socket, free cm_srvc_handle and return */ - if ( cm_ptr != NULL ) { - if ( cm_ptr->l_socket >= 0 ) { - close( cm_ptr->l_socket ); + if (cm_ptr != NULL) { + if (cm_ptr->socket >= 0) { + close(cm_ptr->socket ); cm_ptr->socket = -1; } /* cr_thread will free */ sp_ptr->cm_srvc_handle = NULL; + write(g_scm_pipe[1], "w", sizeof "w"); } return DAT_SUCCESS; } @@ -720,23 +918,22 @@ dapls_ib_accept_connection ( DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - "dapls_ib_accept_connection(cr %p ep %p prd %p,%d)\n", - cr_handle, ep_handle, p_data, p_size ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + "dapls_ib_accept_connection(cr %p ep %p prd %p,%d)\n", + cr_handle, ep_handle, p_data, p_size ); - cr_ptr = (DAPL_CR *) cr_handle; - ep_ptr = (DAPL_EP *) ep_handle; + cr_ptr = (DAPL_CR *)cr_handle; + ep_ptr = (DAPL_EP *)ep_handle; /* allocate and attach a QP if necessary */ - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) { + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { DAT_RETURN status; - status = dapls_ib_qp_alloc( ep_ptr->header.owner_ia, - ep_ptr, ep_ptr ); - if ( status != DAT_SUCCESS ) + status = dapls_ib_qp_alloc(ep_ptr->header.owner_ia, + ep_ptr, ep_ptr); + if (status != DAT_SUCCESS) return status; } - - return ( dapli_socket_accept_final(ep_ptr, cr_ptr, p_size, p_data) ); + return(dapli_socket_accept_usr(ep_ptr, cr_ptr, p_size, p_data)); } @@ -757,23 +954,32 @@ dapls_ib_accept_connection ( * */ DAT_RETURN -dapls_ib_reject_connection ( - IN ib_cm_handle_t ib_cm_handle, - IN int reject_reason, - IN DAT_COUNT private_data_size, - IN const DAT_PVOID private_data) +dapls_ib_reject_connection( + IN dp_ib_cm_handle_t cm_ptr, + IN int reason, + IN DAT_COUNT psize, + IN const DAT_PVOID pdata) { - ib_cm_srvc_handle_t cm_ptr = ib_cm_handle; + struct iovec iovec; dapl_dbg_log (DAPL_DBG_TYPE_EP, - "dapls_ib_reject_connection(cm_handle %p reason %x)\n", - ib_cm_handle, reject_reason ); - - /* just close the socket and return */ - if ( cm_ptr->socket > 0 ) { - close( cm_ptr->socket ); + " reject(cm %p reason %x, pdata %p, psize %d)\n", + cm_ptr, reason, pdata, psize); + + /* write reject data to indicate reject */ + if (cm_ptr->socket >= 0) { + cm_ptr->dst.rej = (uint16_t)reason; + cm_ptr->dst.rej = htons(cm_ptr->dst.rej); + iovec.iov_base = &cm_ptr->dst; + iovec.iov_len = sizeof(ib_qp_cm_t); + writev(cm_ptr->socket, &iovec, 1); + close(cm_ptr->socket); cm_ptr->socket = -1; } + + /* cr_thread will destroy CR */ + cm_ptr->state = SCM_REJECTED; + write(g_scm_pipe[1], "w", sizeof "w"); return DAT_SUCCESS; } @@ -799,7 +1005,7 @@ dapls_ib_cm_remote_addr ( OUT DAT_SOCK_ADDR6 *remote_ia_address ) { DAPL_HEADER *header; - ib_cm_handle_t ib_cm_handle; + dp_ib_cm_handle_t ib_cm_handle; dapl_dbg_log (DAPL_DBG_TYPE_EP, "dapls_ib_cm_remote_addr(dat_handle %p, ....)\n", @@ -829,7 +1035,6 @@ dapls_ib_cm_remote_addr ( * Input: * prd_ptr private data pointer * conn_op connection operation type - * hca_ptr hca pointer, needed for transport type * * If prd_ptr is NULL, this is a query for the max size supported by * the provider, otherwise it is the actual size of the private data @@ -843,10 +1048,9 @@ dapls_ib_cm_remote_addr ( * length of private data * */ -int dapls_ib_private_data_size ( - IN DAPL_PRIVATE *prd_ptr, - IN DAPL_PDATA_OP conn_op, - IN DAPL_HCA *hca_ptr) +int dapls_ib_private_data_size( IN DAPL_PRIVATE *prd_ptr, + IN DAPL_PDATA_OP conn_op, + IN DAPL_HCA *hca_ptr) { int size; @@ -995,24 +1199,25 @@ dapls_ib_get_cm_event ( return ib_cm_event; } -/* async CR processing thread to avoid blocking applications */ +/* outbound/inbound CR processing thread to avoid blocking applications */ +#define SCM_MAX_CONN 8192 void cr_thread(void *arg) { struct dapl_hca *hca_ptr = arg; - ib_cm_srvc_handle_t cr, next_cr; - int max_fd; - fd_set rfd,rfds; - struct timeval to; + dp_ib_cm_handle_t cr, next_cr; + int ret,idx; + char rbuf[2]; + struct pollfd ufds[SCM_MAX_CONN]; dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cr_thread: ENTER hca %p\n",hca_ptr); dapl_os_lock( &hca_ptr->ib_trans.lock ); hca_ptr->ib_trans.cr_state = IB_THREAD_RUN; while (hca_ptr->ib_trans.cr_state == IB_THREAD_RUN) { - - FD_ZERO( &rfds ); - max_fd = -1; - + idx=0; + ufds[idx].fd = g_scm_pipe[0]; /* wakeup and process work */ + ufds[idx].events = POLLIN; + if (!dapl_llist_is_empty(&hca_ptr->ib_trans.list)) next_cr = dapl_llist_peek_head (&hca_ptr->ib_trans.list); else @@ -1020,51 +1225,70 @@ void cr_thread(void *arg) while (next_cr) { cr = next_cr; - dapl_dbg_log (DAPL_DBG_TYPE_CM," thread: cm_ptr %p\n", cr ); - if (cr->l_socket == -1 || + if ((cr->socket == -1) || hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) { - dapl_dbg_log(DAPL_DBG_TYPE_CM," thread: Freeing %p\n", cr); + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: Free %p\n", cr); next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list, (DAPL_LLIST_ENTRY*)&cr->entry ); dapl_llist_remove_entry(&hca_ptr->ib_trans.list, (DAPL_LLIST_ENTRY*)&cr->entry); - dapl_os_free( cr, sizeof(*cr) ); + dapl_os_free(cr, sizeof(*cr)); continue; } - - FD_SET( cr->l_socket, &rfds ); /* add to select set */ - if ( cr->l_socket > max_fd ) - max_fd = cr->l_socket; - - /* individual select poll to check for work */ - FD_ZERO(&rfd); - FD_SET(cr->l_socket, &rfd); - dapl_os_unlock(&hca_ptr->ib_trans.lock); - to.tv_sec = 0; - to.tv_usec = 0; - if ( select(cr->l_socket + 1,&rfd, NULL, NULL, &to) < 0) { - dapl_dbg_log (DAPL_DBG_TYPE_CM, - " thread: ERR %s on cr %p sk %d\n", - strerror(errno), cr, cr->l_socket); - close(cr->l_socket); - cr->l_socket = -1; - } else if ( FD_ISSET(cr->l_socket, &rfd) && - dapli_socket_accept(cr)) { - close(cr->l_socket); - cr->l_socket = -1; + + if (idx==SCM_MAX_CONN-1) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "SCM ERR: cm_thread exceeded FD_SETSIZE %d\n",idx+1); + continue; } - dapl_os_lock( &hca_ptr->ib_trans.lock ); + + /* Add to ufds for poll, check for immediate work */ + ufds[++idx].fd = cr->socket; /* add listen or cr */ + ufds[idx].events = POLLIN; + + /* check socket for event, accept in or connect out */ + dapl_dbg_log(DAPL_DBG_TYPE_CM," poll cr=%p, fd=%d,%d\n", + cr, cr->socket, ufds[idx].fd); + dapl_os_unlock(&hca_ptr->ib_trans.lock); + ret = poll(&ufds[idx],1,1); + dapl_dbg_log(DAPL_DBG_TYPE_CM," poll wakeup ret=%d cr->st=%d ev=%d fd=%d\n", + ret,cr->state,ufds[idx].revents,ufds[idx].fd); + + /* data on listen, qp exchange, and on disconnect request */ + if ((ret == 1) && ufds[idx].revents == POLLIN) { + if (cr->socket > 0) { + if (cr->state == SCM_LISTEN) + dapli_socket_accept(cr); + else if (cr->state == SCM_ACCEPTED) + dapli_socket_accept_rtu(cr); + else if (cr->state == SCM_CONN_PENDING) + dapli_socket_connect_rtu(cr); + else if (cr->state == SCM_CONNECTED) + dapli_socket_disconnect(cr); + } + } else if (ret != 0) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " cr_thread(cr=%p) st=%d poll ERR= %s\n", + cr,cr->state,strerror(errno)); + /* POLLUP or poll error case, issue event if connected */ + if (cr->state == SCM_CONNECTED) + dapli_socket_disconnect(cr); + } + dapl_os_lock(&hca_ptr->ib_trans.lock); next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list, - (DAPL_LLIST_ENTRY*)&cr->entry ); + (DAPL_LLIST_ENTRY*)&cr->entry); } - dapl_os_unlock( &hca_ptr->ib_trans.lock ); - to.tv_sec = 0; - to.tv_usec = 100000; /* wakeup and check destroy */ - select(max_fd + 1, &rfds, NULL, NULL, &to); - dapl_os_lock( &hca_ptr->ib_trans.lock ); + dapl_os_unlock(&hca_ptr->ib_trans.lock); + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: sleep, %d\n", idx+1); + poll(ufds,idx+1,-1); /* infinite, all sockets and pipe */ + /* if pipe used to wakeup, consume */ + if (ufds[0].revents == POLLIN) + read(g_scm_pipe[0], rbuf, 2); + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: wakeup\n"); + dapl_os_lock(&hca_ptr->ib_trans.lock); } - dapl_os_unlock( &hca_ptr->ib_trans.lock ); + dapl_os_unlock(&hca_ptr->ib_trans.lock); hca_ptr->ib_trans.cr_state = IB_THREAD_EXIT; dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cr_thread(hca %p) exit\n",hca_ptr); } diff --git a/dapl/openib_scm/dapl_ib_cq.c b/dapl/openib_scm/dapl_ib_cq.c index 7ac7037..56b729e 100644 --- a/dapl/openib_scm/dapl_ib_cq.c +++ b/dapl/openib_scm/dapl_ib_cq.c @@ -97,7 +97,7 @@ void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr) while (hca_ptr->ib_trans.cq_state != IB_THREAD_EXIT) { struct timespec sleep, remain; sleep.tv_sec = 0; - sleep.tv_nsec = 200000000; /* 200 ms */ + sleep.tv_nsec = 2000000; /* 2 ms */ dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " cq_thread_destroy: waiting for cq_thread\n"); nanosleep (&sleep, &remain); @@ -422,12 +422,21 @@ DAT_RETURN dapls_ib_cq_free ( IN DAPL_IA *ia_ptr, IN DAPL_EVD *evd_ptr) { - if ( evd_ptr->ib_cq_handle != IB_INVALID_HANDLE ) { - /* copy all entries on CQ to EVD before destroying */ - dapls_evd_copy_cq(evd_ptr); + DAT_EVENT event; + + if (evd_ptr->ib_cq_handle != IB_INVALID_HANDLE) { + /* pull off CQ and EVD entries and toss */ + while (dapls_ib_completion_poll == DAT_SUCCESS); + while (dapl_evd_dequeue(evd_ptr,&event) != DAT_QUEUE_EMPTY); +#if 1 + ibv_destroy_cq(evd_ptr->ib_cq_handle); + evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; + return DAT_SUCCESS; +#else if (ibv_destroy_cq(evd_ptr->ib_cq_handle)) return(dapl_convert_errno(errno,"destroy_cq")); evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; +#endif } return DAT_SUCCESS; } @@ -600,7 +609,7 @@ dapls_ib_wait_object_wait ( status = ETIMEDOUT; dapl_dbg_log (DAPL_DBG_TYPE_CM, - " cq_object_wait: RET evd %p ibv_cq %p ibv_ctx %p %s\n", + " cq_object_wait: RET evd %p ibv_cq %p %s\n", evd_ptr, ibv_cq,strerror(errno)); return(dapl_convert_errno(status,"cq_wait_object_wait")); diff --git a/dapl/openib_scm/dapl_ib_dto.h b/dapl/openib_scm/dapl_ib_dto.h index bea3e4d..6ba669f 100644 --- a/dapl/openib_scm/dapl_ib_dto.h +++ b/dapl/openib_scm/dapl_ib_dto.h @@ -35,7 +35,7 @@ * * Description: * - * The uDAPL openib provider - DTO operations and CQE macros + * The OpenIB uCMA provider - DTO operations and CQE macros * ************************************************************************ **** * Source Control System Information @@ -50,6 +50,10 @@ #include "dapl_ib_util.h" +#ifdef DAT_EXTENSIONS +#include +#endif + #define DEFAULT_DS_ENTRIES 8 STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p); @@ -61,26 +65,26 @@ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p); */ STATIC _INLINE_ DAT_RETURN dapls_ib_post_recv ( - IN DAPL_EP *ep_ptr, + IN DAPL_EP *ep_ptr, IN DAPL_COOKIE *cookie, - IN DAT_COUNT segments, + IN DAT_COUNT segments, IN DAT_LMR_TRIPLET *local_iov ) { - ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; - ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; - struct ibv_recv_wr wr; - struct ibv_recv_wr *bad_wr; - DAT_COUNT i, total_len; - int ret; + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; + ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; + struct ibv_recv_wr wr; + struct ibv_recv_wr *bad_wr; + DAT_COUNT i, total_len; + int ret; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_rcv: ep %p cookie %p segs %d l_iov %p\n", - ep_ptr, cookie, segments, local_iov); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_rcv: ep %p cookie %p segs %d l_iov %p\n", + ep_ptr, cookie, segments, local_iov); - if ( segments <= DEFAULT_DS_ENTRIES ) + if (segments <= DEFAULT_DS_ENTRIES) ds_array_p = ds_array; else - ds_array_start_p = ds_array_p = + ds_array_start_p = ds_array_p = dapl_os_alloc(segments * sizeof(ib_data_segment_t)); if (NULL == ds_array_p) @@ -93,18 +97,18 @@ dapls_ib_post_recv ( wr.wr_id = (uint64_t)(uintptr_t)cookie; wr.sg_list = ds_array_p; - for (i = 0; i < segments; i++ ) { - if ( !local_iov[i].segment_length ) + for (i = 0; i < segments; i++) { + if (!local_iov[i].segment_length) continue; - ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; + ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; ds_array_p->length = local_iov[i].segment_length; - ds_array_p->lkey = local_iov[i].lmr_context; + ds_array_p->lkey = local_iov[i].lmr_context; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " post_rcv: l_key 0x%x va %p len %d\n", - ds_array_p->lkey, ds_array_p->addr, - ds_array_p->length ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_rcv: l_key 0x%x va %p len %d\n", + ds_array_p->lkey, ds_array_p->addr, + ds_array_p->length ); total_len += ds_array_p->length; wr.num_sge++; @@ -117,15 +121,14 @@ dapls_ib_post_recv ( ret = ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr); if (ds_array_start_p != NULL) - dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); + dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); if (ret) - return( dapl_convert_errno(EFAULT,"ibv_recv") ); + return( dapl_convert_errno(errno,"ibv_recv") ); return DAT_SUCCESS; } - /* * dapls_ib_post_send * @@ -133,35 +136,37 @@ dapls_ib_post_recv ( */ STATIC _INLINE_ DAT_RETURN dapls_ib_post_send ( - IN DAPL_EP *ep_ptr, - IN ib_send_op_type_t op_type, - IN DAPL_COOKIE *cookie, - IN DAT_COUNT segments, - IN DAT_LMR_TRIPLET *local_iov, - IN const DAT_RMR_TRIPLET *remote_iov, - IN DAT_COMPLETION_FLAGS completion_flags) + IN DAPL_EP *ep_ptr, + IN ib_send_op_type_t op_type, + IN DAPL_COOKIE *cookie, + IN DAT_COUNT segments, + IN DAT_LMR_TRIPLET *local_iov, + IN const DAT_RMR_TRIPLET *remote_iov, + IN DAT_COMPLETION_FLAGS completion_flags) { - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_snd: ep %p op %d ck %p sgs %d l_iov %p r_iov %p f %d\n", - ep_ptr, op_type, cookie, segments, local_iov, - remote_iov, completion_flags); - - ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; - ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; - struct ibv_send_wr wr; - struct ibv_send_wr *bad_wr; - ib_hca_transport_t *ibt_ptr = &ep_ptr->header.owner_ia->hca_ptr->ib_trans; - DAT_COUNT i, total_len; - int ret; + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: ep %p op %d ck %p sgs", + "%d l_iov %p r_iov %p f %d\n", + ep_ptr, op_type, cookie, segments, local_iov, + remote_iov, completion_flags); + + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; + ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; + struct ibv_send_wr wr; + struct ibv_send_wr *bad_wr; + ib_hca_transport_t *ibt_ptr = + &ep_ptr->header.owner_ia->hca_ptr->ib_trans; + DAT_COUNT i, total_len; + int ret; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_snd: ep %p cookie %p segs %d l_iov %p\n", - ep_ptr, cookie, segments, local_iov); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: ep %p cookie %p segs %d l_iov %p\n", + ep_ptr, cookie, segments, local_iov); - if( segments <= DEFAULT_DS_ENTRIES ) + if(segments <= DEFAULT_DS_ENTRIES) ds_array_p = ds_array; else - ds_array_start_p = ds_array_p = + ds_array_start_p = ds_array_p = dapl_os_alloc(segments * sizeof(ib_data_segment_t)); if (NULL == ds_array_p) @@ -180,14 +185,14 @@ dapls_ib_post_send ( if ( !local_iov[i].segment_length ) continue; - ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; + ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; ds_array_p->length = local_iov[i].segment_length; - ds_array_p->lkey = local_iov[i].lmr_context; + ds_array_p->lkey = local_iov[i].lmr_context; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " post_snd: lkey 0x%x va %p len %d \n", - ds_array_p->lkey, ds_array_p->addr, - ds_array_p->length ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: lkey 0x%x va %p len %d\n", + ds_array_p->lkey, ds_array_p->addr, + ds_array_p->length ); total_len += ds_array_p->length; wr.num_sge++; @@ -196,20 +201,22 @@ dapls_ib_post_send ( if (cookie != NULL) cookie->val.dto.size = total_len; - - if ((op_type == OP_RDMA_WRITE) || (op_type == OP_RDMA_READ)) { - wr.wr.rdma.remote_addr = remote_iov->target_address; + + if (wr.num_sge && + (op_type == OP_RDMA_WRITE || op_type == OP_RDMA_READ)) { + wr.wr.rdma.remote_addr = remote_iov->virtual_address; wr.wr.rdma.rkey = remote_iov->rmr_context; - dapl_dbg_log ( DAPL_DBG_TYPE_EP, - " post_snd_rdma: rkey 0x%x va %#016Lx\n", - wr.wr.rdma.rkey, wr.wr.rdma.remote_addr ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd_rdma: rkey 0x%x va %#016Lx\n", + wr.wr.rdma.rkey, wr.wr.rdma.remote_addr); } + /* inline data for send or write ops */ - if ((total_len <= ibt_ptr->max_inline_send ) && + if ((total_len <= ibt_ptr->max_inline_send) && ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE))) wr.send_flags |= IBV_SEND_INLINE; - + /* set completion flags in work request */ wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG & completion_flags) ? 0 : IBV_SEND_SIGNALED; @@ -218,24 +225,205 @@ dapls_ib_post_send ( wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG & completion_flags) ? IBV_SEND_SOLICITED : 0; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", - wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", + wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); ret = ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr); - + if (ds_array_start_p != NULL) - dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); + dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); if (ret) - return( dapl_convert_errno(EFAULT,"ibv_send") ); + return( dapl_convert_errno(errno,"ibv_send") ); + + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); + return DAT_SUCCESS; +} + +/* map Work Completions to DAPL WR operations */ +STATIC _INLINE_ DAT_DTOS dapls_cqe_dtos_opcode(ib_work_completion_t *cqe_p) +{ + switch (cqe_p->opcode) { + + case IBV_WC_SEND: + return (DAT_DTO_SEND); + case IBV_WC_RDMA_READ: + return (DAT_DTO_RDMA_READ); + case IBV_WC_BIND_MW: + return (DAT_DTO_BIND_MW); +#ifdef DAT_EXTENSIONS + case IBV_WC_RDMA_WRITE: + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) + return (DAT_IB_DTO_RDMA_WRITE_IMMED); + else + return (DAT_DTO_RDMA_WRITE); + case IBV_WC_COMP_SWAP: + return (DAT_IB_DTO_CMP_SWAP); + case IBV_WC_FETCH_ADD: + return (DAT_IB_DTO_FETCH_ADD); + case IBV_WC_RECV_RDMA_WITH_IMM: + return (DAT_IB_DTO_RECV_IMMED); +#else + case IBV_WC_RDMA_WRITE: + return (DAT_DTO_RDMA_WRITE); +#endif + case IBV_WC_RECV: + return (DAT_DTO_RECEIVE); + default: + return (0xff); + } +} +#define DAPL_GET_CQE_DTOS_OPTYPE(cqe_p) dapls_cqe_dtos_opcode(cqe_p) + + +#ifdef DAT_EXTENSIONS +/* + * dapls_ib_post_ext_send + * + * Provider specific extended Post SEND function for atomics + * OP_COMP_AND_SWAP and OP_FETCH_AND_ADD + */ +STATIC _INLINE_ DAT_RETURN +dapls_ib_post_ext_send ( + IN DAPL_EP *ep_ptr, + IN ib_send_op_type_t op_type, + IN DAPL_COOKIE *cookie, + IN DAT_COUNT segments, + IN DAT_LMR_TRIPLET *local_iov, + IN const DAT_RMR_TRIPLET *remote_iov, + IN DAT_UINT32 immed_data, + IN DAT_UINT64 compare_add, + IN DAT_UINT64 swap, + IN DAT_COMPLETION_FLAGS completion_flags) +{ + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: ep %p op %d ck %p sgs", + "%d l_iov %p r_iov %p f %d\n", + ep_ptr, op_type, cookie, segments, local_iov, + remote_iov, completion_flags); + + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; + ib_data_segment_t *ds_array_p, *ds_array_start_p; + struct ibv_send_wr wr; + struct ibv_send_wr *bad_wr; + DAT_COUNT i, total_len; + int ret; + + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: ep %p cookie %p segs %d l_iov %p\n", + ep_ptr, cookie, segments, local_iov); + + if(segments <= DEFAULT_DS_ENTRIES) + ds_array_p = ds_array; + else + ds_array_start_p = ds_array_p = + dapl_os_alloc(segments * sizeof(ib_data_segment_t)); + + if (NULL == ds_array_p) + return (DAT_INSUFFICIENT_RESOURCES); + + /* setup the work request */ + wr.next = 0; + wr.opcode = op_type; + wr.num_sge = 0; + wr.send_flags = 0; + wr.wr_id = (uint64_t)(uintptr_t)cookie; + wr.sg_list = ds_array_p; + total_len = 0; + + for (i = 0; i < segments; i++ ) { + if ( !local_iov[i].segment_length ) + continue; + + ds_array_p->addr = (uint64_t) local_iov[i].virtual_address; + ds_array_p->length = local_iov[i].segment_length; + ds_array_p->lkey = local_iov[i].lmr_context; + + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: lkey 0x%x va %p len %d\n", + ds_array_p->lkey, ds_array_p->addr, + ds_array_p->length ); + + total_len += ds_array_p->length; + wr.num_sge++; + ds_array_p++; + } - dapl_dbg_log (DAPL_DBG_TYPE_EP," post_snd: returned\n"); + if (cookie != NULL) + cookie->val.dto.size = total_len; + + switch (op_type) { + case OP_RDMA_WRITE_IMM: + /* OP_RDMA_WRITE)IMMED has direct IB wr_type mapping */ + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_ext: rkey 0x%x va %#016Lx immed=0x%x\n", + remote_iov?remote_iov->rmr_context:0, + remote_iov?remote_iov->virtual_address:0, + immed_data); + + wr.imm_data = immed_data; + if (wr.num_sge) { + wr.wr.rdma.remote_addr = remote_iov->virtual_address; + wr.wr.rdma.rkey = remote_iov->rmr_context; + } + break; + case OP_COMP_AND_SWAP: + /* OP_COMP_AND_SWAP has direct IB wr_type mapping */ + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_ext: OP_COMP_AND_SWAP=%lx," + "%lx rkey 0x%x va %#016Lx\n", + compare_add, swap, remote_iov->rmr_context, + remote_iov->virtual_address); + + wr.wr.atomic.compare_add = compare_add; + wr.wr.atomic.swap = swap; + wr.wr.atomic.remote_addr = remote_iov->virtual_address; + wr.wr.atomic.rkey = remote_iov->rmr_context; + break; + case OP_FETCH_AND_ADD: + /* OP_FETCH_AND_ADD has direct IB wr_type mapping */ + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_ext: OP_FETCH_AND_ADD=%lx," + "%lx rkey 0x%x va %#016Lx\n", + compare_add, remote_iov->rmr_context, + remote_iov->virtual_address); + + wr.wr.atomic.compare_add = compare_add; + wr.wr.atomic.remote_addr = remote_iov->virtual_address; + wr.wr.atomic.rkey = remote_iov->rmr_context; + break; + default: + break; + } + + /* set completion flags in work request */ + wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG & + completion_flags) ? 0 : IBV_SEND_SIGNALED; + wr.send_flags |= (DAT_COMPLETION_BARRIER_FENCE_FLAG & + completion_flags) ? IBV_SEND_FENCE : 0; + wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG & + completion_flags) ? IBV_SEND_SOLICITED : 0; + + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", + wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); + + ret = ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr); + + if (segments > DEFAULT_DS_ENTRIES) + dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); + + if (ret) + return( dapl_convert_errno(errno,"ibv_send") ); + + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); return DAT_SUCCESS; } +#endif STATIC _INLINE_ DAT_RETURN -dapls_ib_optional_prv_dat ( +dapls_ib_optional_prv_dat( IN DAPL_CR *cr_ptr, IN const void *event_data, OUT DAPL_CR **cr_pp) @@ -243,34 +431,69 @@ dapls_ib_optional_prv_dat ( return DAT_SUCCESS; } +/* map Work Completions to DAPL WR operations */ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) { - switch (cqe_p->opcode) { + switch (cqe_p->opcode) { case IBV_WC_SEND: - return (OP_SEND); + return (OP_SEND); case IBV_WC_RDMA_WRITE: - return (OP_RDMA_WRITE); + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) + return (OP_RDMA_WRITE_IMM); + else + return (OP_RDMA_WRITE); case IBV_WC_RDMA_READ: - return (OP_RDMA_READ); + return (OP_RDMA_READ); case IBV_WC_COMP_SWAP: - return (OP_COMP_AND_SWAP); + return (OP_COMP_AND_SWAP); case IBV_WC_FETCH_ADD: - return (OP_FETCH_AND_ADD); + return (OP_FETCH_AND_ADD); case IBV_WC_BIND_MW: - return (OP_BIND_MW); + return (OP_BIND_MW); case IBV_WC_RECV: - return (OP_RECEIVE); + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) + return (OP_RECEIVE_IMM); + else + return (OP_RECEIVE); case IBV_WC_RECV_RDMA_WITH_IMM: - return (OP_RECEIVE_IMM); + return (OP_RECEIVE_IMM); default: - return (OP_INVALID); - } + return (OP_INVALID); + } +} + +#define DAPL_GET_CQE_OPTYPE(cqe_p) dapls_cqe_opcode(cqe_p) +#define DAPL_GET_CQE_WRID(cqe_p) ((ib_work_completion_t*)cqe_p)->wr_id +#define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t*)cqe_p)->status +#define DAPL_GET_CQE_VENDOR_ERR(cqe_p) ((ib_work_completion_t*)cqe_p)->vendor_err +#define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t*)cqe_p)->byte_len +#define DAPL_GET_CQE_IMMED_DATA(cqe_p) ((ib_work_completion_t*)cqe_p)->imm_data + +STATIC _INLINE_ char * dapls_dto_op_str(int op) +{ + static char *optable[] = + { + "OP_RDMA_WRITE", + "OP_RDMA_WRITE_IMM", + "OP_SEND", + "OP_SEND_IMM", + "OP_RDMA_READ", + "OP_COMP_AND_SWAP", + "OP_FETCH_AND_ADD", + "OP_RECEIVE", + "OP_RECEIVE_IMM", + "OP_BIND_MW", + 0 + }; + return ((op < 0 || op > 9) ? "Invalid CQE OP?" : optable[op]); +} + +static _INLINE_ char * +dapls_cqe_op_str(IN ib_work_completion_t *cqe_ptr) +{ + return dapls_dto_op_str(DAPL_GET_CQE_OPTYPE(cqe_ptr)); } -#define DAPL_GET_CQE_OPTYPE(cqe_p) dapls_cqe_opcode(cqe_p) -#define DAPL_GET_CQE_WRID(cqe_p) ((ib_work_completion_t*)cqe_p)->wr_id -#define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t*)cqe_p)->status -#define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t*)cqe_p)->byte_len -#define DAPL_GET_CQE_IMMED_DATA(cqe_p) ((ib_work_completion_t*)cqe_p)->imm_data +#define DAPL_GET_CQE_OP_STR(cqe) dapls_cqe_op_str(cqe) #endif /* _DAPL_IB_DTO_H_ */ diff --git a/dapl/openib_scm/dapl_ib_extensions.c b/dapl/openib_scm/dapl_ib_extensions.c new file mode 100755 index 0000000..1402057 --- /dev/null +++ b/dapl/openib_scm/dapl_ib_extensions.c @@ -0,0 +1,313 @@ +/* + * Copyright (c) 2007 Intel Corporation. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/********************************************************************** + * + * MODULE: dapl_ib_extensions.c + * + * PURPOSE: Extensions routines for OpenIB uCMA provider + * + * $Id: $ + * + **********************************************************************/ + +#include "dapl.h" +#include "dapl_adapter_util.h" +#include "dapl_evd_util.h" +#include "dapl_ib_util.h" +#include "dapl_ep_util.h" +#include "dapl_cookie.h" +#include + + +DAT_RETURN +dapli_post_ext( IN DAT_EP_HANDLE ep_handle, + IN DAT_UINT64 cmp_add, + IN DAT_UINT64 swap, + IN DAT_UINT32 immed_data, + IN DAT_COUNT segments, + IN DAT_LMR_TRIPLET *local_iov, + IN DAT_DTO_COOKIE user_cookie, + IN const DAT_RMR_TRIPLET *remote_iov, + IN int op_type, + IN DAT_COMPLETION_FLAGS flags ); + + +/* + * dapl_extensions + * + * Process extension requests + * + * Input: + * ext_type, + * ... + * + * Output: + * Depends.... + * + * Returns: + * DAT_SUCCESS + * DAT_NOT_IMPLEMENTED + * ..... + * + */ +DAT_RETURN +dapl_extensions(IN DAT_HANDLE dat_handle, + IN DAT_EXTENDED_OP ext_op, + IN va_list args) +{ + DAT_EP_HANDLE ep; + DAT_LMR_TRIPLET *lmr_p; + DAT_DTO_COOKIE cookie; + const DAT_RMR_TRIPLET *rmr_p; + DAT_UINT64 dat_uint64a, dat_uint64b; + DAT_UINT32 dat_uint32; + DAT_COUNT segments = 1; + DAT_COMPLETION_FLAGS comp_flags; + DAT_RETURN status = DAT_NOT_IMPLEMENTED; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dapl_extensions(hdl %p operation %d, ...)\n", + dat_handle, ext_op); + + DAPL_CNTR(DCNT_EXTENSION); + + switch ((int)ext_op) + { + + case DAT_IB_RDMA_WRITE_IMMED_OP: + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " WRITE_IMMED_DATA extension call\n"); + + ep = dat_handle; /* ep_handle */ + segments = va_arg( args, DAT_COUNT); /* num segments */ + lmr_p = va_arg( args, DAT_LMR_TRIPLET*); + cookie = va_arg( args, DAT_DTO_COOKIE); + rmr_p = va_arg( args, const DAT_RMR_TRIPLET*); + dat_uint32 = va_arg( args, DAT_UINT32); /* immed data */ + comp_flags = va_arg( args, DAT_COMPLETION_FLAGS); + + status = dapli_post_ext(ep, 0, 0, dat_uint32, segments, lmr_p, + cookie, rmr_p, OP_RDMA_WRITE_IMM, + comp_flags ); + break; + + case DAT_IB_CMP_AND_SWAP_OP: + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " CMP_AND_SWAP extension call\n"); + + ep = dat_handle; /* ep_handle */ + dat_uint64a = va_arg( args, DAT_UINT64); /* cmp_value */ + dat_uint64b = va_arg( args, DAT_UINT64); /* swap_value */ + lmr_p = va_arg( args, DAT_LMR_TRIPLET*); + cookie = va_arg( args, DAT_DTO_COOKIE); + rmr_p = va_arg( args, const DAT_RMR_TRIPLET*); + comp_flags = va_arg( args, DAT_COMPLETION_FLAGS); + + status = dapli_post_ext(ep, dat_uint64a, dat_uint64b, + 0, segments, lmr_p, cookie, rmr_p, + OP_COMP_AND_SWAP, comp_flags ); + break; + + case DAT_IB_FETCH_AND_ADD_OP: + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " FETCH_AND_ADD extension call\n"); + + ep = dat_handle; /* ep_handle */ + dat_uint64a = va_arg( args, DAT_UINT64); /* add value */ + lmr_p = va_arg( args, DAT_LMR_TRIPLET*); + cookie = va_arg( args, DAT_DTO_COOKIE); + rmr_p = va_arg( args, const DAT_RMR_TRIPLET*); + comp_flags = va_arg( args, DAT_COMPLETION_FLAGS); + + status = dapli_post_ext(ep, dat_uint64a, 0, 0, segments, + lmr_p, cookie, rmr_p, + OP_FETCH_AND_ADD, comp_flags ); + + break; + + default: + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "unsupported extension(%d)\n", (int)ext_op); + } + + return(status); +} + + +DAT_RETURN +dapli_post_ext( IN DAT_EP_HANDLE ep_handle, + IN DAT_UINT64 cmp_add, + IN DAT_UINT64 swap, + IN DAT_UINT32 immed_data, + IN DAT_COUNT segments, + IN DAT_LMR_TRIPLET *local_iov, + IN DAT_DTO_COOKIE user_cookie, + IN const DAT_RMR_TRIPLET *remote_iov, + IN int op_type, + IN DAT_COMPLETION_FLAGS flags ) +{ + DAPL_EP *ep_ptr; + ib_qp_handle_t qp_ptr; + DAPL_COOKIE *cookie = NULL; + DAT_RETURN dat_status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + " post_ext_op: ep %p cmp_val %d " + "swap_val %d cookie 0x%x, r_iov %p, flags 0x%x\n", + ep_handle, (unsigned)cmp_add, (unsigned)swap, + (unsigned)user_cookie.as_64, remote_iov, flags); + + if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + return(DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP)); + + ep_ptr = (DAPL_EP *) ep_handle; + qp_ptr = ep_ptr->qp_handle; + + /* + * Synchronization ok since this buffer is only used for send + * requests, which aren't allowed to race with each other. + */ + dat_status = dapls_dto_cookie_alloc(&ep_ptr->req_buffer, + DAPL_DTO_TYPE_EXTENSION, + user_cookie, + &cookie); + if (dat_status != DAT_SUCCESS) + goto bail; + + /* + * Take reference before posting to avoid race conditions with + * completions + */ + dapl_os_atomic_inc(&ep_ptr->req_count); + + /* + * Invoke provider specific routine to post DTO + */ + dat_status = dapls_ib_post_ext_send(ep_ptr, + op_type, + cookie, + segments, /* data segments */ + local_iov, + remote_iov, + immed_data, /* immed data */ + cmp_add, /* compare or add */ + swap, /* swap */ + flags); + + if (dat_status != DAT_SUCCESS) { + dapl_os_atomic_dec(&ep_ptr->req_count); + dapls_cookie_dealloc(&ep_ptr->req_buffer, cookie); + } + +bail: + return dat_status; + +} + + +/* + * New provider routine to process extended DTO events + */ +void +dapls_cqe_to_event_extension(IN DAPL_EP *ep_ptr, + IN DAPL_COOKIE *cookie, + IN ib_work_completion_t *cqe_ptr, + IN DAT_EVENT *event_ptr) +{ + uint32_t ibtype; + DAT_DTO_COMPLETION_EVENT_DATA *dto = + &event_ptr->event_data.dto_completion_event_data; + DAT_IB_EXTENSION_EVENT_DATA *ext_data = + (DAT_IB_EXTENSION_EVENT_DATA *) + &event_ptr->event_extension_data[0]; + DAT_DTO_COMPLETION_STATUS dto_status; + + /* Get status from cqe */ + dto_status = dapls_ib_get_dto_status(cqe_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_EVD, + " cqe_to_event_ext: dto_ptr %p ext_ptr %p status %d\n", + dto, ext_data, dto_status); + + event_ptr->event_number = DAT_IB_DTO_EVENT; + dto->ep_handle = cookie->ep; + dto->user_cookie = cookie->val.dto.cookie; + dto->operation = DAPL_GET_CQE_DTOS_OPTYPE(cqe_ptr); /* new for 2.0 */ + dto->status = ext_data->status = dto_status; + + if (dto_status != DAT_DTO_SUCCESS) + return; + + /* + * Get operation type from CQ work completion entry and + * if extented operation then set extended event data + */ + ibtype = DAPL_GET_CQE_OPTYPE(cqe_ptr); + + switch (ibtype) { + + case OP_RDMA_WRITE_IMM: + dapl_dbg_log (DAPL_DBG_TYPE_EVD, + " cqe_to_event_ext: OP_RDMA_WRITE_IMMED\n"); + + /* type and outbound rdma write transfer size */ + dto->transfered_length = cookie->val.dto.size; + ext_data->type = DAT_IB_RDMA_WRITE_IMMED; + break; + case OP_RECEIVE_IMM: + dapl_dbg_log (DAPL_DBG_TYPE_EVD, + " cqe_to_event_ext: OP_RECEIVE_RDMA_IMMED\n"); + + /* immed recvd, type and inbound rdma write transfer size */ + dto->transfered_length = DAPL_GET_CQE_BYTESNUM(cqe_ptr); + ext_data->type = DAT_IB_RDMA_WRITE_IMMED_DATA; + ext_data->val.immed.data = DAPL_GET_CQE_IMMED_DATA(cqe_ptr); + break; + case OP_COMP_AND_SWAP: + dapl_dbg_log (DAPL_DBG_TYPE_EVD, + " cqe_to_event_ext: COMP_AND_SWAP_RESP\n"); + + /* original data is returned in LMR provided with post */ + ext_data->type = DAT_IB_CMP_AND_SWAP; + dto->transfered_length = DAPL_GET_CQE_BYTESNUM(cqe_ptr); + break; + case OP_FETCH_AND_ADD: + dapl_dbg_log (DAPL_DBG_TYPE_EVD, + " cqe_to_event_ext: FETCH_AND_ADD_RESP\n"); + + /* original data is returned in LMR provided with post */ + ext_data->type = DAT_IB_FETCH_AND_ADD; + dto->transfered_length = DAPL_GET_CQE_BYTESNUM(cqe_ptr); + break; + default: + /* not extended operation */ + ext_data->status = DAT_IB_OP_ERR; + dto->status = DAT_DTO_ERR_TRANSPORT; + break; + } +} diff --git a/dapl/openib_scm/dapl_ib_mem.c b/dapl/openib_scm/dapl_ib_mem.c index de36c0f..5bd5342 100644 --- a/dapl/openib_scm/dapl_ib_mem.c +++ b/dapl/openib_scm/dapl_ib_mem.c @@ -1,4 +1,6 @@ /* + * Copyright (c) 2005-2007 Intel Corporation. All rights reserved. + * * This Software is licensed under one of the following licenses: * * 1) under the terms of the "Common Public License 1.0" a copy of which is @@ -25,12 +27,11 @@ /********************************************************************** * - * MODULE: dapl_det_mem.c + * MODULE: dapl_ib_mem.c * - * PURPOSE: Intel DET APIs: Memory windows, registration, - * and protection domain + * PURPOSE: Memory windows, registration, and protection domain * - * $Id: $ + * $Id:$ * **********************************************************************/ @@ -61,8 +62,7 @@ * */ STATIC _INLINE_ int -dapls_convert_privileges ( - IN DAT_MEM_PRIV_FLAGS privileges) +dapls_convert_privileges(IN DAT_MEM_PRIV_FLAGS privileges) { int access = 0; @@ -79,6 +79,11 @@ dapls_convert_privileges ( access |= IBV_ACCESS_REMOTE_READ; if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges) access |= IBV_ACCESS_REMOTE_READ; +#ifdef DAT_EXTENSIONS + if (DAT_IB_MEM_PRIV_REMOTE_ATOMIC & privileges) + access |= IBV_ACCESS_REMOTE_ATOMIC; +#endif + return access; } @@ -101,16 +106,15 @@ dapls_convert_privileges ( * */ DAT_RETURN -dapls_ib_pd_alloc ( - IN DAPL_IA *ia_ptr, - IN DAPL_PZ *pz ) +dapls_ib_pd_alloc(IN DAPL_IA *ia_ptr, IN DAPL_PZ *pz) { /* get a protection domain */ pz->pd_handle = ibv_alloc_pd(ia_ptr->hca_ptr->ib_hca_handle); if (!pz->pd_handle) return(dapl_convert_errno(ENOMEM,"alloc_pd")); - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " pd_alloc: pd_handle=%p\n", + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " pd_alloc: pd_handle=%p\n", pz->pd_handle ); return DAT_SUCCESS; @@ -134,13 +138,18 @@ dapls_ib_pd_alloc ( * */ DAT_RETURN -dapls_ib_pd_free ( - IN DAPL_PZ *pz ) +dapls_ib_pd_free(IN DAPL_PZ *pz ) { if (pz->pd_handle != IB_INVALID_HANDLE) { +#if 1 + ibv_dealloc_pd(pz->pd_handle); + pz->pd_handle = IB_INVALID_HANDLE; + return DAT_SUCCESS; +#else if (ibv_dealloc_pd(pz->pd_handle)) return(dapl_convert_errno(errno,"dealloc_pd")); pz->pd_handle = IB_INVALID_HANDLE; +#endif } return DAT_SUCCESS; } @@ -165,28 +174,37 @@ dapls_ib_pd_free ( * */ DAT_RETURN -dapls_ib_mr_register ( - IN DAPL_IA *ia_ptr, - IN DAPL_LMR *lmr, - IN DAT_PVOID virt_addr, - IN DAT_VLEN length, - IN DAT_MEM_PRIV_FLAGS privileges) +dapls_ib_mr_register(IN DAPL_IA *ia_ptr, + IN DAPL_LMR *lmr, + IN DAT_PVOID virt_addr, + IN DAT_VLEN length, + IN DAT_MEM_PRIV_FLAGS privileges, + IN DAT_VA_TYPE va_type) { ib_pd_handle_t ib_pd_handle; + struct ibv_device *ibv_dev = ia_ptr->hca_ptr->ib_hca_handle->device; ib_pd_handle = ((DAPL_PZ *)lmr->param.pz_handle)->pd_handle; - dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, - " mr_register: ia=%p, lmr=%p va=%p ln=%d pv=0x%x\n", - ia_ptr, lmr, virt_addr, length, privileges ); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " mr_register: ia=%p, lmr=%p va=%p ln=%d pv=0x%x\n", + ia_ptr, lmr, virt_addr, length, privileges ); /* TODO: shared memory */ if (lmr->param.mem_type == DAT_MEM_TYPE_SHARED_VIRTUAL) { - dapl_dbg_log( DAPL_DBG_TYPE_ERR, - " mr_register_shared: NOT IMPLEMENTED\n"); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mr_register_shared: NOT IMPLEMENTED\n"); return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } + /* iWARP only support */ + if ((va_type == DAT_VA_TYPE_ZB) && + (ibv_dev->transport_type != IBV_TRANSPORT_IWARP)) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " va_type == DAT_VA_TYPE_ZB: NOT SUPPORTED\n"); + return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + } + /* local read is default on IB */ lmr->mr_handle = ibv_reg_mr(((DAPL_PZ *)lmr->param.pz_handle)->pd_handle, @@ -200,16 +218,16 @@ dapls_ib_mr_register ( lmr->param.lmr_context = lmr->mr_handle->lkey; lmr->param.rmr_context = lmr->mr_handle->rkey; lmr->param.registered_size = length; - lmr->param.registered_address = (DAT_VADDR)(uintptr_t) virt_addr; + lmr->param.registered_address = (DAT_VADDR)(uintptr_t)virt_addr; - dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, - " mr_register: mr=%p h %x pd %p ctx %p ,lkey=0x%x, rkey=0x%x priv=%x\n", - lmr->mr_handle, lmr->mr_handle->handle, - lmr->mr_handle->pd, - lmr->mr_handle->context, - lmr->mr_handle->lkey, - lmr->mr_handle->rkey, - length, dapls_convert_privileges(privileges) ); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " mr_register: mr=%p addr=%p h %x pd %p ctx %p " + "lkey=0x%x rkey=0x%x priv=%x\n", + lmr->mr_handle, lmr->mr_handle->addr, + lmr->mr_handle->handle, + lmr->mr_handle->pd, lmr->mr_handle->context, + lmr->mr_handle->lkey, lmr->mr_handle->rkey, + length, dapls_convert_privileges(privileges)); return DAT_SUCCESS; } @@ -231,8 +249,7 @@ dapls_ib_mr_register ( * */ DAT_RETURN -dapls_ib_mr_deregister ( - IN DAPL_LMR *lmr ) +dapls_ib_mr_deregister(IN DAPL_LMR *lmr) { if (lmr->mr_handle != IB_INVALID_HANDLE) { if (ibv_dereg_mr(lmr->mr_handle)) @@ -251,8 +268,8 @@ dapls_ib_mr_deregister ( * Input: * ia_ptr IA handle * lmr pointer to dapl_lmr struct - * virt_addr virtual address of beginning of mem region - * length length of memory region + * privileges + * va_type * * Output: * none @@ -263,13 +280,15 @@ dapls_ib_mr_deregister ( * */ DAT_RETURN -dapls_ib_mr_register_shared ( - IN DAPL_IA *ia_ptr, - IN DAPL_LMR *lmr, - IN DAT_MEM_PRIV_FLAGS privileges ) +dapls_ib_mr_register_shared(IN DAPL_IA *ia_ptr, + IN DAPL_LMR *lmr, + IN DAT_MEM_PRIV_FLAGS privileges, + IN DAT_VA_TYPE va_type) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mr_register_shared: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mr_register_shared: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* @@ -289,12 +308,13 @@ dapls_ib_mr_register_shared ( * */ DAT_RETURN -dapls_ib_mw_alloc ( - IN DAPL_RMR *rmr ) +dapls_ib_mw_alloc (IN DAPL_RMR *rmr) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_alloc: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_alloc: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* @@ -314,11 +334,12 @@ dapls_ib_mw_alloc ( * */ DAT_RETURN -dapls_ib_mw_free ( - IN DAPL_RMR *rmr ) +dapls_ib_mw_free(IN DAPL_RMR *rmr) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_free: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_free: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* @@ -339,17 +360,18 @@ dapls_ib_mw_free ( * */ DAT_RETURN -dapls_ib_mw_bind ( - IN DAPL_RMR *rmr, - IN DAPL_LMR *lmr, - IN DAPL_EP *ep, - IN DAPL_COOKIE *cookie, - IN DAT_VADDR virtual_address, - IN DAT_VLEN length, - IN DAT_MEM_PRIV_FLAGS mem_priv, - IN DAT_BOOLEAN is_signaled) +dapls_ib_mw_bind(IN DAPL_RMR *rmr, + IN DAPL_LMR *lmr, + IN DAPL_EP *ep, + IN DAPL_COOKIE *cookie, + IN DAT_VADDR virtual_address, + IN DAT_VLEN length, + IN DAT_MEM_PRIV_FLAGS mem_priv, + IN DAT_BOOLEAN is_signaled) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_bind: NOT IMPLEMENTED\n"); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_bind: NOT IMPLEMENTED\n"); + return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } @@ -372,14 +394,15 @@ dapls_ib_mw_bind ( * */ DAT_RETURN -dapls_ib_mw_unbind ( - IN DAPL_RMR *rmr, - IN DAPL_EP *ep, - IN DAPL_COOKIE *cookie, - IN DAT_BOOLEAN is_signaled ) +dapls_ib_mw_unbind(IN DAPL_RMR *rmr, + IN DAPL_EP *ep, + IN DAPL_COOKIE *cookie, + IN DAT_BOOLEAN is_signaled ) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_unbind: NOT IMPLEMENTED\n"); - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " mw_unbind: NOT IMPLEMENTED\n"); + + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); } /* diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index 3a1e3c8..1eba2bd 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -110,15 +110,23 @@ dapls_ib_qp_alloc ( /* Setup attributes and create qp */ dapl_os_memzero((void*)&qp_create, sizeof(qp_create)); qp_create.send_cq = req_cq; - qp_create.recv_cq = rcv_cq; qp_create.cap.max_send_wr = attr->max_request_dtos; - qp_create.cap.max_recv_wr = attr->max_recv_dtos; qp_create.cap.max_send_sge = attr->max_request_iov; - qp_create.cap.max_recv_sge = attr->max_recv_iov; qp_create.cap.max_inline_data = ia_ptr->hca_ptr->ib_trans.max_inline_send; qp_create.qp_type = IBV_QPT_RC; qp_create.qp_context = (void*)ep_ptr; + /* ibv assumes rcv_cq is never NULL, set to req_cq */ + if (rcv_cq == NULL) { + qp_create.recv_cq = req_cq; + qp_create.cap.max_recv_wr = 0; + qp_create.cap.max_recv_sge = 0; + } else { + qp_create.recv_cq = rcv_cq; + qp_create.cap.max_recv_wr = attr->max_recv_dtos; + qp_create.cap.max_recv_sge = attr->max_recv_iov; + } + ep_ptr->qp_handle = ibv_create_qp( ib_pd_handle, &qp_create); if (!ep_ptr->qp_handle) return(dapl_convert_errno(ENOMEM, "create_qp")); @@ -298,9 +306,10 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IN ib_qp_state_t qp_state, IN ib_qp_cm_t *qp_cm ) { - struct ibv_qp_attr qp_attr; + struct ibv_qp_attr qp_attr; enum ibv_qp_attr_mask mask = IBV_QP_STATE; - DAPL_EP *ep_ptr = (DAPL_EP*)qp_handle->qp_context; + DAPL_EP *ep_ptr = (DAPL_EP*)qp_handle->qp_context; + DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; dapl_os_memzero((void*)&qp_attr, sizeof(qp_attr)); qp_attr.qp_state = qp_state; @@ -315,14 +324,16 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER; + qp_attr.qp_state = IBV_QPS_RTR; - qp_attr.path_mtu = IBV_MTU_1024; + qp_attr.path_mtu = IBV_MTU_2048; qp_attr.dest_qp_num = qp_cm->qpn; qp_attr.rq_psn = 1; qp_attr.max_dest_rd_atomic = ep_ptr->param.ep_attr.max_rdma_read_out; - qp_attr.min_rnr_timer = 12; + qp_attr.min_rnr_timer = ia_ptr->hca_ptr->ib_trans.rnr_timer; qp_attr.ah_attr.is_global = 0; + qp_attr.ah_attr.grh.dgid = qp_cm->gid; qp_attr.ah_attr.dlid = qp_cm->lid; qp_attr.ah_attr.sl = 0; qp_attr.ah_attr.src_path_bits = 0; @@ -343,30 +354,25 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC; + qp_attr.qp_state = IBV_QPS_RTS; - qp_attr.timeout = 14; - qp_attr.retry_cnt = 7; - qp_attr.rnr_retry = 7; + qp_attr.timeout = ia_ptr->hca_ptr->ib_trans.ack_timer; + qp_attr.retry_cnt = ia_ptr->hca_ptr->ib_trans.ack_retry; + qp_attr.rnr_retry = ia_ptr->hca_ptr->ib_trans.rnr_retry; qp_attr.sq_psn = 1; qp_attr.max_rd_atomic = ep_ptr->param.ep_attr.max_rdma_read_out; - dapl_dbg_log (DAPL_DBG_TYPE_EP, - " modify_qp_rts: psn %x rd_atomic %d\n", - qp_attr.sq_psn, qp_attr.max_rd_atomic ); + dapl_dbg_log(DAPL_DBG_TYPE_EP, + " modify_qp_rts: psn %x rd_atomic %d ack %d " + " retry %d rnr_retry %d\n", + qp_attr.sq_psn, qp_attr.max_rd_atomic, + qp_attr.timeout, qp_attr.retry_cnt, + qp_attr.rnr_retry ); break; } case IBV_QPS_INIT: { - DAPL_IA *ia_ptr; - DAPL_EP *ep_ptr; - /* need to find way back to port num */ - ep_ptr = (DAPL_EP*)qp_handle->qp_context; - if (ep_ptr) - ia_ptr = ep_ptr->header.owner_ia; - else - break; - mask |= IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS; @@ -377,7 +383,8 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ | - IBV_ACCESS_REMOTE_ATOMIC; + IBV_ACCESS_REMOTE_ATOMIC | + IBV_ACCESS_MW_BIND; dapl_dbg_log (DAPL_DBG_TYPE_EP, " modify_qp_init: pi %x port %x acc %x\n", diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index 5e34b47..9bf0e2b 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -62,6 +62,7 @@ static const char rcsid[] = "$Id: $"; #include int g_dapl_loopback_connection = 0; +int g_scm_pipe[2]; /* just get IP address for hostname */ DAT_RETURN getipaddr( char *addr, int addr_len) @@ -70,14 +71,14 @@ DAT_RETURN getipaddr( char *addr, int addr_len) struct hostent *h_ptr; struct utsname ourname; - if ( uname( &ourname ) < 0 ) + if (uname( &ourname ) < 0) return DAT_INTERNAL_ERROR; - h_ptr = gethostbyname( ourname.nodename ); - if ( h_ptr == NULL ) + h_ptr = gethostbyname(ourname.nodename); + if (h_ptr == NULL) return DAT_INTERNAL_ERROR; - if ( h_ptr->h_addrtype == AF_INET ) { + if (h_ptr->h_addrtype == AF_INET) { int i; struct in_addr **alist = (struct in_addr **)h_ptr->h_addr_list; @@ -87,18 +88,17 @@ DAT_RETURN getipaddr( char *addr, int addr_len) /* Walk the list of addresses for host */ for (i=0; alist[i] != NULL; i++) { - - /* first non-loopback address */ - if ( *(uint32_t*)alist[i] != htonl(0x7f000001) ) { - dapl_os_memcpy( &ipv4_addr->sin_addr, - h_ptr->h_addr_list[i], - 4 ); + /* first non-loopback address */ + if (*(uint32_t*)alist[i] != htonl(0x7f000001)) { + dapl_os_memcpy(&ipv4_addr->sin_addr, + h_ptr->h_addr_list[i], + 4); break; } } /* if no acceptable address found */ if (*(uint32_t*)&ipv4_addr->sin_addr == 0) - return DAT_INVALID_ADDRESS; + return DAT_INVALID_ADDRESS; } else return DAT_INVALID_ADDRESS; @@ -122,6 +122,10 @@ DAT_RETURN getipaddr( char *addr, int addr_len) */ int32_t dapls_ib_init (void) { + /* create pipe for waking up thread */ + if (pipe(g_scm_pipe)) + return 1; + return 0; } @@ -156,7 +160,7 @@ DAT_RETURN dapls_ib_open_hca ( int i; DAT_RETURN dat_status = DAT_SUCCESS; - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " open_hca: %s - %p\n", hca_name, hca_ptr ); /* Get list of all IB devices, find match, open */ @@ -170,65 +174,83 @@ DAT_RETURN dapls_ib_open_hca ( for (i = 0; dev_list[i]; ++i) { hca_ptr->ib_trans.ib_dev = dev_list[i]; - if (!strcmp(ibv_get_device_name(hca_ptr->ib_trans.ib_dev),hca_name)) + if (!strcmp(ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + hca_name)) goto found; } - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: IB device %s not found\n", - hca_name); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: device %s not found\n", + hca_name); goto err; found: - dapl_dbg_log (DAPL_DBG_TYPE_UTIL," open_hca: Found dev %s %016llx\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), - (unsigned long long)bswap_64(ibv_get_device_guid(hca_ptr->ib_trans.ib_dev))); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL," open_hca: Found dev %s %016llx\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + (unsigned long long) + bswap_64(ibv_get_device_guid(hca_ptr->ib_trans.ib_dev))); hca_ptr->ib_hca_handle = ibv_open_device(hca_ptr->ib_trans.ib_dev); if (!hca_ptr->ib_hca_handle) { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: IB dev open failed for %s\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev) ); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: dev open failed for %s, err=%s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + strerror(errno)); goto err; } - /* set inline max with enviroment or default */ + /* set RC tunables via enviroment or default */ hca_ptr->ib_trans.max_inline_send = - dapl_os_get_env_val ( "DAPL_MAX_INLINE", INLINE_SEND_DEFAULT ); + dapl_os_get_env_val("DAPL_MAX_INLINE", INLINE_SEND_DEFAULT); + hca_ptr->ib_trans.ack_retry = + dapl_os_get_env_val("DAPL_ACK_RETRY", SCM_ACK_RETRY); + hca_ptr->ib_trans.ack_timer = + dapl_os_get_env_val("DAPL_ACK_TIMER", SCM_ACK_TIMER); + hca_ptr->ib_trans.rnr_retry = + dapl_os_get_env_val("DAPL_RNR_RETRY", SCM_RNR_RETRY); + hca_ptr->ib_trans.rnr_timer = + dapl_os_get_env_val("DAPL_RNR_TIMER", SCM_RNR_TIMER); /* initialize cq_lock */ dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.cq_lock); - if (dat_status != DAT_SUCCESS) - { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: failed to init cq_lock\n"); + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: failed to init cq_lock\n"); goto bail; } /* EVD events without direct CQ channels, non-blocking */ hca_ptr->ib_trans.ib_cq = ibv_create_comp_channel(hca_ptr->ib_hca_handle); + if (hca_ptr->ib_trans.ib_cq == NULL) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: ibv_create_comp_channel ERR %s\n", + strerror(errno)); + goto bail; + } + opts = fcntl(hca_ptr->ib_trans.ib_cq->fd, F_GETFL); /* uCQ */ if (opts < 0 || fcntl(hca_ptr->ib_trans.ib_cq->fd, F_SETFL, opts | O_NONBLOCK) < 0) { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: ERR with CQ FD\n" ); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: fcntl on ib_cq->fd %d ERR %d %s\n", + hca_ptr->ib_trans.ib_cq->fd, opts, + strerror(errno)); goto bail; } if (dapli_cq_thread_init(hca_ptr)) { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: cq_thread_init failed for %s\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev) ); + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: cq_thread_init failed for %s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev)); goto bail; } /* initialize cr_list lock */ dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.lock); - if (dat_status != DAT_SUCCESS) - { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: failed to init lock\n"); + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: failed to init cr_list lock\n"); goto bail; } @@ -240,10 +262,9 @@ found: dat_status = dapl_os_thread_create(cr_thread, (void*)hca_ptr, &hca_ptr->ib_trans.thread ); - if (dat_status != DAT_SUCCESS) - { - dapl_dbg_log (DAPL_DBG_TYPE_ERR, - " open_hca: failed to create thread\n"); + if (dat_status != DAT_SUCCESS) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: failed to create thread\n"); goto bail; } @@ -251,7 +272,7 @@ found: while (hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) { struct timespec sleep, remain; sleep.tv_sec = 0; - sleep.tv_nsec = 20000000; /* 20 ms */ + sleep.tv_nsec = 2000000; /* 2 ms */ dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " open_hca: waiting for cr_thread\n"); nanosleep (&sleep, &remain); @@ -259,16 +280,15 @@ found: /* get the IP address of the device */ dat_status = getipaddr((char*)&hca_ptr->hca_address, - sizeof(DAT_SOCK_ADDR6) ); - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, - " open_hca: %s, port %d, %s %d.%d.%d.%d\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), hca_ptr->port_num, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_family == AF_INET ? "AF_INET":"AF_INET6", - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 0 & 0xff, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 8 & 0xff, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 16 & 0xff, - ((struct sockaddr_in *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff ); - + sizeof(DAT_SOCK_ADDR6)); + + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " open_hca: devname %s, port %d, hostname_IP %s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + hca_ptr->port_num, + inet_ntoa(((struct sockaddr_in *) + &hca_ptr->hca_address)->sin_addr)); + ibv_free_device_list(dev_list); return dat_status; @@ -308,15 +328,15 @@ DAT_RETURN dapls_ib_close_hca ( IN DAPL_HCA *hca_ptr ) return(dapl_convert_errno(errno,"ib_close_device")); hca_ptr->ib_hca_handle = IB_INVALID_HANDLE; } - dapl_os_lock_destroy(&hca_ptr->ib_trans.cq_lock); /* destroy cr_thread and lock */ hca_ptr->ib_trans.cr_state = IB_THREAD_CANCEL; + write(g_scm_pipe[1], "w", sizeof "w"); while (hca_ptr->ib_trans.cr_state != IB_THREAD_EXIT) { struct timespec sleep, remain; sleep.tv_sec = 0; - sleep.tv_nsec = 20000000; /* 20 ms */ + sleep.tv_nsec = 2000000; /* 2 ms */ dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " close_hca: waiting for cr_thread\n"); nanosleep (&sleep, &remain); @@ -378,14 +398,11 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->vendor_name[DAT_NAME_MAX_LENGTH - 1] = '\0'; ia_attr->ia_address_ptr = (DAT_IA_ADDRESS_PTR)&hca_ptr->hca_address; - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, - " query_hca: %s %s %d.%d.%d.%d\n", - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_family == AF_INET ? "AF_INET":"AF_INET6", - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 0 & 0xff, - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 8 & 0xff, - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 16 & 0xff, - ((struct sockaddr_in *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 24 & 0xff ); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " query_hca: %s %s \n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + inet_ntoa(((struct sockaddr_in *) + &hca_ptr->hca_address)->sin_addr)); ia_attr->hardware_version_major = dev_attr.hw_ver; /* ia_attr->hardware_version_minor = dev_attr.fw_ver; */ @@ -408,10 +425,21 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->max_pzs = dev_attr.max_pd; ia_attr->max_mtu_size = port_attr.max_msg_sz; ia_attr->max_rdma_size = port_attr.max_msg_sz; + ia_attr->max_iov_segments_per_rdma_read = dev_attr.max_sge; + ia_attr->max_iov_segments_per_rdma_write = dev_attr.max_sge; ia_attr->num_transport_attr = 0; ia_attr->transport_attr = NULL; ia_attr->num_vendor_attr = 0; ia_attr->vendor_attr = NULL; + ia_attr->max_iov_segments_per_rdma_read = dev_attr.max_sge; + ia_attr->max_iov_segments_per_rdma_write = dev_attr.max_sge; +#ifdef DAT_EXTENSIONS + ia_attr->extension_supported = DAT_EXTENSION_IB; + ia_attr->extension_version = DAT_IB_EXTENSION_VERSION; +#endif + hca_ptr->ib_trans.ack_timer = + DAPL_MAX(dev_attr.local_ca_ack_delay, + hca_ptr->ib_trans.ack_timer); dapl_dbg_log (DAPL_DBG_TYPE_UTIL, " query_hca: (%x.%x) ep %d ep_q %d evd %d evd_q %d\n", @@ -420,11 +448,10 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->max_eps, ia_attr->max_dto_per_ep, ia_attr->max_evds, ia_attr->max_evd_qlen ); dapl_dbg_log (DAPL_DBG_TYPE_UTIL, - " query_hca: msg %llu rdma %llu iov %d lmr %d rmr %d\n", + " query_hca: msg %llu rdma %llu iov %d lmr %d rmr %d ack_time %d\n", ia_attr->max_mtu_size, ia_attr->max_rdma_size, ia_attr->max_iov_segments_per_dto, ia_attr->max_lmrs, - ia_attr->max_rmrs ); - + ia_attr->max_rmrs,hca_ptr->ib_trans.ack_timer ); } if (ep_attr != NULL) { @@ -443,7 +470,6 @@ DAT_RETURN dapls_ib_query_hca ( ep_attr->max_recv_dtos, ep_attr->max_recv_iov, ep_attr->max_rdma_read_in, ep_attr->max_rdma_read_out); } - return DAT_SUCCESS; } @@ -508,3 +534,41 @@ DAT_RETURN dapls_ib_setup_async_callback ( return DAT_SUCCESS; } +/* + * dapls_set_provider_specific_attr + * + * Input: + * attr_ptr Pointer provider specific attributes + * + * Output: + * none + * + * Returns: + * void + */ +DAT_NAMED_ATTR ib_attrs[] = { +#ifdef DAT_EXTENSIONS + { + "DAT_EXTENSION_INTERFACE", "TRUE" + }, + { + DAT_IB_ATTR_FETCH_AND_ADD, "TRUE" + }, + { + DAT_IB_ATTR_CMP_AND_SWAP, "TRUE" + }, + { + DAT_IB_ATTR_IMMED_DATA, "TRUE" + }, +#endif +}; + +#define SPEC_ATTR_SIZE( x ) (sizeof( x ) / sizeof( DAT_NAMED_ATTR)) + +void dapls_query_provider_specific_attr( + IN DAT_PROVIDER_ATTR *attr_ptr ) +{ + attr_ptr->num_provider_specific_attr = SPEC_ATTR_SIZE(ib_attrs); + attr_ptr->provider_specific_attr = ib_attrs; +} + diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index 0d928df..fd9cd2f 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -71,15 +71,18 @@ typedef ib_hca_handle_t dapl_ibal_ca_t; /* CM mappings, user CM not complete use SOCKETS */ -/* destination info to exchange until real IB CM shows up */ +/* destination info to exchange, define wire protocol version */ +#define DSCM_VER 2 typedef struct _ib_qp_cm { - uint32_t qpn; + uint16_t ver; + uint16_t rej; uint16_t lid; uint16_t port; - int p_size; + uint32_t qpn; + uint32_t p_size; DAT_SOCK_ADDR6 ia_address; - + union ibv_gid gid; } ib_qp_cm_t; /* @@ -94,20 +97,34 @@ struct ib_llist_entry struct dapl_llist_entry *list_head; }; +typedef enum scm_state +{ + SCM_INIT, + SCM_LISTEN, + SCM_CONN_PENDING, + SCM_ACCEPTING, + SCM_ACCEPTED, + SCM_REJECTED, + SCM_CONNECTED, + SCM_DISCONNECTED, + SCM_DESTROY +} SCM_STATE; + struct ib_cm_handle { struct ib_llist_entry entry; + DAPL_OS_LOCK lock; + SCM_STATE state; int socket; - int l_socket; - struct dapl_hca *hca_ptr; - DAT_HANDLE cr; + struct dapl_hca *hca; DAT_HANDLE sp; + struct dapl_ep *ep; ib_qp_cm_t dst; unsigned char p_data[256]; }; -typedef struct ib_cm_handle *ib_cm_handle_t; -typedef ib_cm_handle_t ib_cm_srvc_handle_t; +typedef struct ib_cm_handle *dp_ib_cm_handle_t; +typedef dp_ib_cm_handle_t ib_cm_srvc_handle_t; DAT_RETURN getipaddr(char *addr, int addr_len); @@ -163,6 +180,12 @@ typedef struct ibv_comp_channel *ib_wait_obj_handle_t; /* inline send rdma threshold */ #define INLINE_SEND_DEFAULT 128 +/* RC timer - retry count defaults */ +#define SCM_ACK_TIMER 15 /* 5 bits, 4.096us*2^ack_timer. 15 == 134ms */ +#define SCM_ACK_RETRY 7 /* 3 bits, 7 * 134ms = 940ms */ +#define SCM_RNR_TIMER 28 /* 5 bits, 28 == 163ms, 31 == 491ms */ +#define SCM_RNR_RETRY 7 /* 3 bits, 7 == infinite */ + /* CM private data areas */ #define IB_MAX_REQ_PDATA_SIZE 92 #define IB_MAX_REP_PDATA_SIZE 196 @@ -268,7 +291,10 @@ typedef struct _ib_hca_transport ib_async_cq_handler_t async_cq_error; ib_async_dto_handler_t async_cq; ib_async_qp_handler_t async_qp_error; - + uint8_t ack_timer; + uint8_t ack_retry; + uint8_t rnr_timer; + uint8_t rnr_retry; } ib_hca_transport_t; /* provider specfic fields for shared memory support */ diff --git a/dapl/udapl/libdaploscm.map b/dapl/udapl/libdaploscm.map new file mode 100755 index 0000000..45edb4d --- /dev/null +++ b/dapl/udapl/libdaploscm.map @@ -0,0 +1,7 @@ +DAPL_SCM_2.0 { + global: + dat_provider_fini; + dat_provider_init; + dapl_extensions; + local: *; +}; -- 1.5.2.5 From hrosenstock at xsigo.com Fri Jun 20 11:59:09 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 11:59:09 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_sw_info_rcv.c: Cosmetic formatting changes Message-ID: <1213988350.13056.204.camel@hrosenstock-ws.xsigo.com> opensm/osm_sw_info_rcv.c: Cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sw_info_rcv.c b/opensm/opensm/osm_sw_info_rcv.c index d5b28e3..e047e7b 100644 --- a/opensm/opensm/osm_sw_info_rcv.c +++ b/opensm/opensm/osm_sw_info_rcv.c @@ -304,8 +304,7 @@ __osm_si_rcv_process_new(IN osm_sm_t * sm, /* set subnet max mlid to the minimum MulticastFDBCap of all switches */ if (p_sw->mcast_tbl.max_mlid_ho < sm->p_subn->max_multicast_lid_ho) { - sm->p_subn->max_multicast_lid_ho = - p_sw->mcast_tbl.max_mlid_ho; + sm->p_subn->max_multicast_lid_ho = p_sw->mcast_tbl.max_mlid_ho; OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Subnet max multicast lid is 0x%X\n", sm->p_subn->max_multicast_lid_ho); @@ -423,9 +422,9 @@ __osm_si_rcv_process_existing(IN osm_sm_t * sm, is_change_detected = TRUE; } else { /* - If something changed, then just signal the state - manager. Don't attempt to probe further during - a light sweep. + If something changed, then just signal the + state manager. Don't attempt to probe + further during a light sweep. */ if (ib_switch_info_get_state_change(p_si)) { osm_dump_switch_info(sm->p_log, p_si, @@ -483,7 +482,6 @@ void osm_si_rcv_process(IN void *context, IN void *data) /* Acquire the switch object and add the switch info. */ - p_context = osm_madw_get_si_context_ptr(p_madw); node_guid = p_context->node_guid; From arlin.r.davis at intel.com Fri Jun 20 11:59:25 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 11:59:25 -0700 Subject: [ofa-general] [PATCH 5/7][v2.0] dapl scm: support global routing and set mtu based on active_mtu Message-ID: Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_scm/dapl_ib_qp.c | 42 +++++++++++++++++++++------------------ dapl/openib_scm/dapl_ib_util.c | 7 ++++++ dapl/openib_scm/dapl_ib_util.h | 11 +++++++++- 3 files changed, 40 insertions(+), 20 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c index 1eba2bd..8577131 100644 --- a/dapl/openib_scm/dapl_ib_qp.c +++ b/dapl/openib_scm/dapl_ib_qp.c @@ -25,9 +25,9 @@ /********************************************************************** * - * MODULE: dapl_det_qp.c + * MODULE: dapl_ib_qp.c * - * PURPOSE: QP routines for access to DET Verbs + * PURPOSE: QP routines for access to ofa rdma verbs * * $Id: $ **********************************************************************/ @@ -285,17 +285,12 @@ void dapls_ib_reinit_ep ( IN DAPL_EP *ep_ptr) { - if ( ep_ptr->qp_handle != IB_INVALID_HANDLE ) { /* move to RESET state and then to INIT */ dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_RESET, 0); dapls_modify_qp_state(ep_ptr->qp_handle, IBV_QPS_INIT, 0); ep_ptr->qp_state = IB_QP_STATE_INIT; } - - /* TODO: When IB-CM is implement then handle timewait before - * allowing re-use of this QP - */ } /* @@ -325,19 +320,28 @@ dapls_modify_qp_state ( IN ib_qp_handle_t qp_handle, IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER; - qp_attr.qp_state = IBV_QPS_RTR; - qp_attr.path_mtu = IBV_MTU_2048; - qp_attr.dest_qp_num = qp_cm->qpn; - qp_attr.rq_psn = 1; - qp_attr.max_dest_rd_atomic = + qp_attr.qp_state = IBV_QPS_RTR; + qp_attr.dest_qp_num = qp_cm->qpn; + qp_attr.rq_psn = 1; + qp_attr.path_mtu = + ia_ptr->hca_ptr->ib_trans.mtu; + qp_attr.max_dest_rd_atomic = ep_ptr->param.ep_attr.max_rdma_read_out; - qp_attr.min_rnr_timer = ia_ptr->hca_ptr->ib_trans.rnr_timer; - qp_attr.ah_attr.is_global = 0; - qp_attr.ah_attr.grh.dgid = qp_cm->gid; - qp_attr.ah_attr.dlid = qp_cm->lid; - qp_attr.ah_attr.sl = 0; - qp_attr.ah_attr.src_path_bits = 0; - qp_attr.ah_attr.port_num = qp_cm->port; + qp_attr.min_rnr_timer = + ia_ptr->hca_ptr->ib_trans.rnr_timer; + qp_attr.ah_attr.dlid = qp_cm->lid; + /* global routing */ + if (ia_ptr->hca_ptr->ib_trans.global) { + qp_attr.ah_attr.is_global = 1; + qp_attr.ah_attr.grh.dgid = qp_cm->gid; + qp_attr.ah_attr.grh.hop_limit = + ia_ptr->hca_ptr->ib_trans.hop_limit; + qp_attr.ah_attr.grh.traffic_class = + ia_ptr->hca_ptr->ib_trans.tclass; + } + qp_attr.ah_attr.sl = 0; + qp_attr.ah_attr.src_path_bits = 0; + qp_attr.ah_attr.port_num = qp_cm->port; dapl_dbg_log (DAPL_DBG_TYPE_EP, " modify_qp_rtr: qpn %x lid %x " diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index 9bf0e2b..9ce9e6a 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -210,6 +210,12 @@ found: dapl_os_get_env_val("DAPL_RNR_RETRY", SCM_RNR_RETRY); hca_ptr->ib_trans.rnr_timer = dapl_os_get_env_val("DAPL_RNR_TIMER", SCM_RNR_TIMER); + hca_ptr->ib_trans.global = + dapl_os_get_env_val("DAPL_GLOBAL_ROUTING", SCM_GLOBAL); + hca_ptr->ib_trans.hop_limit = + dapl_os_get_env_val("DAPL_HOP_LIMIT", SCM_HOP_LIMIT); + hca_ptr->ib_trans.tclass = + dapl_os_get_env_val("DAPL_TCLASS", SCM_TCLASS); /* initialize cq_lock */ dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.cq_lock); @@ -437,6 +443,7 @@ DAT_RETURN dapls_ib_query_hca ( ia_attr->extension_supported = DAT_EXTENSION_IB; ia_attr->extension_version = DAT_IB_EXTENSION_VERSION; #endif + hca_ptr->ib_trans.mtu = port_attr.active_mtu; hca_ptr->ib_trans.ack_timer = DAPL_MAX(dev_attr.local_ca_ack_delay, hca_ptr->ib_trans.ack_timer); diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index fd9cd2f..39eb245 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -186,6 +186,11 @@ typedef struct ibv_comp_channel *ib_wait_obj_handle_t; #define SCM_RNR_TIMER 28 /* 5 bits, 28 == 163ms, 31 == 491ms */ #define SCM_RNR_RETRY 7 /* 3 bits, 7 == infinite */ +/* Global routing defaults */ +#define SCM_GLOBAL 0 /* global routing is disabled */ +#define SCM_HOP_LIMIT 0xff +#define SCM_TCLASS 0 + /* CM private data areas */ #define IB_MAX_REQ_PDATA_SIZE 92 #define IB_MAX_REP_PDATA_SIZE 196 @@ -193,7 +198,7 @@ typedef struct ibv_comp_channel *ib_wait_obj_handle_t; #define IB_MAX_DREQ_PDATA_SIZE 220 #define IB_MAX_DREP_PDATA_SIZE 224 -/* DTO OPs, ordered for DAPL ENUM definitions ???*/ +/* DTO OPs, ordered for DAPL ENUM definitions */ #define OP_RDMA_WRITE IBV_WR_RDMA_WRITE #define OP_RDMA_WRITE_IMM IBV_WR_RDMA_WRITE_WITH_IMM #define OP_SEND IBV_WR_SEND @@ -295,6 +300,10 @@ typedef struct _ib_hca_transport uint8_t ack_retry; uint8_t rnr_timer; uint8_t rnr_retry; + uint8_t global; + uint8_t hop_limit; + uint8_t tclass; + uint8_t mtu; } ib_hca_transport_t; /* provider specfic fields for shared memory support */ -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 12:00:31 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 12:00:31 -0700 Subject: [ofa-general] [PATCH 6/7][v2.0] dapl scm: add stdout logging for uname and gethostbyname errors during open. Message-ID: Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_scm/dapl_ib_util.c | 11 +++++++++-- 1 files changed, 9 insertions(+), 2 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index 9ce9e6a..362710d 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -71,12 +71,19 @@ DAT_RETURN getipaddr( char *addr, int addr_len) struct hostent *h_ptr; struct utsname ourname; - if (uname( &ourname ) < 0) + if (uname(&ourname) < 0) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: uname err=%s\n", strerror(errno)); return DAT_INTERNAL_ERROR; + } h_ptr = gethostbyname(ourname.nodename); - if (h_ptr == NULL) + if (h_ptr == NULL) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: gethostbyname err=%s\n", + strerror(errno)); return DAT_INTERNAL_ERROR; + } if (h_ptr->h_addrtype == AF_INET) { int i; -- 1.5.2.5 From arlin.r.davis at intel.com Fri Jun 20 12:10:39 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Fri, 20 Jun 2008 12:10:39 -0700 Subject: [ofa-general] [PATCH 7/7][v2.0] dapl scm: fix ibv_destroy_cq busy error condition during dat_evd_free Message-ID: Problem surfaced while running Intel MPI 3.1 and mpich2-test suite. dapli_destroy_cm was incorrectly removing reference to qp_handle in endpoint and destroy_cq and destroy_pd code was ignoring verbs errors. Signed-off by: Arlin Davis ardavis at ichips.intel.com --- dapl/openib_scm/dapl_ib_cm.c | 4 +--- dapl/openib_scm/dapl_ib_cq.c | 13 ++++--------- dapl/openib_scm/dapl_ib_mem.c | 8 +------- 3 files changed, 6 insertions(+), 19 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index 9b30625..b87c060 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -108,10 +108,8 @@ static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) dapl_os_lock(&cm_ptr->lock); cm_ptr->state = SCM_DESTROY; - if (cm_ptr->ep) { + if (cm_ptr->ep) cm_ptr->ep->cm_handle = IB_INVALID_HANDLE; - cm_ptr->ep->qp_handle = IB_INVALID_HANDLE; - } /* close socket if still active */ if (cm_ptr->socket >= 0) { diff --git a/dapl/openib_scm/dapl_ib_cq.c b/dapl/openib_scm/dapl_ib_cq.c index 56b729e..7d6bd4f 100644 --- a/dapl/openib_scm/dapl_ib_cq.c +++ b/dapl/openib_scm/dapl_ib_cq.c @@ -423,20 +423,15 @@ DAT_RETURN dapls_ib_cq_free ( IN DAPL_EVD *evd_ptr) { DAT_EVENT event; + ib_work_completion_t wc; if (evd_ptr->ib_cq_handle != IB_INVALID_HANDLE) { /* pull off CQ and EVD entries and toss */ - while (dapls_ib_completion_poll == DAT_SUCCESS); - while (dapl_evd_dequeue(evd_ptr,&event) != DAT_QUEUE_EMPTY); -#if 1 - ibv_destroy_cq(evd_ptr->ib_cq_handle); - evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; - return DAT_SUCCESS; -#else + while (ibv_poll_cq(evd_ptr->ib_cq_handle, 1, &wc) == 1); + while (dapl_evd_dequeue(evd_ptr, &event) == DAT_SUCCESS); if (ibv_destroy_cq(evd_ptr->ib_cq_handle)) - return(dapl_convert_errno(errno,"destroy_cq")); + return(dapl_convert_errno(errno,"ibv_destroy_cq")); evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; -#endif } return DAT_SUCCESS; } diff --git a/dapl/openib_scm/dapl_ib_mem.c b/dapl/openib_scm/dapl_ib_mem.c index 5bd5342..54340ed 100644 --- a/dapl/openib_scm/dapl_ib_mem.c +++ b/dapl/openib_scm/dapl_ib_mem.c @@ -141,15 +141,9 @@ DAT_RETURN dapls_ib_pd_free(IN DAPL_PZ *pz ) { if (pz->pd_handle != IB_INVALID_HANDLE) { -#if 1 - ibv_dealloc_pd(pz->pd_handle); - pz->pd_handle = IB_INVALID_HANDLE; - return DAT_SUCCESS; -#else if (ibv_dealloc_pd(pz->pd_handle)) - return(dapl_convert_errno(errno,"dealloc_pd")); + return(dapl_convert_errno(errno,"ibv_dealloc_pd")); pz->pd_handle = IB_INVALID_HANDLE; -#endif } return DAT_SUCCESS; } -- 1.5.2.5 From 8vladimir.goldman at stonybrook.edu Fri Jun 20 10:48:47 2008 From: 8vladimir.goldman at stonybrook.edu (hugues colin) Date: Fri, 20 Jun 2008 17:48:47 +0000 Subject: [ofa-general] world of clocks Message-ID: <000801c8d30c$02b6923e$c8524e85@jynxprjc> " My order arrived yesterday via registered mail in good order THE WATCH IS BEAUTIFUL AND EVEN BETTER THAN I EXPECTED." Try it for yourself - u will be amazed!! - The worlds largest online retailer of luxury products, including: Rolex Sports Models Rolex Datejusts Breitling Cartier Porsche Design Dolce & Gabbana Dior Gucci Hermes Watches Patek Philippe Visit - www.iuejsu.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Fri Jun 20 12:42:31 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 22:42:31 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_sa_mcmember_record.c: Fix some off by 1 issues related to max_multicast_lid_ho In-Reply-To: <1213988259.13056.203.camel@hrosenstock-ws.xsigo.com> References: <1213988259.13056.203.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620194231.GL3158@sashak.voltaire.com> On 11:57 Fri 20 Jun , Hal Rosenstock wrote: > opensm/osm_sa_mcmember_record.c: Fix some off by 1 issues related to > max_multicast_lid_ho > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 20 12:42:58 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Jun 2008 22:42:58 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_sw_info_rcv.c: Cosmetic formatting changes In-Reply-To: <1213988350.13056.204.camel@hrosenstock-ws.xsigo.com> References: <1213988350.13056.204.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080620194258.GM3158@sashak.voltaire.com> On 11:59 Fri 20 Jun , Hal Rosenstock wrote: > opensm/osm_sw_info_rcv.c: Cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From ctung at NetEffect.com Fri Jun 20 13:09:49 2008 From: ctung at NetEffect.com (Chien Tung) Date: Fri, 20 Jun 2008 15:09:49 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statisticsattributes to the RDMA sysfs. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int><5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC08235DA9@venom2> > how do you handle stats if I change routing so an offloaded > connection goes from physical port 1 to physical port 2? Depends on what you mean by changing the route. We can switch a connection between physical ports within our chip. In that case the stats stays with the netdev. > Or presumably you have support for port aggregation in hardware > so an offloaded connection can use both physical ports at the > same time? We do support port aggregation in our silicon. However, a connection is bound to a physical port but switchable (see above). > Actually it seems that you attach a struct ib_device to each > netdev (eg a two-port NIC gets two RDMA devices with iw_nes) > so Steve's scheme should work fine for you too? (I just > looked on a system with a two-port NIC and I do see nes0 and > nes1 devices) After some discussion with our architect, Steve's scheme will work for us by limiting to one netdev per physical port. I will look into implementing the same scheme. Thank you (Steve + Roland) for driving this. Chien From caitlin.bestler at gmail.com Fri Jun 20 13:56:36 2008 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 20 Jun 2008 13:56:36 -0700 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> Message-ID: <469958e00806201356j1460f0bueb10b8374bf15d16@mail.gmail.com> On Thu, Jun 19, 2008 at 8:16 AM, Roland Dreier wrote: > > The "Retransmission count" is per netdev and we support multiple netdev > > per physical port. We have designed our silicon to support stats at > > netdev level and it doesn't make sense for us to aggregate stats from > > all netdevs and report at a device level. Perhaps we can compromise, > > Chelsio can give device level stats at /sys/class/infiniband/$device and > > for NetEffect, we can provide netdev level stats below it? > > Hmm, kind of a mess if we can't achieve some uniformity... > > how do you handle stats if I change routing so an offloaded connection > goes from physical port 1 to physical port 2? Or presumably you have > support for port aggregation in hardware so an offloaded connection can > use both physical ports at the same time? > > Actually it seems that you attach a struct ib_device to each netdev (eg > a two-port NIC gets two RDMA devices with iw_nes) so Steve's scheme > should work fine for you too? (I just looked on a system with a > two-port NIC and I do see nes0 and nes1 devices) > > - R. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From caitlin.bestler at neterion.com Fri Jun 20 14:03:13 2008 From: caitlin.bestler at neterion.com (Caitlin Bestler) Date: Fri, 20 Jun 2008 14:03:13 -0700 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> Message-ID: <469958e00806201403i76cb5841ja39d955c02e958fe@mail.gmail.com> On Thu, Jun 19, 2008 at 8:16 AM, Roland Dreier wrote: > > The "Retransmission count" is per netdev and we support multiple netdev > > per physical port. We have designed our silicon to support stats at > > netdev level and it doesn't make sense for us to aggregate stats from > > all netdevs and report at a device level. Perhaps we can compromise, > > Chelsio can give device level stats at /sys/class/infiniband/$device and > > for NetEffect, we can provide netdev level stats below it? > > Hmm, kind of a mess if we can't achieve some uniformity... > There's an even tougher potential issue on the "uniformity" issue than whether statistics are talled per port or per device -- and that is whether they are gathered separately for offload services and the default stack or not. I can envision designs where the statistics for the offload services would have to be gathered separately, and those where they would have to be gathered for all the traffic (no matter which L4/L5 service was being used). One way to deal with that might be to define the "iwarp statistics" as being those for the interface that were *not* reported to the general purpose stack. Or it might be a flag indicating whether the two sets of statistics were intermingled or not. From swise at opengridcomputing.com Fri Jun 20 14:15:35 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 20 Jun 2008 16:15:35 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <469958e00806201403i76cb5841ja39d955c02e958fe@mail.gmail.com> References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> <469958e00806201403i76cb5841ja39d955c02e958fe@mail.gmail.com> Message-ID: <485C1DF7.8000306@opengridcomputing.com> Caitlin Bestler wrote: > On Thu, Jun 19, 2008 at 8:16 AM, Roland Dreier wrote: > >> > The "Retransmission count" is per netdev and we support multiple netdev >> > per physical port. We have designed our silicon to support stats at >> > netdev level and it doesn't make sense for us to aggregate stats from >> > all netdevs and report at a device level. Perhaps we can compromise, >> > Chelsio can give device level stats at /sys/class/infiniband/$device and >> > for NetEffect, we can provide netdev level stats below it? >> >> Hmm, kind of a mess if we can't achieve some uniformity... >> >> > > There's an even tougher potential issue on the "uniformity" issue than > whether statistics are talled per port or per device -- and that is whether > they are gathered separately for offload services and the default stack > or not. > > I can envision designs where the statistics for the offload services > would have to be gathered separately, and those where they would > have to be gathered for all the traffic (no matter which L4/L5 service > was being used). > > One way to deal with that might be to define the "iwarp statistics" > as being those for the interface that were *not* reported to the > general purpose stack. Or it might be a flag indicating whether > the two sets of statistics were intermingled or not. > The L3 stats for cxgb3 are for both IP packets going to the native stack -and- the TCP offload engine. The L4 stats are just for the TCP offload engine. But I'm not sure how much we care about this. As long as the consumer knows what they mean. And I don't believe the consumer needs an API to understand this (vs just release notes per device). Steve. From Caitlin.Bestler at neterion.com Fri Jun 20 15:27:07 2008 From: Caitlin.Bestler at neterion.com (Caitlin Bestler) Date: Fri, 20 Jun 2008 18:27:07 -0400 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <485C1DF7.8000306@opengridcomputing.com> References: <20080617213856.8122.96180.stgit@dell3.ogc.int> <5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> <469958e00806201403i76cb5841ja39d955c02e958fe@mail.gmail.com> <485C1DF7.8000306@opengridcomputing.com> Message-ID: <78C9135A3D2ECE4B8162EBDCE82CAD7703C9BEB7@nekter> > > But I'm not sure how much we care about this. As long as the consumer > knows what they mean. And I don't believe the consumer needs an API to > understand this (vs just release notes per device). > I agree that release notes are probably adequate. Or at worst a MIB field. It isn't something run-time decisions are going to be based on. From sriram at pnl.gov Fri Jun 20 15:29:06 2008 From: sriram at pnl.gov (Krishnamoorthy, Sriram) Date: Fri, 20 Jun 2008 15:29:06 -0700 Subject: [ofa-general] IBV_WC_RETRY_EXC_ERR causes In-Reply-To: <485B8B33.5080801@gmail.com> References: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF983@EMAIL02.pnl.gov> <485B8B33.5080801@gmail.com> Message-ID: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF987@EMAIL02.pnl.gov> >IBV_WC_RETRY_EXE_ERR means that there wasn't any ack by the receiver after 4.096*(2 > >power 18) * 7 usec. Does an ack from the receiver require the process/thread to be awake? I have been trying to get a small test case, and sleeping without posting enough recv-s seems to occasionally result in IBV_WC_RETRY_EXC_ERR (instead of IBV_WC_RNR_RETRY_EXC_ERR which occurs a lot more often, and of course with much smaller timeout, retry_count, and rnr_retry_count). >It can happen because of several reasons: >1) bad QP attributes >2) the remote side wasn't exists or it is in bad state >3) rare, but congestion in the network can causes this too >7 means infinite retry only for RNR flow, for retry flow 7 is the number of time of the >retransmission. >How do you connect the both sides? >maybe the sender send messages to QP wasn't transfered to (at least) RTR state? All queue pairs are transitioned into RTS state before any communication. All queue pairs are transitioned to RTR state, then there is an MPI barrier (which could be using its own queue pairs or sockets), and then all queue pairs are transitioned into RTS state. All error messages out of verbs API are checked. Is it possible for a queue pair to transition into an error state and it is identified first as an IBV_WC_RETRY_EXC_ERR and not as a local error? Thanks, Sriram.K From hrosenstock at xsigo.com Fri Jun 20 15:21:37 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 20 Jun 2008 15:21:37 -0700 Subject: [ofa-general] [PATCH] opensm/osm_lid_mgr.c: Fix off by 1 issue with max_lid Message-ID: <1214000497.13056.207.camel@hrosenstock-ws.xsigo.com> opensm/osm_lid_mgr.c: Fix off by 1 issue with max_lid Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index 0652cd0..b2a6320 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -596,7 +596,7 @@ AfterScanningLids: p_range->min_lid = lid; } if (p_range) { - p_range->max_lid = p_mgr->p_subn->max_unicast_lid_ho - 1; + p_range->max_lid = p_mgr->p_subn->max_unicast_lid_ho; cl_qlist_insert_tail(&p_mgr->free_ranges, &p_range->item); OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "final free lid range [0x%x:0x%x]\n", From vlad at lists.openfabrics.org Sat Jun 21 02:25:26 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 21 Jun 2008 02:25:26 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080621-0200 daily build status Message-ID: <20080621092526.67815E61426@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Failed: Build failed on i686 with linux-2.6.16 Build failed on i686 with linux-2.6.19 Build failed on i686 with linux-2.6.18 Build failed on i686 with linux-2.6.17 Build failed on i686 with linux-2.6.21.1 Build failed on i686 with linux-2.6.22 Build failed on x86_64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.22.5-31-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22.5-31-default_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.22.5-31-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.21.1 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.21.1_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.21.1' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.22 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.22_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.22' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.16 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: 'mac' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.c:357: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes/nes.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check/drivers/infiniband/hw/nes] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.16_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.16' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080621-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From kliteyn at dev.mellanox.co.il Sat Jun 21 04:46:59 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sat, 21 Jun 2008 14:46:59 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <20080620161340.GJ3158@sashak.voltaire.com> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> Message-ID: <485CEA33.6090006@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Hi Yevgeny, > > On 16:11 Mon 16 Jun , Yevgeny Kliteynik wrote: >> Adding log message for matched criteria of >> the QoS policy rule. > > Could you explain what is the purpose of this code? And how it should be > useful in run-time? > > I'm asking because for me it looks like "pure" debug stuff which tries > to debug qos rules matching code. Many QoS matching rules are overlapping, and the user not always gets what he wants. This log message helps debug policy file, and shows why a certain ulp/application got a certain sl. -- Yevgeny > Sasha > From dotanba at gmail.com Sat Jun 21 06:23:33 2008 From: dotanba at gmail.com (Dotan Barak) Date: Sat, 21 Jun 2008 15:23:33 +0200 Subject: [ofa-general] IBV_WC_RETRY_EXC_ERR causes In-Reply-To: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF987@EMAIL02.pnl.gov> References: <7A4164E2C8D5B5498B9719E9A51BA8AE2EF983@EMAIL02.pnl.gov> <485B8B33.5080801@gmail.com> <7A4164E2C8D5B5498B9719E9A51BA8AE2EF987@EMAIL02.pnl.gov> Message-ID: <485D00D5.1040002@gmail.com> Krishnamoorthy, Sriram wrote: >> IBV_WC_RETRY_EXE_ERR means that there wasn't any ack by the receiver >> > after 4.096*(2 > > >> power 18) * 7 usec. >> > Does an ack from the receiver require the process/thread to be awake? I > have been trying to get a small test case, and sleeping without posting > enough recv-s seems to occasionally result in IBV_WC_RETRY_EXC_ERR > (instead of IBV_WC_RNR_RETRY_EXC_ERR which occurs a lot more often, and > of course with much smaller timeout, retry_count, and rnr_retry_count). > No, the ack is being handled in the HCA level (unless, the transport of the IB is being implemented in SW...) If putting sleep in your code causes IBV_WC_RETRY_EXC_ERR i would suspect SW bugs ... >> It can happen because of several reasons: >> 1) bad QP attributes >> 2) the remote side wasn't exists or it is in bad state >> 3) rare, but congestion in the network can causes this too >> > > >> 7 means infinite retry only for RNR flow, for retry flow 7 is the >> > number of time of the > >> retransmission. >> > > >> How do you connect the both sides? >> maybe the sender send messages to QP wasn't transfered to (at least) >> > RTR state? > > All queue pairs are transitioned into RTS state before any > communication. All queue pairs are transitioned to RTR state, then there > is an MPI barrier (which could be using its own queue pairs or sockets), > and then all queue pairs are transitioned into RTS state. > Good, i was afraid from any race when one side start to send messages and the other side wasn't in RTR state. > All error messages out of verbs API are checked. Is it possible for a > queue pair to transition into an error state and it is identified first > as an IBV_WC_RETRY_EXC_ERR and not as a local error? > Theoretically: no. Is this is the first message that is being passed in those QPs? Can you check the QP state of the remote side when you get such an error? > Thanks, > Sriram.K > You are welcome Dotan From dotanba at gmail.com Sat Jun 21 06:26:03 2008 From: dotanba at gmail.com (Dotan Barak) Date: Sat, 21 Jun 2008 15:26:03 +0200 Subject: [ofa-general] maximum IB/iWARP max_inline_data size In-Reply-To: References: Message-ID: <485D016B.4080009@gmail.com> Jeff Squyres wrote: > If you ibv_create_qp() with max_inline_data==0: > > - Mellanox ConnectX reports a number that is a formula based on > several values, such as max_send_sge and max_recv_sge. But its actual > max inline value (I think Pasha told me) is 1k. > > - Chelsio T3 always reports a fixed value back -- its actual > max_inline_data value (64). > > - Preliminary tests with NetEffect NE020 seem to show same behavior as > T3 (always report back 64). > > Our bias in OMPI is to figure such values dynamically at run-time. > With T3/NE020, it seems to be easy. Note that if you choose a > max_inline_data value too large, ibv_create_qp will simply fail. So > how can we figure out what the max_inline_data value is for Mellanox > adapters at run-time? (I'd prefer not to repeatedly trying to create > a QP with increasing values of max_inline_data until it fails). Is it possible to add such a capability to the HCA capabilities structure (the high limit which is supported by the device)? thanks Dotan From artisandesigns at sympatico.ca Sat Jun 21 05:57:42 2008 From: artisandesigns at sympatico.ca (Free Lotto Email Promo) Date: Sat, 21 Jun 2008 8:57:42 -0400 Subject: [ofa-general] Contact Mr Richard Smith Message-ID: <20080621125748.UMBV1729.tomts35-srv.bellnexxia.net@toip38-bus.srvr.bell.ca> we are happy to announce to you that you won the sum of $2,000,000.00 from our monthly FREELOTTO PROMOTION PROGRAM,you are advice to get back to us, to claim your prize.Contact Richard Smith E-mail:richard.smith02 at hotmail.co.uk From sashak at voltaire.com Sat Jun 21 06:10:51 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Jun 2008 16:10:51 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <485CEA33.6090006@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> <485CEA33.6090006@dev.mellanox.co.il> Message-ID: <20080621131051.GB1191@sashak.voltaire.com> On 14:46 Sat 21 Jun , Yevgeny Kliteynik wrote: > > Many QoS matching rules are overlapping, and the user not always > gets what he wants. This log message helps debug policy file, > and shows why a certain ulp/application got a certain sl. Ok, then should something like a patch below be sufficient for this? Sasha diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index f1d4e54..7999122 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -701,6 +701,11 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( if (list_iterator == cl_list_end(&p_qos_policy->qos_match_rules)) return NULL; + OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, + "request matched by \'%s\' rule, qos_level is \'%s\'\n", + p_qos_match_rule->use ? p_qos_match_rule->use : "no desc", + p_qos_match_rule->p_qos_level->name); + return p_qos_match_rule; } /* __qos_policy_get_match_rule_by_pr() */ From niggardlinessp66 at afm-reisen.de Sat Jun 21 08:12:16 2008 From: niggardlinessp66 at afm-reisen.de (Frances Hale) Date: Sat, 21 Jun 2008 17:12:16 +0200 Subject: [ofa-general] Changing careers but lack the right Degree? oyv qin opuyg Message-ID: <01c8d3c1$f43f5800$46c86455@niggardlinessp66> Want the deg pc ree but can not find the ti xe me? WHAT A GRE ch AT ID bb EA! We pro hsb vide a co vcg ncept that will allow anyone with sufficient work experience to obtain a fully verifiable Un py ivers ti ity Deg jh ree. Ba xp chel am ors, Ma tpk ste zy rs or ev qf en a Do njt cto dm rate. Think of it, within four to six weeks, you too could be a co ld lle qx ge gr zi adu yqi ate. Many people sh ys are the same fru xqv stra vf tion, they are all doing the work of the person that has the de ykv gree and the person that has the d pp egree is getting all the mo jdn ney. Do not you think that it is time you were paid fair comp cq ensa nf tion for the level of work you are already doing? This is your chance t nqa o fin bkp ally make the right move and receive your due benefits. If you are like most people, you are more than qu ot alif eyi ied with your experience, but are lacking that pre owf stig jot ious piece of paper known as a di bi plo tue ma that is often the pas ut sp lr ort to suc cg cess. CA gx LL US TODAY AND GIVE YOUR WORK EXP sd ERIE ceq NCE THE CHANCE TO EARN YOU THE HIG mak HER CO thl MPENS do ATION YOU DE pf SER irf VE! 1-5 rx 2 kr 0-2 gmi 0 bn 3-0 zo 21 vq 3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Sat Jun 21 10:33:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Jun 2008 20:33:29 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_lid_mgr.c: Fix off by 1 issue with max_lid In-Reply-To: <1214000497.13056.207.camel@hrosenstock-ws.xsigo.com> References: <1214000497.13056.207.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080621173329.GD1191@sashak.voltaire.com> On 15:21 Fri 20 Jun , Hal Rosenstock wrote: > opensm/osm_lid_mgr.c: Fix off by 1 issue with max_lid > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From kliteyn at dev.mellanox.co.il Sat Jun 21 12:45:24 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sat, 21 Jun 2008 22:45:24 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <20080621131051.GB1191@sashak.voltaire.com> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> <485CEA33.6090006@dev.mellanox.co.il> <20080621131051.GB1191@sashak.voltaire.com> Message-ID: <485D5A54.3030302@dev.mellanox.co.il> Sasha Khapyorsky wrote: > On 14:46 Sat 21 Jun , Yevgeny Kliteynik wrote: >> Many QoS matching rules are overlapping, and the user not always >> gets what he wants. This log message helps debug policy file, >> and shows why a certain ulp/application got a certain sl. > > Ok, then should something like a patch below be sufficient for this? > > Sasha > > diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c > index f1d4e54..7999122 100644 > --- a/opensm/opensm/osm_qos_policy.c > +++ b/opensm/opensm/osm_qos_policy.c > @@ -701,6 +701,11 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( > if (list_iterator == cl_list_end(&p_qos_policy->qos_match_rules)) > return NULL; > > + OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_DEBUG, > + "request matched by \'%s\' rule, qos_level is \'%s\'\n", > + p_qos_match_rule->use ? p_qos_match_rule->use : "no desc", > + p_qos_match_rule->p_qos_level->name); > + No, it won't be sufficient, because some matching rules do not explain how do they match the path queries. For instance, "sdp" doesn't say what service ID does it cover. -- Yevgeny > return p_qos_match_rule; > } /* __qos_policy_get_match_rule_by_pr() */ > > From hygienicsrsi94 at wilnsdorf.de Sat Jun 21 16:41:12 2008 From: hygienicsrsi94 at wilnsdorf.de (Laurel Dillard) Date: Sat, 21 Jun 2008 20:41:12 -0300 Subject: [ofa-general] =?iso-8859-1?q?Tiered_of_been_passed_over_for_that_?= =?iso-8859-1?q?promotion_because_you_don=92t_have_the_proper_Degre?= =?iso-8859-1?q?e=3F_pic_vtfv_lgrm?= Message-ID: <690288082.93919701153878@wilnsdorf.de> De twt gre cxj e - ad lyc van wq cem hs ent Ac djx ade qc mic Qu hb alifi sd cati qbq ons available from prestigious Non-Ac zar cre mx dit uj ed Un gqw ive xp rsi bki ties Do you have the kn jp owl oe edge and the experience but lack the qua jdg lifica lsf tio qs ns? Are you getting turned down time and time again for the job of your dreams because you just don't have the right letters after your name? Get the pre ngq sti zr ge that you deserve today! Move ahead in your ca ovy re zpj er today! Ba xw che guo lor htm s, M sv ast mpg ers and D but oct fv ora xxs te D eij egr ih ees available in y our field! No exa vrf minat wl ions! No cl aeo ass eb es! No te vo xtbo ngd oks! Call to register and receive your qu ez alifi ezu catio hw ns within days! Leave a Mes pup sage 24 ho yoa urs a day 7 da vng ys a we oa ek! 1 zy -5 gv 2 tw 0-2 fr 0 alk 3-0 vw 21 rbr 3 Co qlv nfid cnx enti dui ali pi ty as xqg su it red! -------------- next part -------------- An HTML attachment was scrubbed... URL: From benh at kernel.crashing.org Sat Jun 21 17:31:22 2008 From: benh at kernel.crashing.org (Benjamin Herrenschmidt) Date: Sun, 22 Jun 2008 10:31:22 +1000 Subject: [ofa-general] Re: [PATCH REPOST #2] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806131655.43410.ossrosch@linux.vnet.ibm.com> References: <200806131655.43410.ossrosch@linux.vnet.ibm.com> Message-ID: <1214094682.8011.200.camel@pasglop> On Fri, 2008-06-13 at 16:55 +0200, Stefan Roscher wrote: > During corner case testing, we noticed that some versions of ehca > do not properly transition to interrupt done in special load situations. > This can be resolved by periodically triggering EOI through H_EOI, > if eqes are pending. > > Signed-off-by: Stefan Roscher Acked-by: Benjamin Herrenschmidt --- > As firmware team suggested I moved the call of the EOI h_call into > the handler function, this ensures that we will call EOI only when we > find a valid eqe on the event queue. > Additionally I changed the calculation of the xirr value as Roland suggested. > > drivers/infiniband/hw/ehca/ehca_irq.c | 9 +++++++-- > drivers/infiniband/hw/ehca/hcp_if.c | 10 ++++++++++ > drivers/infiniband/hw/ehca/hcp_if.h | 1 + > 3 files changed, 18 insertions(+), 2 deletions(-) > > diff --git a/drivers/infiniband/hw/ehca/ehca_irq.c b/drivers/infiniband/hw/ehca/ehca_irq.c > index ce1ab05..0792d93 100644 > --- a/drivers/infiniband/hw/ehca/ehca_irq.c > +++ b/drivers/infiniband/hw/ehca/ehca_irq.c > @@ -531,7 +531,7 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq) > { > struct ehca_eq *eq = &shca->eq; > struct ehca_eqe_cache_entry *eqe_cache = eq->eqe_cache; > - u64 eqe_value; > + u64 eqe_value, ret; > unsigned long flags; > int eqe_cnt, i; > int eq_empty = 0; > @@ -583,8 +583,13 @@ void ehca_process_eq(struct ehca_shca *shca, int is_irq) > ehca_dbg(&shca->ib_device, > "No eqe found for irq event"); > goto unlock_irq_spinlock; > - } else if (!is_irq) > + } else if (!is_irq) { > + ret = hipz_h_eoi(eq->ist); > + if (ret != H_SUCCESS) > + ehca_err(&shca->ib_device, > + "bad return code EOI -rc = %ld\n", ret); > ehca_dbg(&shca->ib_device, "deadman found %x eqe", eqe_cnt); > + } > if (unlikely(eqe_cnt == EHCA_EQE_CACHE_SIZE)) > ehca_dbg(&shca->ib_device, "too many eqes for one irq event"); > /* enable irq for new packets */ > diff --git a/drivers/infiniband/hw/ehca/hcp_if.c b/drivers/infiniband/hw/ehca/hcp_if.c > index 5245e13..415d3a4 100644 > --- a/drivers/infiniband/hw/ehca/hcp_if.c > +++ b/drivers/infiniband/hw/ehca/hcp_if.c > @@ -933,3 +933,13 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, > r_cb, > 0, 0, 0, 0); > } > + > +u64 hipz_h_eoi(int irq) > +{ > + unsigned long xirr; > + > + iosync(); > + xirr = (0xffULL << 24) | irq; > + > + return plpar_hcall_norets(H_EOI, xirr); > +} > diff --git a/drivers/infiniband/hw/ehca/hcp_if.h b/drivers/infiniband/hw/ehca/hcp_if.h > index 60ce02b..2c3c6e0 100644 > --- a/drivers/infiniband/hw/ehca/hcp_if.h > +++ b/drivers/infiniband/hw/ehca/hcp_if.h > @@ -260,5 +260,6 @@ u64 hipz_h_error_data(const struct ipz_adapter_handle adapter_handle, > const u64 ressource_handle, > void *rblock, > unsigned long *byte_count); > +u64 hipz_h_eoi(int irq); > > #endif /* __HCP_IF_H__ */ From Mary.Briggs at borland.com Sat Jun 21 18:47:24 2008 From: Mary.Briggs at borland.com (Mary Briggs) Date: Sat, 21 Jun 2008 18:47:24 -0700 Subject: [ofa-general] WORK WITH US( PLEASE EMAIL :- pearljewelcorp2@yahoo.com.cn ) Message-ID: <7FF927B8BE5B7C43A8657469B2418702076A14@USSVS-EXCH01.amer.borl.net> A RELIABLE REPRESENTATIVE IS NEEDED IN OUR COMPANY IF INTERESTED PLEASE CONTACT US FOR MORE INFORMATIONS. email with the following informations below. Full Names................. Residential .. ........ Phone.. .................... Do get back to me if interested with this email : pearljewelcorp2 at yahoo.com.cn Thanks Mr YEO PENG LEO. From sashak at voltaire.com Sat Jun 21 19:19:24 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 22 Jun 2008 05:19:24 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <485D5A54.3030302@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> <485CEA33.6090006@dev.mellanox.co.il> <20080621131051.GB1191@sashak.voltaire.com> <485D5A54.3030302@dev.mellanox.co.il> Message-ID: <20080622021924.GE1191@sashak.voltaire.com> On 22:45 Sat 21 Jun , Yevgeny Kliteynik wrote: > > No, it won't be sufficient, because some matching rules do not > explain how do they match the path queries. For instance, "sdp" > doesn't say what service ID does it cover. But selected matching rule (and qos level) will be shown - this should provide basic information about what is going on. No? Sasha From ogerlitz at voltaire.com Sat Jun 21 23:11:32 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 22 Jun 2008 09:11:32 +0300 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statisticsattributes to the RDMA sysfs. In-Reply-To: <5E701717F2B2ED4EA60F87C8AA57B7CC08235DA9@venom2> References: <20080617213856.8122.96180.stgit@dell3.ogc.int><5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> <5E701717F2B2ED4EA60F87C8AA57B7CC08235DA9@venom2> Message-ID: <485DED14.9010305@voltaire.com> Chien Tung wrote: >> Actually it seems that you attach a struct ib_device to each netdev (eg a two-port NIC gets two RDMA devices with iw_nes) so Steve's scheme should work fine for you too? (I just looked on a system with a two-port NIC and I do see nes0 and nes1 devices) > > After some discussion with our architect, Steve's scheme will work for us by limiting to one netdev per physical port. Does "one netdev per port" means something (what?) will not be supported when VLANs are (eg nes0.4001, nes1.4002, etc) would be used? Or. From vlad at dev.mellanox.co.il Sat Jun 21 23:25:47 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Sun, 22 Jun 2008 09:25:47 +0300 Subject: [Fwd: [ofa-general] ofa_1_4_kernel 20080616-0200 daily build status] In-Reply-To: <5E701717F2B2ED4EA60F87C8AA57B7CC0795026D@venom2> References: <5E701717F2B2ED4EA60F87C8AA57B7CC0795026D@venom2> Message-ID: <485DF06B.2090703@dev.mellanox.co.il> Glenn Streiff wrote: > Hi, Vlad. > > Noticed the daily build for ofed1.4 was still broken. Since > ofed_1_4/linux-2.6.git hasn't been updated in a couple days, > I assume you were not able to pick up the commit below. > > Let us know if there is anything more to do on the NetEffect side. > > Thanks, > > Glenn > >> -----Original Message----- >> From: Glenn Streiff >> Sent: Thursday, June 19, 2008 9:41 PM >> To: 'vlad at mellanox.co.il'; Chien Tung >> Subject: RE: [Fwd: [ofa-general] ofa_1_4_kernel 20080616-0200 daily >> build status] >> >> >> >>> Hello Glenn, >>> Please arrange backport patches for NES driver for OFED-1.4. >>> >>> git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git >>> git_branch: ofed_kernel >>> >>> See attached email for details. >>> >>> Regards, >>> Vladimir >>> >> Vlad, >> >> Please pull from: >> >> git://git.openfabrics.org/~glenn/ofed-1.4.git ofed_kernel >> >> Chien Tung(1) >> commit c0afe5ef5661bcf8925dc47d4c1100eb26b1e72b >> RDMA/nes: Add ofed1.4 backports >> Done, Regards, Vladimir From kliteyn at dev.mellanox.co.il Sat Jun 21 23:30:19 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sun, 22 Jun 2008 09:30:19 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <20080622021924.GE1191@sashak.voltaire.com> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> <485CEA33.6090006@dev.mellanox.co.il> <20080621131051.GB1191@sashak.voltaire.com> <485D5A54.3030302@dev.mellanox.co.il> <20080622021924.GE1191@sashak.voltaire.com> Message-ID: <485DF17B.2050500@dev.mellanox.co.il> Sasha Khapyorsky wrote: > On 22:45 Sat 21 Jun , Yevgeny Kliteynik wrote: >> No, it won't be sufficient, because some matching rules do not >> explain how do they match the path queries. For instance, "sdp" >> doesn't say what service ID does it cover. > > But selected matching rule (and qos level) will be shown - this should > provide basic information about what is going on. No? ULP section of the policy file doesn't have separate qos-matching-rule and separate qos-level. Each line in this section internally generates matching rule, and links it to one of the pre-generated simple qos levels that has only SL. Do I want to explain it to the user? No. That's the whole point of the ULP section - the user is not aware of the full policy file syntax and semantics. However, the user does know that OSM is enforcing qos according to path queries, and the single message that I added shows *why* certain ulp/application got certain SL. The patch that I've suggested help the user to understand *what* matching rule was matched, but not *why*. -- Yevgeny > Sasha > From kliteyn at dev.mellanox.co.il Sat Jun 21 23:40:00 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sun, 22 Jun 2008 09:40:00 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <485DF17B.2050500@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> <485CEA33.6090006@dev.mellanox.co.il> <20080621131051.GB1191@sashak.voltaire.com> <485D5A54.3030302@dev.mellanox.co.il> <20080622021924.GE1191@sashak.voltaire.com> <485DF17B.2050500@dev.mellanox.co.il> Message-ID: <485DF3C0.4000900@dev.mellanox.co.il> Yevgeny Kliteynik wrote: > Sasha Khapyorsky wrote: >> On 22:45 Sat 21 Jun , Yevgeny Kliteynik wrote: >>> No, it won't be sufficient, because some matching rules do not >>> explain how do they match the path queries. For instance, "sdp" >>> doesn't say what service ID does it cover. >> >> But selected matching rule (and qos level) will be shown - this should >> provide basic information about what is going on. No? > > ULP section of the policy file doesn't have separate qos-matching-rule > and separate qos-level. Each line in this section internally generates > matching rule, and links it to one of the pre-generated simple qos levels > that has only SL. > Do I want to explain it to the user? No. That's the whole point of the ULP > section - the user is not aware of the full policy file syntax and > semantics. > > However, the user does know that OSM is enforcing qos according to path > queries, > and the single message that I added shows *why* certain ulp/application got > certain SL. The patch that I've suggested help the user to understand *what* I mean the patch that *you* have suggested. -- Yevgeny > matching rule was matched, but not *why*. > > -- Yevgeny > >> Sasha >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From ogerlitz at voltaire.com Sun Jun 22 00:16:35 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 22 Jun 2008 10:16:35 +0300 Subject: [ofa-general] Re: [PATCH 2/3] mcast loopback block In-Reply-To: References: <15ddcffd0806031159v32525d28u5e1239662bc912a9@mail.gmail.com> <484E9264.4070708@voltaire.com> <485A77AD.4010009@voltaire.com> Message-ID: <485DFC53.8060604@voltaire.com> Roland Dreier wrote: > I guess the implementation you posted is OK. OK, at first step, as its 2.6.26-rc7 now, we will move forward with these patches for 2.6.27, that is send you a fixed patch set that addresses the other comments and will cover the changes needed for kernel consumers, namely ipoib to take advantage of this. > But I don't know how to expose this to userspace anyway, since no one has figured out the QP flags issues as far as I know. There was some discussion over this list in the past on exposing qp creation flags to user space, with the context being block-multicast-loop, XRC, low latency QPs, etc (see http://lists.openfabrics.org/pipermail/general/2008-March/048435.html). I suggest that we will continue that thread and try to reach a consensus what's the correct way to allow these changes without too much pain. Or. From ogerlitz at voltaire.com Sun Jun 22 00:46:59 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 22 Jun 2008 10:46:59 +0300 Subject: [ofa-general] Re: IB/core: Add creation flags to QPs In-Reply-To: <200803261716.26026.jackm@dev.mellanox.co.il> References: <1205767427.25950.137.camel@mtls03> <200803261620.19395.jackm@dev.mellanox.co.il> <47EA636A.8070604@voltaire.com> <200803261716.26026.jackm@dev.mellanox.co.il> Message-ID: <485E0373.70004@voltaire.com> Jack Morgenstein wrote: > Actually, we did not break the ABI for XRC (we had to do cartwheels to avoid this, but what the heck). > > The create_flags field was added ONLY to struct ib_qp_init_attr (kernel only), > and was not exported upwards to the userspace API (so as not to break the ABI). > > I got around the create_flags problem by adding a new verb to userspace > (ibv_create_xrc_rcv_qp() ) with its own ABI to kernel space. Since the kernel space function (added to uverbs_cmd: ib_uverbs_create_xrc_rcv_qp() ) "knew" that it was creating an XRC_RCV qp, it set the flag in ib_qp_init_attr appropriately. > > Additionally, Eli did not need user-space involvement for his LSO flag -- so we escaped needing to increment the ABI version number. > > If we need to use the create_flags field in userspace, we WILL need to break the ABI. > > You can avoid doing this in one of 2 ways: > Either: create a new QP type > Or: add new verbs to the core. Jack, I am trying to figure out what would be the least painful way to allow for specifi Looking on the libibverbs - XRC instance (git://git.openfabrics.org/ofed_1_3/libibverbs.git) and re-reading your email, my understanding is that the way you went was 1. add a new field at the end of struct ibv_qp_init_attr 2. add bunch of new XRC verbs packed in struct ibv_xrc_ops 3. both create_qp & create_xrc_rcv_qp get ibv_qp_init_attr, the latter looks on the xrc field 4. add ibv_xrc_ops at the end of the ibv_context I have pasted below the relevant code from verbs.h, as I can't point on one patch that does this all, since there were some fixes/changes since the initial commit. So if we want to have qp creation verb that gets creation flags, we can can A. add create_flags field to the end of ibv_qp_init_attr B. introduce struct ibv_qp * (*create_qp_ext)(struct ibv_pd *pd, struct ibv_qp_init_attr *attr) C. enhance struct ibv_context similarily to what was done for xrc This seems to bring to minimum the breakage from the perspective of libibverbs consumers. As for taking this down to uverbs, I am fine with anything you suggest. Or. > struct ibv_qp_init_attr { > void *qp_context; > struct ibv_cq *send_cq; > struct ibv_cq *recv_cq; > struct ibv_srq *srq; > struct ibv_qp_cap cap; > enum ibv_qp_type qp_type; > int sq_sig_all; > struct ibv_xrc_domain *xrc_domain; > }; > > struct ibv_qp * (*create_qp)(struct ibv_pd *pd, struct ibv_qp_init_attr *attr); > > struct ibv_xrc_ops { > ... > int (*create_xrc_rcv_qp)(struct ibv_qp_init_attr *init_attr, uint32_t *xrc_qp_num); > ... > }; > > struct ibv_context { > struct ibv_device *device; > struct ibv_context_ops ops; > int cmd_fd; > int async_fd; > int num_comp_vectors; > pthread_mutex_t mutex; > void *abi_compat; > struct ibv_xrc_ops *xrc_ops; > }; From eli at mellanox.co.il Sun Jun 22 01:52:01 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Sun, 22 Jun 2008 11:52:01 +0300 Subject: [ofa-general] Re: [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: References: <1213628310.5447.91.camel@mtls03> Message-ID: <1214124721.8882.14.camel@mtls03> On Thu, 2008-06-19 at 15:11 -0700, Roland Dreier wrote: > > /** > > + * ib_create_qp_range - Creates a range of QPs associated with the > > + * specified protection domain. > > + * @pd: The protection domain associated with the QP. > > + * @qp_init_attr: A list of initial attributes required to create the > > + * QPs. If QP creation succeeds, then the attributes are updated to > > + * the actual capabilities of the created QP. > > + * @nqps: The number of required QPs > > + * @align: Required alignment for the allocated QP numbers. > > + * @list: pointer to an array to hold allocated QPs > > + */ > > +int ib_create_qp_range(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, > > + int nqps, int align, struct ib_qp *list[]); > > Well, RSS for IPoIB is OK in theory but I think we really are getting to > the point of adding a lot of extra complexity to the RDMA core where > IPoIB is the only user. And this new interface for a QP range seems > like the wrong level of abstraction -- it is exposing the mlx4 > implementation details of RSS. If the next HCA that comes out has a > different RSS scheme then we end up having to change all this. > > So the details of how RSS works should be in low-level driver code. I still need to create a bunch of QPs for this so I have to convey that information to mlx4. We can get rid of the align parameter and use another creation flag to let mlx4 know that we need the QPs aligned to the size of required number of QPs. We also need to let mlx4 know which QP is used as a default QP -- the one QP that receives all rx packets that did not map to any of the other QPs. I thought of doing this through modify QP by putting something like this +struct rca_attr { + int base_qpn; + int num_qps; + int default_qpn; +}; + struct ib_qp_attr { enum ib_qp_state qp_state; enum ib_qp_state cur_qp_state; @@ -621,6 +630,7 @@ struct ib_qp_attr { u8 rnr_retry; u8 alt_port_num; u8 alt_timeout; + struct rca_attr rca; }; Also, after all this IPoIB so we need to handle posting of RX buffers to receive queues at the IPoIB level so we need to assume that multiple RX rings implies multiple QPs and it is likely that any implementation will need similar things. Do you have in mind something specific as for how you would like to see the IB core API modified to support receive core affinity? > > By the way, Michael Chan from Broadcom just posted multi-rx-ring support > for bnx2 to netdev. I did not find this. Is it in a published git tree? Can you send a pointer? -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From ogerlitz at voltaire.com Sun Jun 22 02:10:47 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 22 Jun 2008 12:10:47 +0300 Subject: [ofa-general] Re: [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <1214124721.8882.14.camel@mtls03> References: <1213628310.5447.91.camel@mtls03> <1214124721.8882.14.camel@mtls03> Message-ID: <485E1717.4060205@voltaire.com> Eli Cohen wrote: >> By the way, Michael Chan from Broadcom just posted multi-rx-ring support for bnx2 to netdev. > I did not find this. Is it in a published git tree? Can you send a pointer? see http://marc.info/?l=linux-netdev&m=121383512817970&w=2 Or. > From vlad at lists.openfabrics.org Sun Jun 22 02:36:48 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 22 Jun 2008 02:36:48 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080622-0200 daily build status Message-ID: <20080622093648.9D0BEE6147F@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080622-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From eli at mellanox.co.il Sun Jun 22 03:05:29 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Sun, 22 Jun 2008 13:05:29 +0300 Subject: [ofa-general] Re: [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: <485E1717.4060205@voltaire.com> References: <1213628310.5447.91.camel@mtls03> <1214124721.8882.14.camel@mtls03> <485E1717.4060205@voltaire.com> Message-ID: <1214129129.8882.16.camel@mtls03> On Sun, 2008-06-22 at 12:10 +0300, Or Gerlitz wrote: > see http://marc.info/?l=linux-netdev&m=121383512817970&w=2 > Thanks. -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From stupidzm8 at m-technology.de Sun Jun 22 04:33:43 2008 From: stupidzm8 at m-technology.de (Emmanuel Browning) Date: , 22 Jun 2008 12:33:43 +0100 Subject: [ofa-general] Earn a University Degree based on your professional experience. wi rkplx jgp Message-ID: <998382662.69145839899819@m-technology.de> A Gen cms uine Un tls ivers her ity Deg mwh ree in 4-6 we wtp eks! Have you ever thought that the only thing stopping you from a great j uf ob and better p al ay was a few let iw ters behind your name? Well now you can get them! BA___B fu Sc___MA___MSc___MBA___PhD Within 4-6 weeks! No St rs udy Re jt qui jj red! 100% Ver qmm ifia ppm ble! These are real, ge njk nuine degrees that include Bac ab helors, Ma rk sters, MBA and Do hv ctorate Deg om rees. They are fully ve tes rifia km ble and ce ib rtif jxk ied transcripts are available. Just ca bew ll the number below. You will thank me later� 1-5 kvz 20-2 ar 03-0 rqu 21 dvc 3Leave a message 24 hours a day, 7 days a week including Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Sun Jun 22 04:46:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sun, 22 Jun 2008 04:46:38 -0700 Subject: [ofa-general] [PATCH] opensm: Convert mgrp_mlid_tbl into array Message-ID: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> opensm: Convert mgrp_mlid_tbl into array Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index ef9996c..75be84a 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -446,6 +447,10 @@ typedef struct osm_subn_opt { * Subnet object *********/ +struct _osm_mgrp_mlid_tbl { + void *mgroup[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; +}; + /****s* OpenSM: Subnet/osm_subn_t * NAME * osm_subn_t @@ -467,7 +472,6 @@ typedef struct osm_subn { cl_qmap_t rtr_guid_tbl; cl_qlist_t prefix_routes_list; cl_qmap_t prtn_pkey_tbl; - cl_qmap_t mgrp_mlid_tbl; cl_qmap_t sm_guid_tbl; cl_qlist_t sa_sr_list; cl_qlist_t sa_infr_list; @@ -490,6 +494,7 @@ typedef struct osm_subn { boolean_t first_time_master_sweep; boolean_t coming_out_of_standby; unsigned need_update; + struct _osm_mgrp_mlid_tbl mgrp_mlid_tbl; } osm_subn_t; /* * FIELDS @@ -513,10 +518,6 @@ typedef struct osm_subn { * Container of pointers to all Partition objects in the subnet. * Indexed by P_KEY. * -* mgrp_mlid_tbl -* Container of pointers to all Multicast Group objects in the subnet. -* Indexed by MLID. -* * sm_guid_tbl * Container of pointers to SM objects representing other SMs * on the subnet. @@ -615,6 +616,10 @@ typedef struct osm_subn { * This flag should be on during first non-master heavy (including * pre-master discovery stage) * +* mgrp_mlid_tbl +* Array of pointers to all Multicast Group objects in the subnet. +* Indexed by MLID offset from base MLID. +* * SEE ALSO * Subnet object *********/ diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index e2f5828..44ee8eb 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -163,6 +164,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) ib_gid_t port_gid; ib_mad_notice_attr_t notice; ib_api_status_t status; + int index; OSM_LOG_ENTER(sm->p_log); @@ -206,14 +208,15 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { - p_mgrp = - (osm_mgrp_t *) cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, - p_mcm->mlid); - if (p_mgrp != - (osm_mgrp_t *) cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { - osm_mgrp_remove_port(sm->p_subn, sm->p_log, - p_mgrp, p_port->guid); - osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); + for (index = 0; + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { + p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; + if (p_mgrp) { + osm_mgrp_remove_port(sm->p_subn, sm->p_log, + p_mgrp, p_port->guid); + osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); + } } p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index d7a0f56..15a585e 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -1132,6 +1133,8 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, IN ib_net64_t port_guid) { ib_api_status_t status; + osm_mgrp_t *p_mgroup; + int index; OSM_LOG_ENTER(sm->p_log); @@ -1154,8 +1157,15 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, /* Send a Report to any InformInfo registered for Trap 67 : MCGroup delete */ osm_mgrp_send_delete_notice(sm->p_subn, sm->p_log, p_mgrp); - cl_qmap_remove_item(&sm->p_subn->mgrp_mlid_tbl, - (cl_map_item_t *) p_mgrp); + for (index = 0; + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { + p_mgroup = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; + if (p_mgroup == p_mgrp) { + sm->p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; + break; + } + } osm_mgrp_delete(p_mgrp); } @@ -1171,16 +1181,14 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) osm_signal_t signal; osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; - cl_qmap_t *p_mcast_tbl; cl_qlist_t *p_list = &sm->mgrp_list; osm_mgrp_t *p_mgrp; - osm_mgrp_t *p_next_mgrp; boolean_t pending_transactions = FALSE; + int index; OSM_LOG_ENTER(sm->p_log); p_sw_tbl = &sm->p_subn->sw_guid_tbl; - p_mcast_tbl = &sm->p_subn->mgrp_mlid_tbl; /* While holding the lock, iterate over all the established multicast groups, servicing each in turn. @@ -1189,17 +1197,19 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) */ CL_PLOCK_EXCL_ACQUIRE(sm->p_lock); - p_mgrp = (osm_mgrp_t *) cl_qmap_head(p_mcast_tbl); - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(p_mcast_tbl)) { + for (index = 0; + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { /* We reached here due to some change that caused a heavy sweep of the subnet. Not due to a specific multicast request. So the request type is subnet_change and the port guid is 0. */ - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); - mcast_mgr_process_mgrp(sm, p_mgrp, - OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, 0); - p_mgrp = p_next_mgrp; + p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; + if (p_mgrp) + mcast_mgr_process_mgrp(sm, p_mgrp, + OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, + 0); } /* @@ -1233,13 +1243,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) static osm_mgrp_t *__get_mgrp_by_mlid(osm_sm_t * sm, IN ib_net16_t const mlid) { - cl_map_item_t *map_item; - - map_item = cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, mlid); - if (map_item == cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { - return NULL; - } - return (osm_mgrp_t *) map_item; + return(sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); } /********************************************************************** diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index f1d4e54..5a8dd3b 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2006 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -792,11 +793,8 @@ static void __qos_policy_validate_pkey( if (!p_prtn->mlid) return; - p_mgrp = (osm_mgrp_t *) cl_qmap_get( - &p_qos_policy->p_subn->mgrp_mlid_tbl, - p_prtn->mlid); - if (p_mgrp == (osm_mgrp_t *) - cl_qmap_end(&p_qos_policy->p_subn->mgrp_mlid_tbl)) { + p_mgrp = p_qos_policy->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(p_prtn->mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp) { OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, "ERR AC16: MCast group for partition with " "pkey 0x%04X not found\n", diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index adf4931..4695bc0 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -558,12 +559,11 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) p_mcm_port->scope_state, p_mcm_port->proxy_join); } -static void sa_dump_one_mgrp(cl_map_item_t * p_map_item, void *cxt) +static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) { struct opensm_dump_context dump_context; osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; FILE *file = ((struct opensm_dump_context *)cxt)->file; - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; fprintf(file, "MC Group 0x%04x %s:" " mgid=0x%016" PRIx64 ":0x%016" PRIx64 @@ -700,13 +700,20 @@ static void sa_dump_one_service(cl_list_item_t * p_list_item, void *cxt) static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) { struct opensm_dump_context dump_context; + osm_mgrp_t *p_mgrp; + int index; dump_context.p_osm = p_osm; dump_context.file = file; OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump multicast:\n"); cl_plock_acquire(&p_osm->lock); - cl_qmap_apply_func(&p_osm->subn.mgrp_mlid_tbl, - sa_dump_one_mgrp, &dump_context); + for (index = 0; + index <= p_osm->subn.max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { + p_mgrp = p_osm->subn.mgrp_mlid_tbl.mgroup[index]; + if (p_mgrp) + sa_dump_one_mgrp(p_mgrp, &dump_context); + } OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform:\n"); cl_qlist_apply_func(&p_osm->subn.sa_infr_list, sa_dump_one_inform, &dump_context); @@ -729,14 +736,12 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, unsigned well_known) { ib_net64_t comp_mask; - cl_map_item_t *p_next; osm_mgrp_t *p_mgrp; cl_plock_excl_acquire(&p_osm->lock); - if ((p_next = cl_qmap_get(&p_osm->subn.mgrp_mlid_tbl, mlid)) != - cl_qmap_end(&p_osm->subn.mgrp_mlid_tbl)) { - p_mgrp = (osm_mgrp_t *) p_next; + p_mgrp = p_osm->subn.mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (p_mgrp) { if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, sizeof(ib_gid_t))) { OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 46c87c7..7049fc0 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -96,13 +97,7 @@ typedef struct osm_sa_mcmr_search_ctxt { static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, IN ib_net16_t const mlid) { - cl_map_item_t *map_item; - - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { - return NULL; - } - return (osm_mgrp_t *) map_item; + return(sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); } /********************************************************************* @@ -154,16 +149,20 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho - && cl_qmap_get(&p_subn->mgrp_mlid_tbl, - requested_mlid) == - cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { + && !p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { mlid = cl_ntoh16(requested_mlid); goto Exit; } /* If MCGroups table is empty, first return the min mlid */ - p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { + max_num_mlids = sa->p_subn->max_multicast_lid_ho - + IB_LID_MCAST_START_HO + 1; + for (idx = 0; idx < max_num_mlids; idx++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; + if (p_mgrp) + break; + } + if (!p_mgrp) { mlid = IB_LID_MCAST_START_HO; OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, "No multicast groups found using minimal mlid:0x%04X\n", @@ -171,9 +170,6 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) goto Exit; } - max_num_mlids = - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; - /* track all used mlids in the array (by mlid index) */ used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); if (!used_mlids_array) @@ -181,9 +177,10 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); /* scan all available multicast groups in the DB and fill in the table */ - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { + for (idx = 0; idx < max_num_mlids; idx++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; /* ignore mgrps marked for deletion */ - if (p_mgrp->to_be_deleted == FALSE) { + if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 " : " "0x%016" PRIx64 "\n", @@ -202,11 +199,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) cl_ntoh16(p_mgrp->mlid), max_num_mlids + IB_LID_MCAST_START_HO); } else { - used_mlids_array[cl_ntoh16(p_mgrp->mlid) - - IB_LID_MCAST_START_HO] = 1; + used_mlids_array[idx] = 1; } } - p_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); } /* Find "mlid holes" in the mgrp table */ @@ -247,8 +242,7 @@ __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) not a well known group */ if (p_mgrp && cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && p_mgrp->well_known == FALSE) { - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, - (cl_map_item_t *) p_mgrp); + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; osm_mgrp_delete(p_mgrp); } } @@ -1033,21 +1027,17 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* since we might have an old group by that mlid one whose deletion was delayed for an idle time we need to deallocate it first */ - p_prev_mgrp = - (osm_mgrp_t *) cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); - if (p_prev_mgrp != - (osm_mgrp_t *) cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { + p_prev_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (p_prev_mgrp) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Found previous group for mlid:0x%04x - " "Destroying it first\n", cl_ntoh16(mlid)); - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, - (cl_map_item_t *) p_prev_mgrp); + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; osm_mgrp_delete(p_prev_mgrp); } - cl_qmap_insert(&sa->p_subn->mgrp_mlid_tbl, - mlid, &(*pp_mgrp)->map_item); + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; /* Send a Report to any InformInfo registered for Trap 66: MCGroup create */ @@ -1069,9 +1059,8 @@ typedef struct osm_sa_pr_mcmr_search_ctxt { /********************************************************************** *********************************************************************/ static void -__search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) +__search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) { - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = (osm_sa_pr_mcmr_search_ctxt_t *) context; const ib_gid_t *p_recvd_mgid; @@ -1135,23 +1124,30 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) **********************************************************************/ ib_api_status_t osm_get_mgrp_by_mgid(IN osm_sa_t *sa, - IN ib_gid_t *p_mgid, - OUT osm_mgrp_t **pp_mgrp) + IN ib_gid_t *p_mgid, + OUT osm_mgrp_t **pp_mgrp) { osm_sa_pr_mcmr_search_ctxt_t mcmr_search_context; + osm_mgrp_t *p_mgrp; + int index; mcmr_search_context.p_mgid = p_mgid; mcmr_search_context.sa = sa; mcmr_search_context.p_mgrp = NULL; - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, - __search_mgrp_by_mgid, &mcmr_search_context); - - if (mcmr_search_context.p_mgrp == NULL) - return IB_NOT_FOUND; - - *pp_mgrp = mcmr_search_context.p_mgrp; - return IB_SUCCESS; + for (index = 0; + index <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[index]; + if (p_mgrp) { + __search_mgrp_by_mgid(p_mgrp, &mcmr_search_context); + if (mcmr_search_context.p_mgrp) { + *pp_mgrp = mcmr_search_context.p_mgrp; + return IB_SUCCESS; + } + } + } + return IB_NOT_FOUND; } /********************************************************************** @@ -1619,10 +1615,9 @@ Exit: Match the given mgrp to the requested mcmr **********************************************************************/ static void -__osm_sa_mcm_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, +__osm_sa_mcm_by_comp_mask_cb(IN osm_mgrp_t * const p_mgrp, IN void *context) { - const osm_mgrp_t *const p_mgrp = (osm_mgrp_t *) p_map_item; osm_sa_mcmr_search_ctxt_t *const p_ctxt = (osm_sa_mcmr_search_ctxt_t *) context; osm_sa_t *sa = p_ctxt->sa; @@ -1811,6 +1806,8 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, ib_net64_t comp_mask; osm_physp_t *p_req_physp; boolean_t trusted_req; + osm_mgrp_t *p_mgrp; + int index; OSM_LOG_ENTER(sa->p_log); @@ -1846,8 +1843,14 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, CL_PLOCK_ACQUIRE(sa->p_lock); /* simply go over all MCGs and match */ - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, - __osm_sa_mcm_by_comp_mask_cb, &context); + for (index = 0; + index <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[index]; + if (p_mgrp) { + __osm_sa_mcm_by_comp_mask_cb(p_mgrp, &context); + } + } CL_PLOCK_RELEASE(sa->p_lock); diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 2a9155c..fb42afb 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -1472,14 +1473,7 @@ __osm_pr_rcv_process_pair(IN osm_sa_t * sa, static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, IN ib_net16_t const mlid) { - cl_map_item_t *map_item; - - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); - - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) - return NULL; - - return (osm_mgrp_t *) map_item; + return(sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); } /********************************************************************** diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index 7393ead..c6526e7 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -498,7 +499,6 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, { osm_mgrp_t *p_mgrp; osm_port_t *p_port; - cl_qmap_t *p_tbl; ib_api_status_t status = IB_SUCCESS; osm_mcm_info_t *p_mcm; @@ -525,9 +525,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, /* * If this multicast group does not already exist, create it. */ - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp) { OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, "Creating group, MLID 0x%X\n", cl_ntoh16(mlid)); @@ -540,7 +539,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, goto Exit; } - cl_qmap_insert(p_tbl, mlid, &p_mgrp->map_item); + p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = p_mgrp; } else { /* * The group already exists. If the port is not a @@ -603,7 +602,6 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, { osm_mgrp_t *p_mgrp; osm_port_t *p_port; - cl_qmap_t *p_tbl; ib_api_status_t status = IB_SUCCESS; OSM_LOG_ENTER(p_sm->p_log); @@ -630,9 +628,8 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, /* * Get the multicast group object for this group. */ - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp) { CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 812b022..3e1e3b2 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -73,6 +74,8 @@ static const char null_str[] = "(null)"; **********************************************************************/ void osm_subn_construct(IN osm_subn_t * const p_subn) { + int index; + memset(p_subn, 0, sizeof(*p_subn)); cl_ptr_vector_construct(&p_subn->port_lid_tbl); cl_qmap_init(&p_subn->sw_guid_tbl); @@ -84,19 +87,23 @@ void osm_subn_construct(IN osm_subn_t * const p_subn) cl_qlist_init(&p_subn->prefix_routes_list); cl_qmap_init(&p_subn->rtr_guid_tbl); cl_qmap_init(&p_subn->prtn_pkey_tbl); - cl_qmap_init(&p_subn->mgrp_mlid_tbl); + for (index = 0; + index <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) + p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; } /********************************************************************** **********************************************************************/ void osm_subn_destroy(IN osm_subn_t * const p_subn) { + int index; osm_node_t *p_node, *p_next_node; osm_port_t *p_port, *p_next_port; osm_switch_t *p_sw, *p_next_sw; osm_remote_sm_t *p_rsm, *p_next_rsm; osm_prtn_t *p_prtn, *p_next_prtn; - osm_mgrp_t *p_mgrp, *p_next_mgrp; + osm_mgrp_t *p_mgrp; osm_infr_t *p_infr, *p_next_infr; /* it might be a good idea to de-allocate all known objects */ @@ -139,12 +146,13 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) osm_prtn_delete(&p_prtn); } - p_next_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); - while (p_next_mgrp != - (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { - p_mgrp = p_next_mgrp; - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); - osm_mgrp_delete(p_mgrp); + for (index = 0; + index <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + index++) { + p_mgrp = p_subn->mgrp_mlid_tbl.mgroup[index]; + if (p_mgrp) + osm_mgrp_delete(p_mgrp); + p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; } p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); From hrosenstock at xsigo.com Sun Jun 22 04:50:06 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Sun, 22 Jun 2008 04:50:06 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_sw_info_rcv.c: Cosmetic formatting change Message-ID: <1214135406.13056.219.camel@hrosenstock-ws.xsigo.com> opensm/osm_sw_info_rcv.c: Cosmetic formatting change Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_sw_info_rcv.c b/opensm/opensm/osm_sw_info_rcv.c index e047e7b..3c68519 100644 --- a/opensm/opensm/osm_sw_info_rcv.c +++ b/opensm/opensm/osm_sw_info_rcv.c @@ -312,8 +312,7 @@ __osm_si_rcv_process_new(IN osm_sm_t * sm, /* set subnet max unicast lid to the minimum LinearFDBCap of all switches */ if (p_sw->fwd_tbl.p_lin_tbl->size < sm->p_subn->max_unicast_lid_ho) { - sm->p_subn->max_unicast_lid_ho = - p_sw->fwd_tbl.p_lin_tbl->size; + sm->p_subn->max_unicast_lid_ho = p_sw->fwd_tbl.p_lin_tbl->size; OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Subnet max unicast lid is 0x%X\n", sm->p_subn->max_unicast_lid_ho); From ronli at voltaire.com Sun Jun 22 09:36:40 2008 From: ronli at voltaire.com (Ron Livne) Date: Sun, 22 Jun 2008 16:36:40 +0000 (UTC) Subject: [ofa-general] [PATCH 2/3 v2] mlx4: multicast cast loopback block Message-ID: mlx4: enable discarding/passing multicast loopback packets by FW/HW. When attaching a QP to a multicast group, checks if mcast loopback packets should be blocked (receives it as a parameter). If so, it sets the mcast loopback block bit to 1. When querying a QP (which is mlx4_ib_qp) using ib_qp_query, the field create_flags of the struct ib_qp_init_attr that is given as a parameter, will now be set with the flags that were used when this QP was created. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c 2008-06-22 15:36:40.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c 2008-06-22 15:36:55.000000000 +0300 @@ -90,7 +90,8 @@ static int mlx4_ib_query_device(struct i props->device_cap_flags = IB_DEVICE_CHANGE_PHY_PORT | IB_DEVICE_PORT_ACTIVE_EVENT | IB_DEVICE_SYS_IMAGE_GUID | - IB_DEVICE_RC_RNR_NAK_GEN; + IB_DEVICE_RC_RNR_NAK_GEN | + IB_DEVICE_BLOCK_LOOPBACK; if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_BAD_PKEY_CNTR) props->device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR; if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_BAD_QKEY_CNTR) @@ -436,8 +437,12 @@ static int mlx4_ib_dealloc_pd(struct ib_ static int mlx4_ib_mcg_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) { - return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, - &to_mqp(ibqp)->mqp, gid->raw); + if (to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK) + return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, + &to_mqp(ibqp)->mqp, gid->raw, 1); + else + return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, + &to_mqp(ibqp)->mqp, gid->raw, 0); } static int mlx4_ib_mcg_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-22 15:36:40.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-22 15:36:55.000000000 +0300 @@ -101,7 +101,8 @@ struct mlx4_ib_wq { }; enum mlx4_ib_qp_flags { - MLX4_IB_QP_LSO = 1 << 0 + MLX4_IB_QP_LSO = 1 << 0, + MLX4_IB_QP_BLOCK_LOOPBACK = 1 << 1, }; struct mlx4_ib_qp { Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c 2008-06-22 15:36:40.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c 2008-06-22 15:36:55.000000000 +0300 @@ -506,6 +506,9 @@ static int create_qp_common(struct mlx4_ } else { qp->sq_no_prefetch = 0; + if (init_attr->create_flags & IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK) + qp->flags |= MLX4_IB_QP_BLOCK_LOOPBACK; + if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO) qp->flags |= MLX4_IB_QP_LSO; @@ -680,7 +683,7 @@ struct ib_qp *mlx4_ib_create_qp(struct i int err; /* We only support LSO, and only for kernel UD QPs. */ - if (init_attr->create_flags & ~IB_QP_CREATE_IPOIB_UD_LSO) + if (init_attr->create_flags & ~(IB_QP_CREATE_IPOIB_UD_LSO | IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK)) return ERR_PTR(-EINVAL); if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO && (pd->uobject || init_attr->qp_type != IB_QPT_UD)) @@ -1862,6 +1865,13 @@ done: qp_init_attr->cap = qp_attr->cap; + qp_init_attr->create_flags = 0; + if (qp->flags & MLX4_IB_QP_BLOCK_LOOPBACK) + qp_init_attr->create_flags |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; + + if (qp->flags & MLX4_IB_QP_LSO) + qp_init_attr->create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + out: mutex_unlock(&qp->mutex); return err; Index: kernels/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c 2008-06-22 15:36:40.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c 2008-06-22 15:36:55.000000000 +0300 @@ -38,6 +38,9 @@ #include "mlx4.h" +#define MGM_QPN_MASK 0x00FFFFFF +#define MGM_BLCK_LB_BIT 30 + struct mlx4_mgm { __be32 next_gid_index; __be32 members_count; @@ -153,7 +156,7 @@ static int find_mgm(struct mlx4_dev *dev return err; } -int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]) +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_cmd_mailbox *mailbox; @@ -202,13 +205,17 @@ int mlx4_multicast_attach(struct mlx4_de } for (i = 0; i < members_count; ++i) - if (mgm->qp[i] == cpu_to_be32(qp->qpn)) { + if ((be32_to_cpu(mgm->qp[i]) & MGM_QPN_MASK) == qp->qpn) { mlx4_dbg(dev, "QP %06x already a member of MGM\n", qp->qpn); err = 0; goto out; } - mgm->qp[members_count++] = cpu_to_be32(qp->qpn); + if (block_mcast_lb) + mgm->qp[members_count++] = cpu_to_be32((qp->qpn & MGM_QPN_MASK) | (1 << MGM_BLCK_LB_BIT)); + else + mgm->qp[members_count++] = cpu_to_be32(qp->qpn & MGM_QPN_MASK); + mgm->members_count = cpu_to_be32(members_count); err = mlx4_WRITE_MCG(dev, index, mailbox); @@ -283,7 +290,7 @@ int mlx4_multicast_detach(struct mlx4_de members_count = be32_to_cpu(mgm->members_count); for (loc = -1, i = 0; i < members_count; ++i) - if (mgm->qp[i] == cpu_to_be32(qp->qpn)) + if ((be32_to_cpu(mgm->qp[i]) & MGM_QPN_MASK) == qp->qpn) loc = i; if (loc == -1) { Index: kernels/linux-2.6.26-rc2/include/linux/mlx4/device.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/include/linux/mlx4/device.h 2008-06-22 15:36:40.000000000 +0300 +++ kernels/linux-2.6.26-rc2/include/linux/mlx4/device.h 2008-06-22 15:36:55.000000000 +0300 @@ -398,7 +398,7 @@ int mlx4_srq_query(struct mlx4_dev *dev, int mlx4_INIT_PORT(struct mlx4_dev *dev, int port); int mlx4_CLOSE_PORT(struct mlx4_dev *dev, int port); -int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]); +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb); int mlx4_multicast_detach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]); int mlx4_map_phys_fmr(struct mlx4_dev *dev, struct mlx4_fmr *fmr, u64 *page_list, From ronli at voltaire.com Sun Jun 22 09:35:55 2008 From: ronli at voltaire.com (Ron Livne) Date: Sun, 22 Jun 2008 16:35:55 +0000 (UTC) Subject: [ofa-general] ***SPAM*** [PATCH 1/3 v2] ib/verbs : multicast loopback blocking Message-ID: ib/verbs: add multicast block loopback capability to the ib device. This patch adds a capability flag for blocking multicast loopback packets in an ib device. This patch also adds a creation flag for QPs: IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK, which means that when attaching this QP, the multicast packets loopback block bit will be turned on. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Index: kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-06-22 15:22:03.000000000 +0300 +++ kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-06-22 15:25:02.000000000 +0300 @@ -106,6 +106,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_SEND_W_INV = (1<<21), + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), }; enum ib_atomic_cap { @@ -499,7 +500,8 @@ enum ib_qp_type { }; enum ib_qp_create_flags { - IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK = 1 << 1, }; struct ib_qp_init_attr { From ronli at voltaire.com Sun Jun 22 09:37:28 2008 From: ronli at voltaire.com (Ron Livne) Date: Sun, 22 Jun 2008 16:37:28 +0000 (UTC) Subject: [ofa-general] ***SPAM*** [PATCH 3/3 v2] ib/ipoib: blocking mcast loopback ipoib packets Message-ID: ipoib: blocking mcast loopback IPoIB packets This patch makes any mcast packet sent via IPoIB from any host, to be blocked when looped back to the same host it was sent from. The blocking is done by the HCA if possible. This patch creates an improvement of up to 39% in bandwidth when seding mcast packets with IPoIB. There is also an improvment of 12% in cpu usage. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Index: kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-02 16:58:56.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-22 15:42:16.000000000 +0300 @@ -153,7 +153,8 @@ int ipoib_transport_dev_init(struct net_ .max_recv_sge = IPOIB_UD_RX_SG }, .sq_sig_type = IB_SIGNAL_ALL_WR, - .qp_type = IB_QPT_UD + .qp_type = IB_QPT_UD, + .create_flags = 0 }; int ret, size; @@ -201,7 +202,10 @@ int ipoib_transport_dev_init(struct net_ init_attr.recv_cq = priv->recv_cq; if (priv->hca_caps & IB_DEVICE_UD_TSO) - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + + if (priv->hca_caps & IB_DEVICE_BLOCK_LOOPBACK) + init_attr.create_flags |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; if (dev->features & NETIF_F_SG) init_attr.cap.max_send_sge = MAX_SKB_FRAGS + 1; From eli at dev.mellanox.co.il Sun Jun 22 08:55:59 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 22 Jun 2008 18:55:59 +0300 Subject: [ofa-general] [PATCH] IB/mthca: Clear ICM pages before handing to FW Message-ID: <20080622155559.GA1473@mtls03> >From 086179f2ab6950b339dbb4e66f749a7c87e9a5be Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Fri, 20 Jun 2008 15:36:44 +0300 Subject: [PATCH] IB/mthca: Clear ICM pages before handing to FW Current memfree FW has a bug which in some cases, assumes that ICM pages passed to it are cleared -- this patch will clear any ICM page passed to the FW. It is essential to use this patch since there are many cards already in use which use FW with this bug. At some time when we're sure all systems are updated with the new FW (yet to be released), we can remove this patch. Signed-off-by: Eli Cohen --- This fixes the bug reported by Arthur from SGI here: http://lists.openfabrics.org/pipermail/general/2008-May/050026.html drivers/infiniband/hw/mthca/mthca_memfree.c | 29 +++++++++++++++++++++++++++ 1 files changed, 29 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/hw/mthca/mthca_memfree.c b/drivers/infiniband/hw/mthca/mthca_memfree.c index 9e77ba9..9596c9b 100644 --- a/drivers/infiniband/hw/mthca/mthca_memfree.c +++ b/drivers/infiniband/hw/mthca/mthca_memfree.c @@ -103,6 +103,33 @@ void mthca_free_icm(struct mthca_dev *dev, struct mthca_icm *icm, int coherent) kfree(icm); } +static void clear_pages(struct page *page, int npages) +{ + struct page **page_arr; + int i; + void *buf; + + page_arr = kmalloc(npages * sizeof *page_arr, GFP_KERNEL); + if (!page_arr) { + printk(KERN_WARNING "page array alloc failure\n"); + return; + } + + for (i = 0; i < npages; ++i) + page_arr[i] = page++; + + buf = vmap(page_arr, npages, VM_MAP, PAGE_KERNEL); + if (!buf) { + printk(KERN_WARNING "vmap failed\n"); + goto exit; + } + memset(buf, 0, PAGE_SIZE * npages); + vunmap(buf); + +exit: + kfree(page_arr); +} + static int mthca_alloc_icm_pages(struct scatterlist *mem, int order, gfp_t gfp_mask) { struct page *page; @@ -111,6 +138,8 @@ static int mthca_alloc_icm_pages(struct scatterlist *mem, int order, gfp_t gfp_m if (!page) return -ENOMEM; + clear_pages(page, 1 << order); + sg_set_page(mem, page, PAGE_SIZE << order, 0); return 0; } -- 1.5.5.4 From rdreier at cisco.com Sun Jun 22 09:11:43 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 22 Jun 2008 09:11:43 -0700 Subject: [ofa-general] Re: [PATCH] IB/mthca: Clear ICM pages before handing to FW In-Reply-To: <20080622155559.GA1473@mtls03> (Eli Cohen's message of "Sun, 22 Jun 2008 18:55:59 +0300") References: <20080622155559.GA1473@mtls03> Message-ID: > Current memfree FW has a bug which in some cases, assumes that ICM > pages passed to it are cleared -- this patch will clear any ICM page > passed to the FW. It is essential to use this patch since there are > many cards already in use which use FW with this bug. At some time > when we're sure all systems are updated with the new FW (yet to be > released), we can remove this patch. realistically I think we leave this workaround forever. it's not like it costs very much. Does mlx4 have any similar problem? > --- > This fixes the bug reported by Arthur from SGI here: > http://lists.openfabrics.org/pipermail/general/2008-May/050026.html This should be in the changelog itself... there's no reason to throw away this useful information when merging the patch. Anyway, thanks for debugging this... one question, why not just use the following (completely untested) change instead? Does this not work? At least using clear_highpage() instead of vmap() by hand seems much simpler too if __GFP_ZERO isn't usable for some reason. diff --git a/drivers/infiniband/hw/mthca/mthca_memfree.c b/drivers/infiniband/hw/mthca/mthca_memfree.c index 9e77ba9..4fb3c85 100644 --- a/drivers/infiniband/hw/mthca/mthca_memfree.c +++ b/drivers/infiniband/hw/mthca/mthca_memfree.c @@ -107,7 +107,7 @@ static int mthca_alloc_icm_pages(struct scatterlist *mem, int order, gfp_t gfp_m { struct page *page; - page = alloc_pages(gfp_mask, order); + page = alloc_pages(gfp_mask | __GFP_ZERO, order); if (!page) return -ENOMEM; From mlm at blfmarketing.com Sun Jun 22 09:25:19 2008 From: mlm at blfmarketing.com (Melvin Finley) Date: Sun, 22 Jun 2008 17:25:19 +0100 Subject: [ofa-general] Luxury Message-ID: <01c8d48c$f15d6980$06573d59@mlm> We are now bringing forward the replicas of the watches,Accessories, boxsets, Cufflinks, Keychains, Lighters, Pens, Tiffany & CoJewelry. A replica presupposes the exact copy of the original, that is thequality and all the other features the original possesses. For instance,youmight get a $18000 Rolex replica for a $229.00 price only! Our replica is ofsuch an amazing quality that it can be destinguished from the original only inthe jewellery workshop.You're anxiously wanting to be glamorous enough fordemanding high soiciety standarts, but your earnings are not that sky high? Youare willing to reward your lover with an outrageous gift but you don't wantthat to empty your wallet? Come, visit our shop! http://brilkean[dot]com/ Note! This link is unclickable! Copy this link and paste into your browser.When pasting don’t forget to change [dot] for ‘.’P.S When applying for a well-paid job, when flirting with agirl, when asking for a credit in the bank you get your chances doubled with ARolex President replica on your hand! PSS: The only missing thing here is a luxury suit! menireoplis poBqsyfpm7D7BqBmzlE -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Sun Jun 22 11:14:22 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 22 Jun 2008 21:14:22 +0300 Subject: [ofa-general] Re: [PATCH] IB/mthca: Clear ICM pages before handing to FW In-Reply-To: References: <20080622155559.GA1473@mtls03> Message-ID: <4e6a6b3c0806221114w72f2afd3mbe256ba8f485602a@mail.gmail.com> On Sun, Jun 22, 2008 at 7:11 PM, Roland Dreier wrote: > > realistically I think we leave this workaround forever. it's not like > it costs very much. I was thinking about applications which create and destroy HCA resources at a high rate which might be affected. > > Does mlx4 have any similar problem? > No, it does not. > > > This fixes the bug reported by Arthur from SGI here: > > http://lists.openfabrics.org/pipermail/general/2008-May/050026.html > > This should be in the changelog itself... there's no reason to throw > away this useful information when merging the patch. OK. > > > Anyway, thanks for debugging this... one question, why not just use the > following (completely untested) change instead? Does this not work? > > At least using clear_highpage() instead of vmap() by hand seems much > simpler too if __GFP_ZERO isn't usable for some reason. Looks to me like using __GFP_ZERO is the cleanest approach. I like less clear_highpage() since it uses kmap_atomic() which could fail. From countdownllx251 at harrysflohmaerkte.de Sun Jun 22 12:26:42 2008 From: countdownllx251 at harrysflohmaerkte.de (Dawn Brantley) Date: , 22 Jun 2008 20:26:42 +0100 Subject: [ofa-general] Need a Promotion. izrdq oit tus Message-ID: <421772319.11630107421943@harrysflohmaerkte.de> De my gre lha e - ad mcm van bvo cem mms ent Ac tm ade vuq mic Qu jna alifi dx cati jpt ons available from prestigious Non-Ac aep cre nu dit bm ed Un op ive uz rsi imf ties Do you have the kn kzf owl jcx edge and the experience but lack the qua iuy lifica jr tio uzs ns? Are you getting turned down time and time again for the job of your dreams because you just don't have the right letters after your name? Get the pre iz sti wb ge that you deserve today! Move ahead in your ca uw re bj er today! Ba mi che udm lor xze s, M by ast ur ers and D dwu oct vp ora xqq te D tw egr yc ees available in y our field! No exa tp minat fvm ions! No cl qqs ass sx es! No te gct xtbo vv oks! Call to register and receive your qu pi alifi wdx catio mps ns within days! Leave a Mes rde sage 24 ho wu urs a day 7 da fx ys a we jt ek! 1 yeo -5 cs 2 dl 0-2 cn 0 bkt 3-0 uy 21 nq 3 Co xt nfid qhh enti gcl ali yl ty as bzl su ero red! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Sun Jun 22 12:57:26 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 22 Jun 2008 12:57:26 -0700 Subject: [ofa-general] Re: [PATCH] IB/mthca: Clear ICM pages before handing to FW In-Reply-To: <4e6a6b3c0806221114w72f2afd3mbe256ba8f485602a@mail.gmail.com> (Eli Cohen's message of "Sun, 22 Jun 2008 21:14:22 +0300") References: <20080622155559.GA1473@mtls03> <4e6a6b3c0806221114w72f2afd3mbe256ba8f485602a@mail.gmail.com> Message-ID: > I was thinking about applications which create and destroy HCA > resources at a high rate which might be affected. I would expect all the other costs of mapping pages into/out of the ICM to be large enough that the cost of zeroing a page is not too big a deal. But if there is anyone who cares we can always make this __GFP_ZERO flag conditional on FW version I guess. > Looks to me like using __GFP_ZERO is the cleanest approach. I like > less clear_highpage() since it uses kmap_atomic() which could fail. To the best of my knowledge, kmap_atomic() cannot fail -- and if you think about it, what could make it fail? The whole point of kmap_atomic() is that there is a per-cpu pre-reserved slot to map the memory at for highmem pages, so it has to always work. As far as I can see, __GFP_ZERO allocations use clear_highpage() internally anyway, so it ends up being the same thing. Since just adding __GFP_ZERO is so much simpler, I'll just commit the patch below and send it to Linus in a day or two if it seems OK to you: commit 801d1ad7b6bdb3418a462c6b4950aee56dbac940 Author: Eli Cohen Date: Sun Jun 22 12:56:58 2008 -0700 IB/mthca: Clear ICM pages before handing to FW Current memfree FW has a bug which in some cases, assumes that ICM pages passed to it are cleared. This patch uses __GFP_ZERO to allocate all ICM pages passed to the FW. Once firmware with a fix is released, we can make the workaround conditional on firmware version. This fixes the bug reported by Arthur Kepner here: http://lists.openfabrics.org/pipermail/general/2008-May/050026.html Signed-off-by: Eli Cohen [ Rewritten to be a one-liner using __GFP_ZERO instead of vmap()ing ICM memory and memset()ing it to 0. - Roland ] Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/hw/mthca/mthca_memfree.c b/drivers/infiniband/hw/mthca/mthca_memfree.c index 9e77ba9..1f7d1a2 100644 --- a/drivers/infiniband/hw/mthca/mthca_memfree.c +++ b/drivers/infiniband/hw/mthca/mthca_memfree.c @@ -107,7 +107,11 @@ static int mthca_alloc_icm_pages(struct scatterlist *mem, int order, gfp_t gfp_m { struct page *page; - page = alloc_pages(gfp_mask, order); + /* + * Use __GFP_ZERO because buggy firmware assumes ICM pages are + * cleared, and subtle failures are seen if they aren't. + */ + page = alloc_pages(gfp_mask | __GFP_ZERO, order); if (!page) return -ENOMEM; From eli at dev.mellanox.co.il Sun Jun 22 23:35:37 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 23 Jun 2008 09:35:37 +0300 Subject: [ofa-general] Re: [PATCH] IB/mthca: Clear ICM pages before handing to FW In-Reply-To: References: <20080622155559.GA1473@mtls03> <4e6a6b3c0806221114w72f2afd3mbe256ba8f485602a@mail.gmail.com> Message-ID: <20080623063537.GB1473@mtls03> On Sun, Jun 22, 2008 at 12:57:26PM -0700, Roland Dreier wrote: > I would expect all the other costs of mapping pages into/out of the ICM > to be large enough that the cost of zeroing a page is not too big a > deal. But if there is anyone who cares we can always make this > __GFP_ZERO flag conditional on FW version I guess. Makes sense. > To the best of my knowledge, kmap_atomic() cannot fail -- and if you > think about it, what could make it fail? The whole point of > kmap_atomic() is that there is a per-cpu pre-reserved slot to map the > memory at for highmem pages, so it has to always work. Looking at the implementation of kmap_atomic() quoted bellow: void *page_address(struct page *page) { unsigned long flags; void *ret; struct page_address_slot *pas; if (!PageHighMem(page)) return lowmem_page_address(page); pas = page_slot(page); ret = NULL; spin_lock_irqsave(&pas->lock, flags); if (!list_empty(&pas->lh)) { struct page_address_map *pam; list_for_each_entry(pam, &pas->lh, list) { if (pam->page == page) { ret = pam->virtual; goto done; } } } done: spin_unlock_irqrestore(&pas->lock, flags); return ret; } static inline void *kmap_atomic(struct page *page, enum km_type idx) { pagefault_disable(); return page_address(page); } I can't see page_address doing anything actively to map the page. It just searches for the page and returns it's mapping. But I'm probably missing something. > As far as I can > see, __GFP_ZERO allocations use clear_highpage() internally anyway, so > it ends up being the same thing. Yes, now I can see that. > > Since just adding __GFP_ZERO is so much simpler, I'll just commit the > patch below and send it to Linus in a day or two if it seems OK to you: > > commit 801d1ad7b6bdb3418a462c6b4950aee56dbac940 > Author: Eli Cohen > Date: Sun Jun 22 12:56:58 2008 -0700 > > IB/mthca: Clear ICM pages before handing to FW > > Current memfree FW has a bug which in some cases, assumes that ICM > pages passed to it are cleared. This patch uses __GFP_ZERO to > allocate all ICM pages passed to the FW. Once firmware with a fix is > released, we can make the workaround conditional on firmware version. > > This fixes the bug reported by Arthur Kepner here: > http://lists.openfabrics.org/pipermail/general/2008-May/050026.html > > Signed-off-by: Eli Cohen > > [ Rewritten to be a one-liner using __GFP_ZERO instead of vmap()ing > ICM memory and memset()ing it to 0. - Roland ] > > Signed-off-by: Roland Dreier > > diff --git a/drivers/infiniband/hw/mthca/mthca_memfree.c b/drivers/infiniband/hw/mthca/mthca_memfree.c > index 9e77ba9..1f7d1a2 100644 > --- a/drivers/infiniband/hw/mthca/mthca_memfree.c > +++ b/drivers/infiniband/hw/mthca/mthca_memfree.c > @@ -107,7 +107,11 @@ static int mthca_alloc_icm_pages(struct scatterlist *mem, int order, gfp_t gfp_m > { > struct page *page; > > - page = alloc_pages(gfp_mask, order); > + /* > + * Use __GFP_ZERO because buggy firmware assumes ICM pages are > + * cleared, and subtle failures are seen if they aren't. > + */ > + page = alloc_pages(gfp_mask | __GFP_ZERO, order); > if (!page) > return -ENOMEM; > Of course. Go ahead and send it. Thanks. From olga.shern at gmail.com Mon Jun 23 01:27:33 2008 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Mon, 23 Jun 2008 11:27:33 +0300 Subject: [ofa-general] Multicast Performance In-Reply-To: <485935D5.1040900@informatik.tu-chemnitz.de> References: <4836E231.4000601@informatik.tu-chemnitz.de> <485935D5.1040900@informatik.tu-chemnitz.de> Message-ID: On 6/18/08, Marcel Heinz wrote: > Hi, > > Marcel Heinz wrote: > > [Multicast bandwidth of only ~250MByte/s] > > I'm still fighting with this issue. I've played around with my code > to track down possible causes of the effects, but it looks like I know > even less than before. > > These are the old results (client is always running on host A, the hosts > running the servers are listed right of the '->'): > > A -> [none] 258.867MB/s > A -> A: 951.288MB/s > A -> B: 258.863MB/s > A -> A,B: 952.234MB/s > A -> B,B: 143.878MB/s > > I've tested with the client attaching it's QP to the mcast group > (but not posting any receive WRs to it). And this already changed > something: > > A -> [none] 303.440MB/s > A -> A: 99.121MB/s > A -> B: 303.426MB/s > A -> A,B: 99.207MB/s > A -> B,B: 143.866MB/s > > The same effect can also be reproduced by starting another instance on > Host A which just attaches a new QP to the mc group, and then doing nothing > else, so it is not related to that particular QP I use for sending. > > After that, I checked what happens if I post just one WR to the client's > receive queue (still attached to the mc group) and don't care about it > any more. I'd expect that for that first WR, the behavior would be the > same as in the scenario with having another server instance running on > that host, and after that, it should behave like in the "attached but no > WRs" scenario above. But this is not the case: > > A -> [none]: 383.125MB/s > A -> A: 104.039MB/s > A -> B: 383.130MB/s > A -> A,B: 104.038MB/s > A -> B,B: 143.920MB/s > > I don't know why, but the overall rate stays at 383MB/s instead of going > back to the 300MB/s from the last test. I've tried to confirm these > numbers by posting more recv WRs (this time also with polling the recv > CQ), so that I really could benchmark the 2 different phases. I chose > 1000000 and got: > > Phase I Phase II > A -> [none]: 975.614MB/s 382.953MB/s > A -> B: 975.614MB/s 382.953MB/s > A -> A,B: 144.615MB/s 144.615MB/s > A -> B,B: 143.911MB/s 143.852MB/s > A -> A: 144.701MB/s 103.944MB/s > > At least my expectations for phase I were met. But I have no idea what > could cause such effects. The fact that performance increases when the > there is a "local" QP attached to the group indicates the this is a > problem at the local side, not involving the switch at all, but I can't > be sure. > > Has anyone an explanation for these numbers? Or any ideas what else I > should check? > > Regards, > Marcel > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > Hi Marcel, We also did tests with multicast traffic and figured out that when using Arbel HCAs there is performance penalty when: 1. Sending MC packets with NO QP attached on local Arble HCA. 2. Receiving MC packet with more then 1 QP attached on a single HCA which causes the back pressure that slows down the sender. Hope this helps Olga From marcel.heinz at informatik.tu-chemnitz.de Mon Jun 23 02:31:26 2008 From: marcel.heinz at informatik.tu-chemnitz.de (Marcel Heinz) Date: Mon, 23 Jun 2008 11:31:26 +0200 Subject: [ofa-general] Multicast Performance In-Reply-To: References: <4836E231.4000601@informatik.tu-chemnitz.de> <485935D5.1040900@informatik.tu-chemnitz.de> Message-ID: <485F6D6E.5020009@informatik.tu-chemnitz.de> Hi Olga, Olga Shern (Voltaire) wrote: > Hi Marcel, > > We also did tests with multicast traffic and figured out that when > using Arbel HCAs As far as I can tell, we're also using "Arbel" HCAs. > there is performance penalty when: > 1. Sending MC packets with NO QP attached on local Arble HCA. > 2. Receiving MC packet with more then 1 QP attached on a single HCA > which causes the back pressure that slows down the sender. > > Hope this helps This explains some of the effects I've seen. We only need one sender and n receivers, all on separate hosts. I've only tested all that other scenarios because I wanted to find out more about why the performance was so bad in that original scenario. Attaching the send QP to the group as a workaround shouldn't be a problem. However, my numbers suggest that attaching alone isn't enough, the data has to be received actually, which hasn't any negative impact on my benchmark, but is consuming up to another 1GByte/s of memory bandwidth, which could hurt the real application. Besides of that, there are still performance problems with n > 1, as I mentioned in the original post (c is the client sending the data, s is the server, receiving): | We could test on another cluster with 6 nodes (also with MT25204 HCAs, | I don't know the OFED version and switch type) and got the following | results: | | Host1 Host2 Host3 Host4 Host5 Host6 Throughput (MByte/s) | 1s 1s 1c 255,15 | 1s 1s 1s 1c 255,22 | 1s 1s 1s 1s 1c 255,22 | 1s 1s 1s 1s 1s 1c 255,22 | | 1s1c 1s 1s 738,64 | 1s1c 1s 1s 1s 695,08 | 1s1c 1s 1s 1s 1s 565,14 | 1s1c 1s 1s 1s 1s 1s 451,90 But maybe that was a problem of the switch or general setup. I will make some tests with our own switch as soon as I got installed an additional host. Testing multicast with only two hosts is a bit odd... ;-) Thank you! Regards, Marcel From vlad at lists.openfabrics.org Mon Jun 23 02:36:13 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 23 Jun 2008 02:36:13 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080623-0200 daily build status Message-ID: <20080623093614.17DC1E28D3F@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080623-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From scandalmongers at jettx.net Mon Jun 23 02:45:53 2008 From: scandalmongers at jettx.net (Jean Hayes) Date: Mon, 23 Jun 2008 17:45:53 +0800 Subject: [ofa-general] =After Effects CS 3 Pro= Message-ID: <000601c8d515$185f7e00$0100007f@jawdle> = Adobe CS3 Master Collection for PC or MAC includes: = InDesign CS3 = Photoshop CS3 = Illustrator CS3 = Acrobat 8 Professional = Flash CS3 Professional = Dreamweaver CS3 = Fireworks CS3 = Contribute CS3 = After Effects CS3 Professional = Premiere Pro CS3 = Encore DVD CS3 = Soundbooth CS3 = blogsoftxp . com in Internet Exp!orer = System Requirements = For PC: = Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core = Duo (or compatible) processor; SSE2-enabled processor required for AMD systems = Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) = 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components = 38GB of available hard-disk space (additional free space required during installation) = Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred = Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) = 1,280x1,024 monitor resolution with 32-bit color adapter = DVD-ROM drive = For MAC: = PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) = Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server = 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components = 36GB of available hard-disk space (additional free space required during installation) = Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred = Core Audio compatible sound card = 1,280x1,024 monitor resolution with 32-bit color adapter = DVD-ROM drive= DVD+-R burner required for DVD creation Young Indians Fight Restrictions on Alcohol The country's government is trying to help former coca farmers to develop legal businesses. People who once grew coca are now selling coffee, honey and cocoa, and running eco-lodges. But a report released this month shows that coca is still king in much of Colombia. From gnnsvtt at soyouvebeendumped.every1.net Mon Jun 23 04:21:06 2008 From: gnnsvtt at soyouvebeendumped.every1.net (Clarice) Date: Mon, 23 Jun 2008 03:21:06 -0800 (PDT) Subject: [ofa-general] is it you? Clarice here Message-ID: <09JSQ542U14YX5ZB2D4O@k.fastfoursvip.com> hello, I am pretty russian girl, bored tonight. would you like to chat with me and see my pics? if so then email me at eClarice356 at easyglo.info From ogerlitz at voltaire.com Mon Jun 23 04:36:39 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 23 Jun 2008 14:36:39 +0300 (IDT) Subject: [ofa-general] performance drop for datagram mode with the new connectx FW Message-ID: Eli, Using the new connectx FW (2.5), I see performance drop to almost zero with ipoib datagram mode. The code that runs on these systems is ofed 1.3 and not mainline kernel, details below. Running netperf With connected mode (64k MTU) I get about 950MB/s where with datagram mode (2k MTU) I get only 20-40MB/s. I used to see about 650MB/s and above with FW 2.3 and datagram mode. Not that it could explain the drop, but the NIC reports to the OS stateless offload support - /sys/class/net/ib1/features is 0x11423 I have opened the ipoib and mlx4 debug prints, and I don't see anything special other then the dmesg get quite filled with ib1: TX ring full, stopping kernel net queue any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which is about five times what I see with ipoib. Or. git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel commit 564e9e9383272f4311fd87ff4e5447cfcebad73a # uname -a Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.1 (Tikanga) From eli at dev.mellanox.co.il Mon Jun 23 04:42:36 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 23 Jun 2008 14:42:36 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: References: Message-ID: <20080623114236.GA11852@mtls03> On Mon, Jun 23, 2008 at 02:36:39PM +0300, Or Gerlitz wrote: > Eli, > > Using the new connectx FW (2.5), I see performance drop to almost > zero with ipoib datagram mode. The code that runs on these systems > is ofed 1.3 and not mainline kernel, details below. > > Running netperf With connected mode (64k MTU) I get about 950MB/s > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to > see about 650MB/s and above with FW 2.3 and datagram mode. Not that > it could explain the drop, but the NIC reports to the OS stateless > offload support - /sys/class/net/ib1/features is 0x11423 > > I have opened the ipoib and mlx4 debug prints, and I don't see anything > special other then the dmesg get quite filled with > > ib1: TX ring full, stopping kernel net queue > > any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which > is about five times what I see with ipoib. > > > Or. > > git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel > commit 564e9e9383272f4311fd87ff4e5447cfcebad73a > > # uname -a > Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 5.1 (Tikanga) Can you tell if changing the FW to 2.3 gives more reasonable results? I don't believe such a drop in performance would have passed the QA tests but I'll check that. From ogerlitz at voltaire.com Mon Jun 23 04:52:11 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 23 Jun 2008 14:52:11 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: <20080623114236.GA11852@mtls03> References: <20080623114236.GA11852@mtls03> Message-ID: <485F8E6B.7060801@voltaire.com> Eli Cohen wrote: > Can you tell if changing the FW to 2.3 gives more reasonable results? I don't believe such a drop in performance would have passed the QA tests but I'll check that. Yes, I will be able to revert to FW 2.3 later this week, but please check also with your QA Or. From eli at dev.mellanox.co.il Mon Jun 23 04:55:57 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 23 Jun 2008 14:55:57 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: <485F8E6B.7060801@voltaire.com> References: <20080623114236.GA11852@mtls03> <485F8E6B.7060801@voltaire.com> Message-ID: <20080623115557.GB11852@mtls03> On Mon, Jun 23, 2008 at 02:52:11PM +0300, Or Gerlitz wrote: > Yes, I will be able to revert to FW 2.3 later this week, but please check > also with your QA > Sure. From vlad at dev.mellanox.co.il Mon Jun 23 05:54:24 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Mon, 23 Jun 2008 15:54:24 +0300 Subject: [ofa-general] [PATCH] IPOIB: add LRO support. Message-ID: <485F9D00.6090406@dev.mellanox.co.il> From eefed5acfbadd6ea250631860ceeb06299af95e4 Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Mon, 23 Jun 2008 15:38:27 +0300 Subject: [PATCH] IPOIB: add LRO support. add "ipoib_use_lro" module parameter to enable LRO. add "ipoib_lro_max_aggr" module parameter to set the Max number of packets to be aggregated. LRO statistics accessible through ethtool. Signed-off-by: Eli Cohen Signed-off-by: Vladimir Sokolovsky --- drivers/infiniband/ulp/ipoib/Kconfig | 1 + drivers/infiniband/ulp/ipoib/ipoib.h | 13 ++++++ drivers/infiniband/ulp/ipoib/ipoib_ethtool.c | 50 ++++++++++++++++++++++ drivers/infiniband/ulp/ipoib/ipoib_ib.c | 8 +++- drivers/infiniband/ulp/ipoib/ipoib_main.c | 57 ++++++++++++++++++++++++++ 5 files changed, 128 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/Kconfig b/drivers/infiniband/ulp/ipoib/Kconfig index 1f76bad..691525c 100644 --- a/drivers/infiniband/ulp/ipoib/Kconfig +++ b/drivers/infiniband/ulp/ipoib/Kconfig @@ -1,6 +1,7 @@ config INFINIBAND_IPOIB tristate "IP-over-InfiniBand" depends on NETDEVICES && INET && (IPV6 || IPV6=n) + select INET_LRO ---help--- Support for the IP-over-InfiniBand protocol (IPoIB). This transports IP packets over InfiniBand so you can use your IB diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 8754b36..985ee87 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -50,6 +50,7 @@ #include #include #include +#include /* constants */ @@ -94,6 +95,9 @@ enum { IPOIB_MCAST_FLAG_BUSY = 2, /* joining or already joined */ IPOIB_MCAST_FLAG_ATTACHED = 3, + IPOIB_MAX_LRO_DESCRIPTORS = 8, + IPOIB_LRO_MAX_AGGR = 64, + MAX_SEND_CQE = 16, IPOIB_CM_COPYBREAK = 256, }; @@ -248,6 +252,11 @@ struct ipoib_ethtool_st { u16 max_coalesced_frames; }; +struct ipoib_lro { + struct net_lro_mgr lro_mgr; + struct net_lro_desc lro_desc[IPOIB_MAX_LRO_DESCRIPTORS]; +}; + /* * Device private locking: tx_lock protects members used in TX fast * path (and we use LLTX so upper layers don't do extra locking). @@ -334,6 +343,8 @@ struct ipoib_dev_priv { int hca_caps; struct ipoib_ethtool_st ethtool; struct timer_list poll_timer; + + struct ipoib_lro lro; }; struct ipoib_ah { @@ -679,6 +690,8 @@ extern struct ib_sa_client ipoib_sa_client; #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG extern int ipoib_debug_level; +extern int ipoib_use_lro; +extern int ipoib_lro_max_aggr; #define ipoib_dbg(priv, format, arg...) \ do { \ diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c index 10279b7..9f7f2c1 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c @@ -86,11 +86,61 @@ static int ipoib_set_coalesce(struct net_device *dev, return 0; } +#define IPOIB_STATS_LEN ARRAY_SIZE(ipoib_gstrings_stats) + +static const char ipoib_gstrings_stats[][ETH_GSTRING_LEN] = { + "LRO aggregated", "LRO flushed", + "LRO avg aggr", "LRO no_desc" +}; + +static void +ipoib_get_strings(struct net_device *netdev, u32 stringset, u8 * data) +{ + switch (stringset) { + case ETH_SS_STATS: + memcpy(data, *ipoib_gstrings_stats, + sizeof(ipoib_gstrings_stats)); + data += sizeof(ipoib_gstrings_stats); + break; + } +} + +static int ipoib_get_sset_count(struct net_device *dev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return IPOIB_STATS_LEN; + default: + return -EOPNOTSUPP; + } +} + +static void ipoib_get_ethtool_stats(struct net_device *dev, + struct ethtool_stats *stats, uint64_t *data) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + int index = 0; + + /* Get LRO statistics */ + data[index++] = priv->lro.lro_mgr.stats.aggregated; + data[index++] = priv->lro.lro_mgr.stats.flushed; + if (priv->lro.lro_mgr.stats.flushed) + data[index++] = priv->lro.lro_mgr.stats.aggregated / + priv->lro.lro_mgr.stats.flushed; + else + data[index++] = 0; + data[index++] = priv->lro.lro_mgr.stats.no_desc; + +} + static const struct ethtool_ops ipoib_ethtool_ops = { .get_drvinfo = ipoib_get_drvinfo, .get_tso = ethtool_op_get_tso, .get_coalesce = ipoib_get_coalesce, .set_coalesce = ipoib_set_coalesce, + .get_strings = ipoib_get_strings, + .get_sset_count = ipoib_get_sset_count, + .get_ethtool_stats = ipoib_get_ethtool_stats, }; void ipoib_set_ethtool_ops(struct net_device *dev) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index eca8518..9126975 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -288,7 +288,10 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) if (test_bit(IPOIB_FLAG_CSUM, &priv->flags) && likely(wc->csum_ok)) skb->ip_summed = CHECKSUM_UNNECESSARY; - netif_receive_skb(skb); + if (ipoib_use_lro) + lro_receive_skb(&priv->lro.lro_mgr, skb, 0); + else + netif_receive_skb(skb); repost: if (unlikely(ipoib_ib_post_receive(dev, wr_id))) @@ -448,6 +451,9 @@ poll_more: goto poll_more; } + if (ipoib_use_lro) + lro_flush_all(&priv->lro.lro_mgr); + return done; } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index bfe1dbf..21a87ba 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -60,6 +60,17 @@ MODULE_PARM_DESC(send_queue_size, "Number of descriptors in send queue"); module_param_named(recv_queue_size, ipoib_recvq_size, int, 0444); MODULE_PARM_DESC(recv_queue_size, "Number of descriptors in receive queue"); +int ipoib_use_lro __read_mostly = 0; +module_param_named(ipoib_use_lro, ipoib_use_lro, int, 0644); +MODULE_PARM_DESC(ipoib_use_lro, " Large Receive Offload, 1: enable, " + "0: disable, Default = 0"); + +int ipoib_lro_max_aggr __read_mostly = IPOIB_LRO_MAX_AGGR; +module_param_named(ipoib_lro_max_aggr, ipoib_lro_max_aggr, int, 0644); +MODULE_PARM_DESC(ipoib_lro_max_aggr, " LRO: Max packets to be aggregated. " + "Default = 64"); + + #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG int ipoib_debug_level; @@ -936,6 +947,50 @@ static const struct header_ops ipoib_header_ops = { .create = ipoib_hard_header, }; +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, + void **tcph, u64 *hdr_flags, void *priv) +{ + unsigned int ip_len; + struct iphdr *iph; + + if (skb->protocol != htons(ETH_P_IP)) + return -1; + + if (skb->ip_summed != CHECKSUM_UNNECESSARY) + return -1; + + /* non tcp packet */ + skb_reset_network_header(skb); + iph = ip_hdr(skb); + if (iph->protocol != IPPROTO_TCP || iph->version != 4) + return -1; + + ip_len = ip_hdrlen(skb); + skb_set_transport_header(skb, ip_len); + *tcph = tcp_hdr(skb); + + /* check if ip header and tcp header are complete */ + if (iph->tot_len < ip_len + tcp_hdrlen(skb)) + return -1; + + *hdr_flags = LRO_IPV4 | LRO_TCP; + *iphdr = iph; + + return 0; +} + +static void ipoib_lro_setup(struct ipoib_dev_priv *priv) +{ + priv->lro.lro_mgr.max_aggr = ipoib_lro_max_aggr; + priv->lro.lro_mgr.max_desc = IPOIB_MAX_LRO_DESCRIPTORS; + priv->lro.lro_mgr.lro_arr = priv->lro.lro_desc; + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; + priv->lro.lro_mgr.features = LRO_F_NAPI; + priv->lro.lro_mgr.dev = priv->dev; + priv->lro.lro_mgr.ip_summed = CHECKSUM_UNNECESSARY; + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; +} + static void ipoib_setup(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -975,6 +1030,8 @@ static void ipoib_setup(struct net_device *dev) priv->dev = dev; + ipoib_lro_setup(priv); + spin_lock_init(&priv->lock); spin_lock_init(&priv->tx_lock); -- 1.5.5.1 From jackm at dev.mellanox.co.il Mon Jun 23 06:01:10 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:10 +0300 Subject: [ofa-general] [PATCH 0/11] XRC patch series, for 2.6.27 Message-ID: <200806231601.10858.jackm@dev.mellanox.co.il> This patch series is the re-posted XRC implementation, updated for Roland's kernel 2.6.27 git tree. (kernel (core and mlx4) and user (libibverbs and libmlx4)). The kernel patches are all based on Roland's git tree: git: //git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git branch: for-2.6.27 commit: 4e5bd8b9744f3791e01f44e1d69ada21a663b798 The original series (without XRC receive QPs) was posted at: http://lists.openfabrics.org/pipermail/general/2007-September/040898.html For the original reposting of the patch series (including XRC Receive QPs), see: http://lists.openfabrics.org/pipermail/general/2008-January/045302.html For the current series, I have split out the XRC RCV additions into separate patches. Other than two kernel-space bug fixes (part of OFED 1.3) and adaptation of the patches to the current libraries and kernel source, there have been no code changes since the last posting. Therefore, please refer to the January patch series (link given above) for explanatory comments. Patches within each module/library (core, mlx4, libibverbs, and libmlx4) should be applied in the order posted. - Jack From jackm at dev.mellanox.co.il Mon Jun 23 06:01:19 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:19 +0300 Subject: [ofa-general] [PATCH 1/11] libibverbs: XRC base implementation Message-ID: <200806231601.19743.jackm@dev.mellanox.co.il> Consolidated XRC implementation patch (from OFED 1.3 libibverbs). This patch includes the following commits from OFED 1.3 libibverbs: Implement eXtended Reliable Connections (a7df4af8eb84738f36db4161a4272fa02fc6741e) Re-define IBV_DEVICE_XRC to conform to its new position (5042a9cab0ae2f7ad61bdf88dfed6fb10b700797) Set "is_srq" flag only when the QP has an SRQ (6f6d29e74ca0c19a8821990aad603e3c575b7f4d) For XRC QPs, return xrc_domain in ibv_query_qp (018c44a44ff0344dfe7cf5f6598f81d81769164e) Signed-off-by: Jack Morgenstein diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h index 67a3bf8..30ba79f 100644 Index: libibverbs/include/infiniband/driver.h =================================================================== --- libibverbs.orig/include/infiniband/driver.h 2008-06-05 15:21:54.000000000 +0300 +++ libibverbs/include/infiniband/driver.h 2008-06-05 15:29:27.000000000 +0300 @@ -99,6 +99,11 @@ int ibv_cmd_create_srq(struct ibv_pd *pd struct ibv_srq *srq, struct ibv_srq_init_attr *attr, struct ibv_create_srq *cmd, size_t cmd_size, struct ibv_create_srq_resp *resp, size_t resp_size); +int ibv_cmd_create_xrc_srq(struct ibv_pd *pd, + struct ibv_srq *srq, struct ibv_srq_init_attr *attr, + uint32_t xrc_domain, uint32_t xrc_cq, + struct ibv_create_xrc_srq *cmd, size_t cmd_size, + struct ibv_create_srq_resp *resp, size_t resp_size); int ibv_cmd_modify_srq(struct ibv_srq *srq, struct ibv_srq_attr *srq_attr, enum ibv_srq_attr_mask srq_attr_mask, @@ -134,6 +139,12 @@ int ibv_cmd_detach_mcast(struct ibv_qp * int ibv_dontfork_range(void *base, size_t size); int ibv_dofork_range(void *base, size_t size); +int ibv_cmd_open_xrc_domain(struct ibv_context *context, int fd, int oflag, + struct ibv_xrc_domain *d, + struct ibv_open_xrc_domain_resp *resp, + size_t resp_size); +int ibv_cmd_close_xrc_domain(struct ibv_xrc_domain *d); + /* * sysfs helper functions Index: libibverbs/include/infiniband/kern-abi.h =================================================================== --- libibverbs.orig/include/infiniband/kern-abi.h 2008-06-05 15:21:54.000000000 +0300 +++ libibverbs/include/infiniband/kern-abi.h 2008-06-05 15:29:27.000000000 +0300 @@ -85,7 +85,10 @@ enum { IB_USER_VERBS_CMD_MODIFY_SRQ, IB_USER_VERBS_CMD_QUERY_SRQ, IB_USER_VERBS_CMD_DESTROY_SRQ, - IB_USER_VERBS_CMD_POST_SRQ_RECV + IB_USER_VERBS_CMD_POST_SRQ_RECV, + IB_USER_VERBS_CMD_CREATE_XRC_SRQ, + IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN, + IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN }; /* @@ -706,6 +709,21 @@ struct ibv_create_srq { __u64 driver_data[0]; }; +struct ibv_create_xrc_srq { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u64 user_handle; + __u32 pd_handle; + __u32 max_wr; + __u32 max_sge; + __u32 srq_limit; + __u32 xrcd_handle; + __u32 xrc_cq; + __u64 driver_data[0]; +}; + struct ibv_create_srq_resp { __u32 srq_handle; __u32 max_wr; @@ -754,6 +772,29 @@ struct ibv_destroy_srq_resp { __u32 events_reported; }; +struct ibv_open_xrc_domain { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u32 fd; + __u32 oflags; + __u64 driver_data[0]; +}; + +struct ibv_open_xrc_domain_resp { + __u32 xrcd_handle; +}; + +struct ibv_close_xrc_domain { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u32 xrcd_handle; + __u64 driver_data[0]; +}; + /* * Compatibility with older ABI versions */ @@ -803,6 +844,9 @@ enum { * trick opcodes in IBV_INIT_CMD() doesn't break. */ IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL_V2 = -1, + IB_USER_VERBS_CMD_CREATE_XRC_SRQ_V2 = -1, + IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN_V2 = -1, + IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN_V2 = -1, }; struct ibv_destroy_cq_v1 { Index: libibverbs/include/infiniband/verbs.h =================================================================== --- libibverbs.orig/include/infiniband/verbs.h 2008-06-05 15:21:54.000000000 +0300 +++ libibverbs/include/infiniband/verbs.h 2008-06-05 15:29:27.000000000 +0300 @@ -92,7 +92,8 @@ enum ibv_device_cap_flags { IBV_DEVICE_SYS_IMAGE_GUID = 1 << 11, IBV_DEVICE_RC_RNR_NAK_GEN = 1 << 12, IBV_DEVICE_SRQ_RESIZE = 1 << 13, - IBV_DEVICE_N_NOTIFY_CQ = 1 << 14 + IBV_DEVICE_N_NOTIFY_CQ = 1 << 14, + IBV_DEVICE_XRC = 1 << 20 }; enum ibv_atomic_cap { @@ -371,6 +372,11 @@ struct ibv_ah_attr { uint8_t port_num; }; +struct ibv_xrc_domain { + struct ibv_context *context; + uint32_t handle; +}; + enum ibv_srq_attr_mask { IBV_SRQ_MAX_WR = 1 << 0, IBV_SRQ_LIMIT = 1 << 1 @@ -390,7 +396,8 @@ struct ibv_srq_init_attr { enum ibv_qp_type { IBV_QPT_RC = 2, IBV_QPT_UC, - IBV_QPT_UD + IBV_QPT_UD, + IBV_QPT_XRC }; struct ibv_qp_cap { @@ -409,6 +416,7 @@ struct ibv_qp_init_attr { struct ibv_qp_cap cap; enum ibv_qp_type qp_type; int sq_sig_all; + struct ibv_xrc_domain *xrc_domain; }; enum ibv_qp_attr_mask { @@ -527,6 +535,7 @@ struct ibv_send_wr { uint32_t remote_qkey; } ud; } wr; + uint32_t xrc_remote_srq_num; }; struct ibv_recv_wr { @@ -554,6 +563,10 @@ struct ibv_srq { pthread_mutex_t mutex; pthread_cond_t cond; uint32_t events_completed; + + uint32_t xrc_srq_num; + struct ibv_xrc_domain *xrc_domain; + struct ibv_cq *xrc_cq; }; struct ibv_qp { @@ -571,6 +584,8 @@ struct ibv_qp { pthread_mutex_t mutex; pthread_cond_t cond; uint32_t events_completed; + + struct ibv_xrc_domain *xrc_domain; }; struct ibv_comp_channel { @@ -625,6 +640,16 @@ struct ibv_device { char ibdev_path[IBV_SYSFS_PATH_MAX]; }; +struct ibv_xrc_ops { + struct ibv_srq * (*create_xrc_srq)(struct ibv_pd *pd, + struct ibv_xrc_domain *xrc_domain, + struct ibv_cq *xrc_cq, + struct ibv_srq_init_attr *srq_init_attr); + struct ibv_xrc_domain * (*open_xrc_domain)(struct ibv_context *context, + int fd, int oflag); + int (*close_xrc_domain)(struct ibv_xrc_domain *d); +}; + struct ibv_context_ops { int (*query_device)(struct ibv_context *context, struct ibv_device_attr *device_attr); @@ -691,6 +716,7 @@ struct ibv_context { int num_comp_vectors; pthread_mutex_t mutex; void *abi_compat; + struct ibv_xrc_ops *xrc_ops; }; /** @@ -913,6 +939,25 @@ struct ibv_srq *ibv_create_srq(struct ib struct ibv_srq_init_attr *srq_init_attr); /** + * ibv_create_xrc_srq - Creates a SRQ associated with the specified protection + * domain and xrc domain. + * @pd: The protection domain associated with the SRQ. + * @xrc_domain: The XRC domain associated with the SRQ. + * @xrc_cq: CQ to report completions for XRC packets on. + * + * @srq_init_attr: A list of initial attributes required to create the SRQ. + * + * srq_attr->max_wr and srq_attr->max_sge are read the determine the + * requested size of the SRQ, and set to the actual values allocated + * on return. If ibv_create_srq() succeeds, then max_wr and max_sge + * will always be at least as large as the requested values. + */ +struct ibv_srq *ibv_create_xrc_srq(struct ibv_pd *pd, + struct ibv_xrc_domain *xrc_domain, + struct ibv_cq *xrc_cq, + struct ibv_srq_init_attr *srq_init_attr); + +/** * ibv_modify_srq - Modifies the attributes for the specified SRQ. * @srq: The SRQ to modify. * @srq_attr: On input, specifies the SRQ attributes to modify. On output, @@ -1093,6 +1138,42 @@ const char *ibv_port_state_str(enum ibv_ */ const char *ibv_event_type_str(enum ibv_event_type event); +/** + * ibv_open_xrc_domain - open an XRC domain + * Returns a reference to an XRC domain. + * + * @context: Device context + * @fd: descriptor for inode associated with the domain + * If fd == -1, no inode is associated with the domain; in this case, + * the only legal value for oflag is O_CREAT + * + * @oflag: oflag values are constructed by OR-ing flags from the following list + * + * O_CREAT + * If a domain belonging to device named by context is already associated + * with the inode, this flag has no effect, except as noted under O_EXCL + * below. Otherwise, a new XRC domain is created and is associated with + * inode specified by fd. + * + * O_EXCL + * If O_EXCL and O_CREAT are set, open will fail if a domain associated with + * the inode exists. The check for the existence of the domain and creation + * of the domain if it does not exist is atomic with respect to other + * processes executing open with fd naming the same inode. + */ +struct ibv_xrc_domain *ibv_open_xrc_domain(struct ibv_context *context, + int fd, int oflag); + +/** + * ibv_close_xrc_domain - close an XRC domain + * If this is the last reference, destroys the domain. + * + * @d: reference to XRC domain to close + * + * close is implicitly performed at process exit. + */ +int ibv_close_xrc_domain(struct ibv_xrc_domain *d); + END_C_DECLS # undef __attribute_const Index: libibverbs/src/cmd.c =================================================================== --- libibverbs.orig/src/cmd.c 2008-06-05 15:21:55.000000000 +0300 +++ libibverbs/src/cmd.c 2008-06-05 15:29:27.000000000 +0300 @@ -483,6 +483,34 @@ int ibv_cmd_create_srq(struct ibv_pd *pd return 0; } +int ibv_cmd_create_xrc_srq(struct ibv_pd *pd, + struct ibv_srq *srq, struct ibv_srq_init_attr *attr, + uint32_t xrcd_handle, uint32_t xrc_cq, + struct ibv_create_xrc_srq *cmd, size_t cmd_size, + struct ibv_create_srq_resp *resp, size_t resp_size) +{ + IBV_INIT_CMD_RESP(cmd, cmd_size, CREATE_XRC_SRQ, resp, resp_size); + cmd->user_handle = (uintptr_t) srq; + cmd->pd_handle = pd->handle; + cmd->max_wr = attr->attr.max_wr; + cmd->max_sge = attr->attr.max_sge; + cmd->srq_limit = attr->attr.srq_limit; + cmd->xrcd_handle = xrcd_handle; + cmd->xrc_cq = xrc_cq; + + if (write(pd->context->cmd_fd, cmd, cmd_size) != cmd_size) + return errno; + + VALGRIND_MAKE_MEM_DEFINED(resp, resp_size); + + srq->handle = resp->srq_handle; + srq->context = pd->context; + attr->attr.max_wr = resp->max_wr; + attr->attr.max_sge = resp->max_sge; + + return 0; +} + static int ibv_cmd_modify_srq_v3(struct ibv_srq *srq, struct ibv_srq_attr *srq_attr, enum ibv_srq_attr_mask srq_attr_mask, @@ -603,7 +631,6 @@ int ibv_cmd_create_qp(struct ibv_pd *pd, cmd->pd_handle = pd->handle; cmd->send_cq_handle = attr->send_cq->handle; cmd->recv_cq_handle = attr->recv_cq->handle; - cmd->srq_handle = attr->srq ? attr->srq->handle : 0; cmd->max_send_wr = attr->cap.max_send_wr; cmd->max_recv_wr = attr->cap.max_recv_wr; cmd->max_send_sge = attr->cap.max_send_sge; @@ -612,6 +639,9 @@ int ibv_cmd_create_qp(struct ibv_pd *pd, cmd->sq_sig_all = attr->sq_sig_all; cmd->qp_type = attr->qp_type; cmd->is_srq = !!attr->srq; + cmd->srq_handle = attr->qp_type == IBV_QPT_XRC ? + (attr->xrc_domain ? attr->xrc_domain->handle : 0) : + (attr->srq ? attr->srq->handle : 0); cmd->reserved = 0; if (write(pd->context->cmd_fd, cmd, cmd_size) != cmd_size) @@ -722,6 +752,8 @@ int ibv_cmd_query_qp(struct ibv_qp *qp, init_attr->recv_cq = qp->recv_cq; init_attr->srq = qp->srq; init_attr->qp_type = qp->qp_type; + if (qp->qp_type == IBV_QPT_XRC) + init_attr->xrc_domain = qp->xrc_domain; init_attr->cap.max_send_wr = resp.max_send_wr; init_attr->cap.max_recv_wr = resp.max_recv_wr; init_attr->cap.max_send_sge = resp.max_send_sge; @@ -1122,3 +1154,41 @@ int ibv_cmd_detach_mcast(struct ibv_qp * return 0; } + +int ibv_cmd_open_xrc_domain(struct ibv_context *context, int fd, int oflag, + struct ibv_xrc_domain *d, + struct ibv_open_xrc_domain_resp *resp, + size_t resp_size) +{ + struct ibv_open_xrc_domain cmd; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD_RESP(&cmd, sizeof cmd, OPEN_XRC_DOMAIN, resp, resp_size); + cmd.fd = fd; + cmd.oflags = oflag; + + if (write(context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + + d->handle = resp->xrcd_handle; + + return 0; +} + +int ibv_cmd_close_xrc_domain(struct ibv_xrc_domain *d) +{ + struct ibv_close_xrc_domain cmd; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD(&cmd, sizeof cmd, CLOSE_XRC_DOMAIN); + cmd.xrcd_handle = d->handle; + + if (write(d->context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + return 0; +} + Index: libibverbs/src/libibverbs.map =================================================================== --- libibverbs.orig/src/libibverbs.map 2008-06-05 15:21:55.000000000 +0300 +++ libibverbs/src/libibverbs.map 2008-06-05 15:29:27.000000000 +0300 @@ -91,6 +91,12 @@ IBVERBS_1.1 { ibv_dontfork_range; ibv_dofork_range; ibv_register_driver; + ibv_create_xrc_srq; + ibv_cmd_create_xrc_srq; + ibv_open_xrc_domain; + ibv_cmd_open_xrc_domain; + ibv_close_xrc_domain; + ibv_cmd_close_xrc_domain; ibv_node_type_str; ibv_port_state_str; Index: libibverbs/src/verbs.c =================================================================== --- libibverbs.orig/src/verbs.c 2008-06-05 15:21:55.000000000 +0300 +++ libibverbs/src/verbs.c 2008-06-05 15:29:27.000000000 +0300 @@ -366,6 +366,9 @@ struct ibv_srq *__ibv_create_srq(struct srq->context = pd->context; srq->srq_context = srq_init_attr->srq_context; srq->pd = pd; + srq->xrc_domain = NULL; + srq->xrc_cq = NULL; + srq->xrc_srq_num = 0; srq->events_completed = 0; pthread_mutex_init(&srq->mutex, NULL); pthread_cond_init(&srq->cond, NULL); @@ -375,6 +378,33 @@ struct ibv_srq *__ibv_create_srq(struct } default_symver(__ibv_create_srq, ibv_create_srq); +struct ibv_srq *__ibv_create_xrc_srq(struct ibv_pd *pd, + struct ibv_xrc_domain *xrc_domain, + struct ibv_cq *xrc_cq, + struct ibv_srq_init_attr *srq_init_attr) +{ + struct ibv_srq *srq; + + if (!pd->context->xrc_ops) + return NULL; + + srq = pd->context->xrc_ops->create_xrc_srq(pd, xrc_domain, + xrc_cq, srq_init_attr); + if (srq) { + srq->context = pd->context; + srq->srq_context = srq_init_attr->srq_context; + srq->pd = pd; + srq->xrc_domain = xrc_domain; + srq->xrc_cq = xrc_cq; + srq->events_completed = 0; + pthread_mutex_init(&srq->mutex, NULL); + pthread_cond_init(&srq->cond, NULL); + } + + return srq; +} +default_symver(__ibv_create_xrc_srq, ibv_create_xrc_srq); + int __ibv_modify_srq(struct ibv_srq *srq, struct ibv_srq_attr *srq_attr, enum ibv_srq_attr_mask srq_attr_mask) @@ -410,6 +440,8 @@ struct ibv_qp *__ibv_create_qp(struct ib qp->qp_type = qp_init_attr->qp_type; qp->state = IBV_QPS_RESET; qp->events_completed = 0; + qp->xrc_domain = qp_init_attr->qp_type == IBV_QPT_XRC ? + qp_init_attr->xrc_domain : NULL; pthread_mutex_init(&qp->mutex, NULL); pthread_cond_init(&qp->cond, NULL); } @@ -543,3 +575,28 @@ int __ibv_detach_mcast(struct ibv_qp *qp return qp->context->ops.detach_mcast(qp, gid, lid); } default_symver(__ibv_detach_mcast, ibv_detach_mcast); + +struct ibv_xrc_domain *__ibv_open_xrc_domain(struct ibv_context *context, + int fd, int oflag) +{ + struct ibv_xrc_domain *d; + + if (!context->xrc_ops) + return NULL; + + d = context->xrc_ops->open_xrc_domain(context, fd, oflag); + if (d) + d->context = context; + + return d; +} +default_symver(__ibv_open_xrc_domain, ibv_open_xrc_domain); + +int __ibv_close_xrc_domain(struct ibv_xrc_domain *d) +{ + if (!d->context->xrc_ops) + return 0; + + return d->context->xrc_ops->close_xrc_domain(d); +} +default_symver(__ibv_close_xrc_domain, ibv_close_xrc_domain); From jackm at dev.mellanox.co.il Mon Jun 23 06:01:24 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:24 +0300 Subject: [ofa-general] [PATCH 2/11] libibverbs: implement XRC RCV qps Message-ID: <200806231601.25166.jackm@dev.mellanox.co.il> Added support for XRC receive-only QPs. (OFED 1.3 libibverbs commit 6e99cddf835d4715ea7ca3641944e6285f27f2df) Signed-off-by: Jack Morgenstein diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h index 30ba79f..e871f7d 100644 Index: libibverbs/include/infiniband/driver.h =================================================================== --- libibverbs.orig/include/infiniband/driver.h 2008-06-05 15:29:27.000000000 +0300 +++ libibverbs/include/infiniband/driver.h 2008-06-05 15:36:03.000000000 +0300 @@ -144,7 +144,15 @@ int ibv_cmd_open_xrc_domain(struct ibv_c struct ibv_open_xrc_domain_resp *resp, size_t resp_size); int ibv_cmd_close_xrc_domain(struct ibv_xrc_domain *d); - +int ibv_cmd_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_rcv_qpn); +int ibv_cmd_modify_xrc_rcv_qp(struct ibv_xrc_domain *d, uint32_t xrc_rcv_qpn, + struct ibv_qp_attr *attr, int attr_mask); +int ibv_cmd_query_xrc_rcv_qp(struct ibv_xrc_domain *d, uint32_t xrc_rcv_qpn, + struct ibv_qp_attr *attr, int attr_mask, + struct ibv_qp_init_attr *init_attr); +int ibv_cmd_reg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num); +int ibv_cmd_unreg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num); /* * sysfs helper functions Index: libibverbs/include/infiniband/kern-abi.h =================================================================== --- libibverbs.orig/include/infiniband/kern-abi.h 2008-06-05 15:29:27.000000000 +0300 +++ libibverbs/include/infiniband/kern-abi.h 2008-06-05 15:36:03.000000000 +0300 @@ -88,7 +88,12 @@ enum { IB_USER_VERBS_CMD_POST_SRQ_RECV, IB_USER_VERBS_CMD_CREATE_XRC_SRQ, IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN, - IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN + IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN, + IB_USER_VERBS_CMD_CREATE_XRC_RCV_QP, + IB_USER_VERBS_CMD_MODIFY_XRC_RCV_QP, + IB_USER_VERBS_CMD_QUERY_XRC_RCV_QP, + IB_USER_VERBS_CMD_REG_XRC_RCV_QP, + IB_USER_VERBS_CMD_UNREG_XRC_RCV_QP, }; /* @@ -570,6 +575,92 @@ struct ibv_destroy_qp_resp { __u32 events_reported; }; +struct ibv_create_xrc_rcv_qp { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u64 user_handle; + __u32 xrc_domain_handle; + __u32 max_send_wr; + __u32 max_recv_wr; + __u32 max_send_sge; + __u32 max_recv_sge; + __u32 max_inline_data; + __u8 sq_sig_all; + __u8 qp_type; + __u8 reserved[2]; + __u64 driver_data[0]; +}; + +struct ibv_create_xrc_rcv_qp_resp { + __u32 qpn; + __u32 reserved; +}; + +struct ibv_modify_xrc_rcv_qp { + __u32 command; + __u16 in_words; + __u16 out_words; + __u32 xrc_domain_handle; + __u32 qp_num; + struct ibv_qp_dest dest; + struct ibv_qp_dest alt_dest; + __u32 attr_mask; + __u32 qkey; + __u32 rq_psn; + __u32 sq_psn; + __u32 dest_qp_num; + __u32 qp_access_flags; + __u16 pkey_index; + __u16 alt_pkey_index; + __u8 qp_state; + __u8 cur_qp_state; + __u8 path_mtu; + __u8 path_mig_state; + __u8 en_sqd_async_notify; + __u8 max_rd_atomic; + __u8 max_dest_rd_atomic; + __u8 min_rnr_timer; + __u8 port_num; + __u8 timeout; + __u8 retry_cnt; + __u8 rnr_retry; + __u8 alt_port_num; + __u8 alt_timeout; + __u8 reserved[2]; + __u64 driver_data[0]; +}; + +struct ibv_query_xrc_rcv_qp { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u32 xrc_domain_handle; + __u32 qp_num; + __u32 attr_mask; + __u64 driver_data[0]; +}; + +struct ibv_reg_xrc_rcv_qp { + __u32 command; + __u16 in_words; + __u16 out_words; + __u32 xrc_domain_handle; + __u32 qp_num; + __u64 driver_data[0]; +}; + +struct ibv_unreg_xrc_rcv_qp { + __u32 command; + __u16 in_words; + __u16 out_words; + __u32 xrc_domain_handle; + __u32 qp_num; + __u64 driver_data[0]; +}; + struct ibv_kern_send_wr { __u64 wr_id; __u32 num_sge; @@ -847,6 +938,11 @@ enum { IB_USER_VERBS_CMD_CREATE_XRC_SRQ_V2 = -1, IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN_V2 = -1, IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN_V2 = -1, + IB_USER_VERBS_CMD_CREATE_XRC_RCV_QP_V2 = -1, + IB_USER_VERBS_CMD_MODIFY_XRC_RCV_QP_V2 = -1, + IB_USER_VERBS_CMD_QUERY_XRC_RCV_QP_V2 = -1, + IB_USER_VERBS_CMD_REG_XRC_RCV_QP_V2 = -1, + IB_USER_VERBS_CMD_UNREG_XRC_RCV_QP_V2 = -1, }; struct ibv_destroy_cq_v1 { Index: libibverbs/include/infiniband/verbs.h =================================================================== --- libibverbs.orig/include/infiniband/verbs.h 2008-06-05 15:29:27.000000000 +0300 +++ libibverbs/include/infiniband/verbs.h 2008-06-05 15:36:03.000000000 +0300 @@ -205,12 +205,17 @@ enum ibv_event_type { IBV_EVENT_CLIENT_REREGISTER }; +enum ibv_event_flags { + IBV_XRC_QP_EVENT_FLAG = 0x80000000, +}; + struct ibv_async_event { union { struct ibv_cq *cq; struct ibv_qp *qp; struct ibv_srq *srq; int port_num; + uint32_t xrc_qp_num; } element; enum ibv_event_type event_type; }; @@ -648,6 +653,22 @@ struct ibv_xrc_ops { struct ibv_xrc_domain * (*open_xrc_domain)(struct ibv_context *context, int fd, int oflag); int (*close_xrc_domain)(struct ibv_xrc_domain *d); + int (*create_xrc_rcv_qp)(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_qp_num); + int (*modify_xrc_rcv_qp)(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, + int attr_mask); + int (*query_xrc_rcv_qp)(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, + int attr_mask, + struct ibv_qp_init_attr *init_attr); + int (*reg_xrc_rcv_qp)(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num); + int (*unreg_xrc_rcv_qp)(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num); + }; struct ibv_context_ops { @@ -1174,6 +1195,100 @@ struct ibv_xrc_domain *ibv_open_xrc_doma */ int ibv_close_xrc_domain(struct ibv_xrc_domain *d); +/** + * ibv_create_xrc_rcv_qp - creates an XRC QP for serving as a receive-side only QP, + * + * This QP is created in kernel space, and persists until the last process registered + * for the QP calls ibv_unreg_xrc_rcv_qp() (at which time the QP is destroyed). + * + * @init_attr: init attributes to use for QP. xrc domain MUST be included here. All other fields + * are ignored. + * + * @xrc_rcv_qpn: qp_num of created QP (if success). To be passed to the remote node (sender). + * The remote node will use xrc_rcv_qpn in ibv_post_send when sending to + * XRC SRQ's on this host in the same xrc domain. + * + * RETURNS: success (0), or a (negative) error value. + * + * NOTE: this verb also registers the calling user-process with the QP at its creation time + * (implicit call to ibv_reg_xrc_rcv_qp), to avoid race conditions. + * The creating process will need to call ibv_unreg_xrc_qp() for the QP to release it from + * this process. + */ +int ibv_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_rcv_qpn); + +/** + * ibv_modify_xrc_rcv_qp - modifies an xrc_rcv qp. + * + * @xrc_domain: xrc domain the QP belongs to (for verification). + * @xrc_qp_num: The (24 bit) number of the XRC QP. + * @attr: modify-qp attributes. The following fields must be specified: + * for RESET_2_INIT: qp_state, pkey_index , port, qp_access_flags + * for INIT_2_RTR: qp_state, path_mtu, dest_qp_num, rq_psn, max_dest_rd_atomic, + * min_rnr_timer, ah_attr + * The QP need not be brought to RTS for the QP to operate as a receive-only QP. + * @attr_mask: bitmap indicating which attributes are provided in the attr struct. + * used for validity checking. The following bits must be set: + * for RESET_2_INIT: IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS, IBV_QP_STATE + * for INIT_2_RTR: IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN, + * IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER, IBV_QP_STATE + * + * RETURNS: success (0), or a (negative) error value. + * + */ +int ibv_modify_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, int attr_mask); + +/** + * ibv_query_xrc_rcv_qp - queries an xrc_rcv qp. + * + * @xrc_domain: xrc domain the QP belongs to (for verification). + * @xrc_qp_num: The (24 bit) number of the XRC QP. + * @attr: for returning qp attributes. + * @attr_mask: bitmap indicating which attributes to return. + * @init_attr: for returning the init attributes + * + * RETURNS: success (0), or a (negative) error value. + * + */ +int ibv_query_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, int attr_mask, + struct ibv_qp_init_attr *init_attr); + +/** + * ibv_reg_xrc_rcv_qp: registers a user process with an XRC QP which serves as + * a receive-side only QP. + * + * @xrc_domain: xrc domain the QP belongs to (for verification). + * @xrc_qp_num: The (24 bit) number of the XRC QP. + * + * RETURNS: success (0), + * or error (-EINVAL), if: + * 1. There is no such QP_num allocated. + * 2. The QP is allocated, but is not an receive XRC QP + * 3. The XRC QP does not belong to the given domain. + */ +int ibv_reg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num); + +/** + * ibv_unreg_xrc_rcv_qp: detaches a user process from an XRC QP serving as + * a receive-side only QP. If as a result, there are no remaining + * userspace processes registered for this XRC QP, it is destroyed. + * + * @xrc_domain: xrc domain the QP belongs to (for verification). + * @xrc_qp_num: The (24 bit) number of the XRC QP. + * + * RETURNS: success (0), + * or error (-EINVAL), if: + * 1. There is no such QP_num allocated. + * 2. The QP is allocated, but is not an XRC QP + * 3. The XRC QP does not belong to the given domain. + * NOTE: I don't see any reason to return a special code if the QP is destroyed -- the unregister simply + * succeeds. + */ +int ibv_unreg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, uint32_t xrc_qp_num); + END_C_DECLS # undef __attribute_const Index: libibverbs/src/cmd.c =================================================================== --- libibverbs.orig/src/cmd.c 2008-06-05 15:29:27.000000000 +0300 +++ libibverbs/src/cmd.c 2008-06-05 15:36:03.000000000 +0300 @@ -828,6 +828,186 @@ int ibv_cmd_modify_qp(struct ibv_qp *qp, return 0; } +int ibv_cmd_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_rcv_qpn) +{ + struct ibv_create_xrc_rcv_qp cmd; + struct ibv_create_xrc_rcv_qp_resp resp; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD_RESP(&cmd, sizeof cmd, CREATE_XRC_RCV_QP, &resp, sizeof resp); + + cmd.xrc_domain_handle = init_attr->xrc_domain->handle; + cmd.max_send_wr = init_attr->cap.max_send_wr; + cmd.max_recv_wr = init_attr->cap.max_recv_wr; + cmd.max_send_sge = init_attr->cap.max_send_sge; + cmd.max_recv_sge = init_attr->cap.max_recv_sge; + cmd.max_inline_data = init_attr->cap.max_inline_data; + cmd.sq_sig_all = init_attr->sq_sig_all; + cmd.qp_type = init_attr->qp_type; + cmd.reserved[0] = cmd.reserved[1] = 0; + + if (write(init_attr->xrc_domain->context->cmd_fd, &cmd, sizeof cmd) != + sizeof cmd) + return errno; + + *xrc_rcv_qpn = resp.qpn; + + return 0; +} + +int ibv_cmd_modify_xrc_rcv_qp(struct ibv_xrc_domain *d, uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, int attr_mask) +{ + struct ibv_modify_xrc_rcv_qp cmd; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD(&cmd, sizeof cmd, MODIFY_XRC_RCV_QP); + + cmd.xrc_domain_handle = d->handle; + cmd.qp_num = xrc_qp_num; + cmd.attr_mask = attr_mask; + cmd.qkey = attr->qkey; + cmd.rq_psn = attr->rq_psn; + cmd.sq_psn = attr->sq_psn; + cmd.dest_qp_num = attr->dest_qp_num; + cmd.qp_access_flags = attr->qp_access_flags; + cmd.pkey_index = attr->pkey_index; + cmd.alt_pkey_index = attr->alt_pkey_index; + cmd.qp_state = attr->qp_state; + cmd.cur_qp_state = attr->cur_qp_state; + cmd.path_mtu = attr->path_mtu; + cmd.path_mig_state = attr->path_mig_state; + cmd.en_sqd_async_notify = attr->en_sqd_async_notify; + cmd.max_rd_atomic = attr->max_rd_atomic; + cmd.max_dest_rd_atomic = attr->max_dest_rd_atomic; + cmd.min_rnr_timer = attr->min_rnr_timer; + cmd.port_num = attr->port_num; + cmd.timeout = attr->timeout; + cmd.retry_cnt = attr->retry_cnt; + cmd.rnr_retry = attr->rnr_retry; + cmd.alt_port_num = attr->alt_port_num; + cmd.alt_timeout = attr->alt_timeout; + + memcpy(cmd.dest.dgid, attr->ah_attr.grh.dgid.raw, 16); + cmd.dest.flow_label = attr->ah_attr.grh.flow_label; + cmd.dest.dlid = attr->ah_attr.dlid; + cmd.dest.reserved = 0; + cmd.dest.sgid_index = attr->ah_attr.grh.sgid_index; + cmd.dest.hop_limit = attr->ah_attr.grh.hop_limit; + cmd.dest.traffic_class = attr->ah_attr.grh.traffic_class; + cmd.dest.sl = attr->ah_attr.sl; + cmd.dest.src_path_bits = attr->ah_attr.src_path_bits; + cmd.dest.static_rate = attr->ah_attr.static_rate; + cmd.dest.is_global = attr->ah_attr.is_global; + cmd.dest.port_num = attr->ah_attr.port_num; + + memcpy(cmd.alt_dest.dgid, attr->alt_ah_attr.grh.dgid.raw, 16); + cmd.alt_dest.flow_label = attr->alt_ah_attr.grh.flow_label; + cmd.alt_dest.dlid = attr->alt_ah_attr.dlid; + cmd.alt_dest.reserved = 0; + cmd.alt_dest.sgid_index = attr->alt_ah_attr.grh.sgid_index; + cmd.alt_dest.hop_limit = attr->alt_ah_attr.grh.hop_limit; + cmd.alt_dest.traffic_class = attr->alt_ah_attr.grh.traffic_class; + cmd.alt_dest.sl = attr->alt_ah_attr.sl; + cmd.alt_dest.src_path_bits = attr->alt_ah_attr.src_path_bits; + cmd.alt_dest.static_rate = attr->alt_ah_attr.static_rate; + cmd.alt_dest.is_global = attr->alt_ah_attr.is_global; + cmd.alt_dest.port_num = attr->alt_ah_attr.port_num; + + cmd.reserved[0] = cmd.reserved[1] = 0; + + if (write(d->context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + + return 0; +} + +int ibv_cmd_query_xrc_rcv_qp(struct ibv_xrc_domain *d, uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, int attr_mask, + struct ibv_qp_init_attr *init_attr) +{ + struct ibv_query_xrc_rcv_qp cmd; + struct ibv_query_qp_resp resp; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD_RESP(&cmd, sizeof cmd, QUERY_XRC_RCV_QP, &resp, sizeof resp); + cmd.xrc_domain_handle = d->handle; + cmd.qp_num = xrc_qp_num; + cmd.attr_mask = attr_mask; + + if (write(d->context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + + VALGRIND_MAKE_MEM_DEFINED(&resp, sizeof resp); + + attr->qkey = resp.qkey; + attr->rq_psn = resp.rq_psn; + attr->sq_psn = resp.sq_psn; + attr->dest_qp_num = resp.dest_qp_num; + attr->qp_access_flags = resp.qp_access_flags; + attr->pkey_index = resp.pkey_index; + attr->alt_pkey_index = resp.alt_pkey_index; + attr->qp_state = resp.qp_state; + attr->cur_qp_state = resp.cur_qp_state; + attr->path_mtu = resp.path_mtu; + attr->path_mig_state = resp.path_mig_state; + attr->sq_draining = resp.sq_draining; + attr->max_rd_atomic = resp.max_rd_atomic; + attr->max_dest_rd_atomic = resp.max_dest_rd_atomic; + attr->min_rnr_timer = resp.min_rnr_timer; + attr->port_num = resp.port_num; + attr->timeout = resp.timeout; + attr->retry_cnt = resp.retry_cnt; + attr->rnr_retry = resp.rnr_retry; + attr->alt_port_num = resp.alt_port_num; + attr->alt_timeout = resp.alt_timeout; + attr->cap.max_send_wr = resp.max_send_wr; + attr->cap.max_recv_wr = resp.max_recv_wr; + attr->cap.max_send_sge = resp.max_send_sge; + attr->cap.max_recv_sge = resp.max_recv_sge; + attr->cap.max_inline_data = resp.max_inline_data; + + memcpy(attr->ah_attr.grh.dgid.raw, resp.dest.dgid, 16); + attr->ah_attr.grh.flow_label = resp.dest.flow_label; + attr->ah_attr.dlid = resp.dest.dlid; + attr->ah_attr.grh.sgid_index = resp.dest.sgid_index; + attr->ah_attr.grh.hop_limit = resp.dest.hop_limit; + attr->ah_attr.grh.traffic_class = resp.dest.traffic_class; + attr->ah_attr.sl = resp.dest.sl; + attr->ah_attr.src_path_bits = resp.dest.src_path_bits; + attr->ah_attr.static_rate = resp.dest.static_rate; + attr->ah_attr.is_global = resp.dest.is_global; + attr->ah_attr.port_num = resp.dest.port_num; + + memcpy(attr->alt_ah_attr.grh.dgid.raw, resp.alt_dest.dgid, 16); + attr->alt_ah_attr.grh.flow_label = resp.alt_dest.flow_label; + attr->alt_ah_attr.dlid = resp.alt_dest.dlid; + attr->alt_ah_attr.grh.sgid_index = resp.alt_dest.sgid_index; + attr->alt_ah_attr.grh.hop_limit = resp.alt_dest.hop_limit; + attr->alt_ah_attr.grh.traffic_class = resp.alt_dest.traffic_class; + attr->alt_ah_attr.sl = resp.alt_dest.sl; + attr->alt_ah_attr.src_path_bits = resp.alt_dest.src_path_bits; + attr->alt_ah_attr.static_rate = resp.alt_dest.static_rate; + attr->alt_ah_attr.is_global = resp.alt_dest.is_global; + attr->alt_ah_attr.port_num = resp.alt_dest.port_num; + + init_attr->cap.max_send_wr = resp.max_send_wr; + init_attr->cap.max_recv_wr = resp.max_recv_wr; + init_attr->cap.max_send_sge = resp.max_send_sge; + init_attr->cap.max_recv_sge = resp.max_recv_sge; + init_attr->cap.max_inline_data = resp.max_inline_data; + init_attr->sq_sig_all = resp.sq_sig_all; + + return 0; +} + static int ibv_cmd_destroy_qp_v1(struct ibv_qp *qp) { struct ibv_destroy_qp_v1 cmd; @@ -1192,3 +1372,36 @@ int ibv_cmd_close_xrc_domain(struct ibv_ return 0; } +int ibv_cmd_reg_xrc_rcv_qp(struct ibv_xrc_domain *d, uint32_t xrc_qp_num) +{ + struct ibv_reg_xrc_rcv_qp cmd; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD(&cmd, sizeof cmd, REG_XRC_RCV_QP); + cmd.xrc_domain_handle = d->handle; + cmd.qp_num = xrc_qp_num; + + if (write(d->context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + return 0; +} + +int ibv_cmd_unreg_xrc_rcv_qp(struct ibv_xrc_domain *d, uint32_t xrc_qp_num) +{ + struct ibv_unreg_xrc_rcv_qp cmd; + + if (abi_ver < 6) + return ENOSYS; + + IBV_INIT_CMD(&cmd, sizeof cmd, UNREG_XRC_RCV_QP); + cmd.xrc_domain_handle = d->handle; + cmd.qp_num = xrc_qp_num; + + if (write(d->context->cmd_fd, &cmd, sizeof cmd) != sizeof cmd) + return errno; + return 0; +} + + Index: libibverbs/src/device.c =================================================================== --- libibverbs.orig/src/device.c 2008-06-05 15:21:55.000000000 +0300 +++ libibverbs/src/device.c 2008-06-05 15:36:03.000000000 +0300 @@ -182,31 +182,33 @@ int __ibv_get_async_event(struct ibv_con event->event_type = ev.event_type; - switch (event->event_type) { - case IBV_EVENT_CQ_ERR: - event->element.cq = (void *) (uintptr_t) ev.element; - break; - - case IBV_EVENT_QP_FATAL: - case IBV_EVENT_QP_REQ_ERR: - case IBV_EVENT_QP_ACCESS_ERR: - case IBV_EVENT_COMM_EST: - case IBV_EVENT_SQ_DRAINED: - case IBV_EVENT_PATH_MIG: - case IBV_EVENT_PATH_MIG_ERR: - case IBV_EVENT_QP_LAST_WQE_REACHED: - event->element.qp = (void *) (uintptr_t) ev.element; - break; - - case IBV_EVENT_SRQ_ERR: - case IBV_EVENT_SRQ_LIMIT_REACHED: - event->element.srq = (void *) (uintptr_t) ev.element; - break; - - default: - event->element.port_num = ev.element; - break; - } + if (event->event_type & IBV_XRC_QP_EVENT_FLAG) { + event->element.xrc_qp_num = ev.element; + } else + switch (event->event_type) { + case IBV_EVENT_CQ_ERR: + event->element.cq = (void *) (uintptr_t) ev.element; + break; + + case IBV_EVENT_QP_FATAL: + case IBV_EVENT_QP_REQ_ERR: + case IBV_EVENT_QP_ACCESS_ERR: + case IBV_EVENT_COMM_EST: + case IBV_EVENT_SQ_DRAINED: + case IBV_EVENT_PATH_MIG: + case IBV_EVENT_PATH_MIG_ERR: + case IBV_EVENT_QP_LAST_WQE_REACHED: + event->element.qp = (void *) (uintptr_t) ev.element; + break; + + case IBV_EVENT_SRQ_ERR: + case IBV_EVENT_SRQ_LIMIT_REACHED: + event->element.srq = (void *) (uintptr_t) ev.element; + break; + default: + event->element.port_num = ev.element; + break; + } if (context->ops.async_event) context->ops.async_event(event); Index: libibverbs/src/libibverbs.map =================================================================== --- libibverbs.orig/src/libibverbs.map 2008-06-05 15:29:27.000000000 +0300 +++ libibverbs/src/libibverbs.map 2008-06-05 15:36:03.000000000 +0300 @@ -97,6 +97,11 @@ IBVERBS_1.1 { ibv_cmd_open_xrc_domain; ibv_close_xrc_domain; ibv_cmd_close_xrc_domain; + ibv_cmd_create_xrc_rcv_qp; + ibv_cmd_modify_xrc_rcv_qp; + ibv_cmd_query_xrc_rcv_qp; + ibv_cmd_reg_xrc_rcv_qp; + ibv_cmd_unreg_xrc_rcv_qp; ibv_node_type_str; ibv_port_state_str; Index: libibverbs/src/verbs.c =================================================================== --- libibverbs.orig/src/verbs.c 2008-06-05 15:29:27.000000000 +0300 +++ libibverbs/src/verbs.c 2008-06-05 15:36:03.000000000 +0300 @@ -600,3 +600,67 @@ int __ibv_close_xrc_domain(struct ibv_xr return d->context->xrc_ops->close_xrc_domain(d); } default_symver(__ibv_close_xrc_domain, ibv_close_xrc_domain); + +int __ibv_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_rcv_qpn) +{ + struct ibv_context *c; + if (!init_attr || !(init_attr->xrc_domain)) + return EINVAL; + + c = init_attr->xrc_domain->context; + if (!c->xrc_ops) + return ENOSYS; + + return c->xrc_ops->create_xrc_rcv_qp(init_attr, + xrc_rcv_qpn); +} +default_symver(__ibv_create_xrc_rcv_qp, ibv_create_xrc_rcv_qp); + +int __ibv_modify_xrc_rcv_qp(struct ibv_xrc_domain *d, + uint32_t xrc_rcv_qpn, + struct ibv_qp_attr *attr, + int attr_mask) +{ + if (!d || !attr) + return EINVAL; + + if (!d->context->xrc_ops) + return ENOSYS; + + return d->context->xrc_ops->modify_xrc_rcv_qp(d, xrc_rcv_qpn, attr, + attr_mask); +} +default_symver(__ibv_modify_xrc_rcv_qp, ibv_modify_xrc_rcv_qp); + +int __ibv_query_xrc_rcv_qp(struct ibv_xrc_domain *d, + uint32_t xrc_rcv_qpn, + struct ibv_qp_attr *attr, + int attr_mask, + struct ibv_qp_init_attr *init_attr) +{ + if (!d) + return EINVAL; + + if (!d->context->xrc_ops) + return ENOSYS; + + return d->context->xrc_ops->query_xrc_rcv_qp(d, xrc_rcv_qpn, attr, + attr_mask, init_attr); +} +default_symver(__ibv_query_xrc_rcv_qp, ibv_query_xrc_rcv_qp); + +int __ibv_reg_xrc_rcv_qp(struct ibv_xrc_domain *d, + uint32_t xrc_rcv_qpn) +{ + return d->context->xrc_ops->reg_xrc_rcv_qp(d, xrc_rcv_qpn); +} +default_symver(__ibv_reg_xrc_rcv_qp, ibv_reg_xrc_rcv_qp); + +int __ibv_unreg_xrc_rcv_qp(struct ibv_xrc_domain *d, + uint32_t xrc_rcv_qpn) +{ + return d->context->xrc_ops->unreg_xrc_rcv_qp(d, xrc_rcv_qpn); +} +default_symver(__ibv_unreg_xrc_rcv_qp, ibv_unreg_xrc_rcv_qp); + From jackm at dev.mellanox.co.il Mon Jun 23 06:01:32 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:32 +0300 Subject: [ofa-general] [PATCH 4/11] libmlx4: XRC receive-only QP implementation Message-ID: <200806231601.32243.jackm@dev.mellanox.co.il> Added support for XRC receive-only QPs. (OFED 1.3 commit 3869d6dab7e12fe452270ca641f7dd7082b42482) Signed-off-by: Jack Morgenstein Index: libmlx4/src/mlx4.c =================================================================== --- libmlx4.orig/src/mlx4.c 2008-06-23 09:04:50.000000000 +0300 +++ libmlx4/src/mlx4.c 2008-06-23 09:10:13.000000000 +0300 @@ -73,6 +73,11 @@ static struct ibv_xrc_ops mlx4_xrc_ops = .create_xrc_srq = mlx4_create_xrc_srq, .open_xrc_domain = mlx4_open_xrc_domain, .close_xrc_domain = mlx4_close_xrc_domain, + .create_xrc_rcv_qp = mlx4_create_xrc_rcv_qp, + .modify_xrc_rcv_qp = mlx4_modify_xrc_rcv_qp, + .query_xrc_rcv_qp = mlx4_query_xrc_rcv_qp, + .reg_xrc_rcv_qp = mlx4_reg_xrc_rcv_qp, + .unreg_xrc_rcv_qp = mlx4_unreg_xrc_rcv_qp, }; #endif Index: libmlx4/src/mlx4.h =================================================================== --- libmlx4.orig/src/mlx4.h 2008-06-23 09:04:50.000000000 +0300 +++ libmlx4/src/mlx4.h 2008-06-23 09:10:13.000000000 +0300 @@ -407,6 +407,21 @@ struct ibv_xrc_domain *mlx4_open_xrc_dom int fd, int oflag); int mlx4_close_xrc_domain(struct ibv_xrc_domain *d); +int mlx4_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_qp_num); +int mlx4_modify_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, + int attr_mask); +int mlx4_query_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, + int attr_mask, + struct ibv_qp_init_attr *init_attr); +int mlx4_reg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num); +int mlx4_unreg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num); #endif Index: libmlx4/src/verbs.c =================================================================== --- libmlx4.orig/src/verbs.c 2008-06-23 09:04:50.000000000 +0300 +++ libmlx4/src/verbs.c 2008-06-23 09:10:13.000000000 +0300 @@ -770,4 +770,59 @@ int mlx4_close_xrc_domain(struct ibv_xrc free(d); return 0; } + +int mlx4_create_xrc_rcv_qp(struct ibv_qp_init_attr *init_attr, + uint32_t *xrc_qp_num) +{ + + return ibv_cmd_create_xrc_rcv_qp(init_attr, xrc_qp_num); +} + +int mlx4_modify_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, + int attr_mask) +{ + return ibv_cmd_modify_xrc_rcv_qp(xrc_domain, xrc_qp_num, + attr, attr_mask); +} + +int mlx4_query_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num, + struct ibv_qp_attr *attr, + int attr_mask, + struct ibv_qp_init_attr *init_attr) +{ + int ret; + + ret = ibv_cmd_query_xrc_rcv_qp(xrc_domain, xrc_qp_num, + attr, attr_mask, init_attr); + if (ret) + return ret; + + init_attr->cap.max_send_wr = init_attr->cap.max_send_sge = 1; + init_attr->cap.max_recv_sge = init_attr->cap.max_recv_wr = 0; + init_attr->cap.max_inline_data = 0; + init_attr->recv_cq = init_attr->send_cq = NULL; + init_attr->srq = NULL; + init_attr->xrc_domain = xrc_domain; + init_attr->qp_type = IBV_QPT_XRC; + init_attr->qp_context = NULL; + attr->cap = init_attr->cap; + + return 0; +} + +int mlx4_reg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num) +{ + return ibv_cmd_reg_xrc_rcv_qp(xrc_domain, xrc_qp_num); +} + +int mlx4_unreg_xrc_rcv_qp(struct ibv_xrc_domain *xrc_domain, + uint32_t xrc_qp_num) +{ + return ibv_cmd_unreg_xrc_rcv_qp(xrc_domain, xrc_qp_num); +} + #endif From jackm at dev.mellanox.co.il Mon Jun 23 06:01:35 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:35 +0300 Subject: [ofa-general] [PATCH 5/11] core: XRC base implementation Message-ID: <200806231601.35420.jackm@dev.mellanox.co.il> IB/core: Implement XRC support at verbs layer (for case in which fd is not used when opening an xrc_domain). Rev 2: a. is_srq flag indicates ONLY srq. b. Check that for XRC QP's, do not have an SRQ and do have an XRC domain. Signed-off-by: Jack Morgenstein Index: infiniband/drivers/infiniband/core/uverbs_main.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_main.c 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs_main.c 2008-06-23 13:59:52.000000000 +0300 @@ -72,6 +72,7 @@ DEFINE_IDR(ib_uverbs_ah_idr); DEFINE_IDR(ib_uverbs_cq_idr); DEFINE_IDR(ib_uverbs_qp_idr); DEFINE_IDR(ib_uverbs_srq_idr); +DEFINE_IDR(ib_uverbs_xrc_domain_idr); static spinlock_t map_lock; static struct ib_uverbs_device *dev_table[IB_UVERBS_MAX_DEVICES]; @@ -108,6 +109,9 @@ static ssize_t (*uverbs_cmd_table[])(str [IB_USER_VERBS_CMD_MODIFY_SRQ] = ib_uverbs_modify_srq, [IB_USER_VERBS_CMD_QUERY_SRQ] = ib_uverbs_query_srq, [IB_USER_VERBS_CMD_DESTROY_SRQ] = ib_uverbs_destroy_srq, + [IB_USER_VERBS_CMD_CREATE_XRC_SRQ] = ib_uverbs_create_xrc_srq, + [IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN] = ib_uverbs_open_xrc_domain, + [IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN] = ib_uverbs_close_xrc_domain, }; static struct vfsmount *uverbs_event_mnt; @@ -211,17 +215,6 @@ static int ib_uverbs_cleanup_ucontext(st kfree(uqp); } - list_for_each_entry_safe(uobj, tmp, &context->cq_list, list) { - struct ib_cq *cq = uobj->object; - struct ib_uverbs_event_file *ev_file = cq->cq_context; - struct ib_ucq_object *ucq = - container_of(uobj, struct ib_ucq_object, uobject); - - idr_remove_uobj(&ib_uverbs_cq_idr, uobj); - ib_destroy_cq(cq); - ib_uverbs_release_ucq(file, ev_file, ucq); - kfree(ucq); - } list_for_each_entry_safe(uobj, tmp, &context->srq_list, list) { struct ib_srq *srq = uobj->object; @@ -234,6 +227,18 @@ static int ib_uverbs_cleanup_ucontext(st kfree(uevent); } + list_for_each_entry_safe(uobj, tmp, &context->cq_list, list) { + struct ib_cq *cq = uobj->object; + struct ib_uverbs_event_file *ev_file = cq->cq_context; + struct ib_ucq_object *ucq = + container_of(uobj, struct ib_ucq_object, uobject); + + idr_remove_uobj(&ib_uverbs_cq_idr, uobj); + ib_destroy_cq(cq); + ib_uverbs_release_ucq(file, ev_file, ucq); + kfree(ucq); + } + /* XXX Free MWs */ list_for_each_entry_safe(uobj, tmp, &context->mr_list, list) { @@ -244,6 +249,14 @@ static int ib_uverbs_cleanup_ucontext(st kfree(uobj); } + list_for_each_entry_safe(uobj, tmp, &context->xrc_domain_list, list) { + struct ib_xrcd *xrcd = uobj->object; + + idr_remove_uobj(&ib_uverbs_xrc_domain_idr, uobj); + ib_dealloc_xrcd(xrcd); + kfree(uobj); + } + list_for_each_entry_safe(uobj, tmp, &context->pd_list, list) { struct ib_pd *pd = uobj->object; Index: infiniband/include/rdma/ib_user_verbs.h =================================================================== --- infiniband.orig/include/rdma/ib_user_verbs.h 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/include/rdma/ib_user_verbs.h 2008-06-23 13:59:52.000000000 +0300 @@ -81,7 +81,10 @@ enum { IB_USER_VERBS_CMD_MODIFY_SRQ, IB_USER_VERBS_CMD_QUERY_SRQ, IB_USER_VERBS_CMD_DESTROY_SRQ, - IB_USER_VERBS_CMD_POST_SRQ_RECV + IB_USER_VERBS_CMD_POST_SRQ_RECV, + IB_USER_VERBS_CMD_CREATE_XRC_SRQ, + IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN, + IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN }; /* @@ -647,6 +650,18 @@ struct ib_uverbs_create_srq { __u64 driver_data[0]; }; +struct ib_uverbs_create_xrc_srq { + __u64 response; + __u64 user_handle; + __u32 pd_handle; + __u32 max_wr; + __u32 max_sge; + __u32 srq_limit; + __u32 xrcd_handle; + __u32 xrc_cq; + __u64 driver_data[0]; +}; + struct ib_uverbs_create_srq_resp { __u32 srq_handle; __u32 max_wr; @@ -686,4 +701,23 @@ struct ib_uverbs_destroy_srq_resp { __u32 events_reported; }; +struct ib_uverbs_open_xrc_domain { + __u64 response; + __u32 fd; + __u32 oflags; + __u64 driver_data[0]; +}; + +struct ib_uverbs_open_xrc_domain_resp { + __u32 xrcd_handle; +}; + +struct ib_uverbs_close_xrc_domain { + __u64 response; + __u32 xrcd_handle; + __u64 driver_data[0]; +}; + + + #endif /* IB_USER_VERBS_H */ Index: infiniband/drivers/infiniband/core/uverbs_cmd.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_cmd.c 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs_cmd.c 2008-06-23 13:59:52.000000000 +0300 @@ -254,6 +254,16 @@ static void put_srq_read(struct ib_srq * put_uobj_read(srq->uobject); } +static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context) +{ + return idr_read_obj(&ib_uverbs_xrc_domain_idr, xrcd_handle, context, 0); +} + +static void put_xrcd_read(struct ib_xrcd *xrcd) +{ + put_uobj_read(xrcd->uobject); +} + ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file, const char __user *buf, int in_len, int out_len) @@ -297,6 +307,7 @@ ssize_t ib_uverbs_get_context(struct ib_ INIT_LIST_HEAD(&ucontext->qp_list); INIT_LIST_HEAD(&ucontext->srq_list); INIT_LIST_HEAD(&ucontext->ah_list); + INIT_LIST_HEAD(&ucontext->xrc_domain_list); ucontext->closing = 0; resp.num_comp_vectors = file->device->num_comp_vectors; @@ -1026,6 +1037,7 @@ ssize_t ib_uverbs_create_qp(struct ib_uv struct ib_srq *srq; struct ib_qp *qp; struct ib_qp_init_attr attr; + struct ib_xrcd *xrcd; int ret; if (out_len < sizeof resp) @@ -1045,13 +1057,17 @@ ssize_t ib_uverbs_create_qp(struct ib_uv init_uobj(&obj->uevent.uobject, cmd.user_handle, file->ucontext, &qp_lock_key); down_write(&obj->uevent.uobject.mutex); - srq = cmd.is_srq ? idr_read_srq(cmd.srq_handle, file->ucontext) : NULL; + srq = (cmd.is_srq && cmd.qp_type != IB_QPT_XRC) ? + idr_read_srq(cmd.srq_handle, file->ucontext) : NULL; + xrcd = cmd.qp_type == IB_QPT_XRC ? + idr_read_xrcd(cmd.srq_handle, file->ucontext) : NULL; pd = idr_read_pd(cmd.pd_handle, file->ucontext); scq = idr_read_cq(cmd.send_cq_handle, file->ucontext, 0); rcq = cmd.recv_cq_handle == cmd.send_cq_handle ? scq : idr_read_cq(cmd.recv_cq_handle, file->ucontext, 1); - if (!pd || !scq || !rcq || (cmd.is_srq && !srq)) { + if (!pd || !scq || !rcq || cmd.is_srq && !srq || + cmd.qp_type == IB_QPT_XRC && !xrcd) { ret = -EINVAL; goto err_put; } @@ -1063,6 +1079,7 @@ ssize_t ib_uverbs_create_qp(struct ib_uv attr.srq = srq; attr.sq_sig_type = cmd.sq_sig_all ? IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR; attr.qp_type = cmd.qp_type; + attr.xrc_domain = xrcd; attr.create_flags = 0; attr.cap.max_send_wr = cmd.max_send_wr; @@ -1090,11 +1107,14 @@ ssize_t ib_uverbs_create_qp(struct ib_uv qp->event_handler = attr.event_handler; qp->qp_context = attr.qp_context; qp->qp_type = attr.qp_type; + qp->xrcd = attr.xrc_domain; atomic_inc(&pd->usecnt); atomic_inc(&attr.send_cq->usecnt); atomic_inc(&attr.recv_cq->usecnt); if (attr.srq) atomic_inc(&attr.srq->usecnt); + else if (attr.xrc_domain) + atomic_inc(&attr.xrc_domain->usecnt); obj->uevent.uobject.object = qp; ret = idr_add_uobj(&ib_uverbs_qp_idr, &obj->uevent.uobject); @@ -1122,6 +1142,8 @@ ssize_t ib_uverbs_create_qp(struct ib_uv put_cq_read(rcq); if (srq) put_srq_read(srq); + if (xrcd) + put_xrcd_read(xrcd); mutex_lock(&file->mutex); list_add_tail(&obj->uevent.uobject.list, &file->ucontext->qp_list); @@ -1148,6 +1170,8 @@ err_put: put_cq_read(rcq); if (srq) put_srq_read(srq); + if (xrcd) + put_xrcd_read(xrcd); put_uobj_write(&obj->uevent.uobject); return ret; @@ -2000,6 +2024,8 @@ ssize_t ib_uverbs_create_srq(struct ib_u srq->uobject = &obj->uobject; srq->event_handler = attr.event_handler; srq->srq_context = attr.srq_context; + srq->xrc_cq = NULL; + srq->xrcd = NULL; atomic_inc(&pd->usecnt); atomic_set(&srq->usecnt, 0); @@ -2045,6 +2071,135 @@ err: return ret; } +ssize_t ib_uverbs_create_xrc_srq(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_create_xrc_srq cmd; + struct ib_uverbs_create_srq_resp resp; + struct ib_udata udata; + struct ib_uevent_object *obj; + struct ib_pd *pd; + struct ib_srq *srq; + struct ib_cq *xrc_cq; + struct ib_xrcd *xrcd; + struct ib_srq_init_attr attr; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + obj = kmalloc(sizeof *obj, GFP_KERNEL); + if (!obj) + return -ENOMEM; + + init_uobj(&obj->uobject, cmd.user_handle, file->ucontext, &srq_lock_key); + down_write(&obj->uobject.mutex); + + pd = idr_read_pd(cmd.pd_handle, file->ucontext); + if (!pd) { + ret = -EINVAL; + goto err; + } + + xrc_cq = idr_read_cq(cmd.xrc_cq, file->ucontext, 0); + if (!xrc_cq) { + ret = -EINVAL; + goto err_put_pd; + } + + xrcd = idr_read_xrcd(cmd.xrcd_handle, file->ucontext); + if (!xrcd) { + ret = -EINVAL; + goto err_put_cq; + } + + + attr.event_handler = ib_uverbs_srq_event_handler; + attr.srq_context = file; + attr.attr.max_wr = cmd.max_wr; + attr.attr.max_sge = cmd.max_sge; + attr.attr.srq_limit = cmd.srq_limit; + + obj->events_reported = 0; + INIT_LIST_HEAD(&obj->event_list); + + srq = pd->device->create_xrc_srq(pd, xrc_cq, xrcd, &attr, &udata); + if (IS_ERR(srq)) { + ret = PTR_ERR(srq); + goto err_put; + } + + srq->device = pd->device; + srq->pd = pd; + srq->uobject = &obj->uobject; + srq->event_handler = attr.event_handler; + srq->srq_context = attr.srq_context; + srq->xrc_cq = xrc_cq; + srq->xrcd = xrcd; + atomic_inc(&pd->usecnt); + atomic_inc(&xrc_cq->usecnt); + atomic_inc(&xrcd->usecnt); + + atomic_set(&srq->usecnt, 0); + + obj->uobject.object = srq; + ret = idr_add_uobj(&ib_uverbs_srq_idr, &obj->uobject); + if (ret) + goto err_destroy; + + memset(&resp, 0, sizeof resp); + resp.srq_handle = obj->uobject.id; + resp.max_wr = attr.attr.max_wr; + resp.max_sge = attr.attr.max_sge; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + ret = -EFAULT; + goto err_copy; + } + + put_xrcd_read(xrcd); + put_cq_read(xrc_cq); + put_pd_read(pd); + + mutex_lock(&file->mutex); + list_add_tail(&obj->uobject.list, &file->ucontext->srq_list); + mutex_unlock(&file->mutex); + + obj->uobject.live = 1; + + up_write(&obj->uobject.mutex); + + return in_len; + +err_copy: + idr_remove_uobj(&ib_uverbs_srq_idr, &obj->uobject); + +err_destroy: + ib_destroy_srq(srq); + +err_put: + put_xrcd_read(xrcd); + +err_put_cq: + put_cq_read(xrc_cq); + +err_put_pd: + put_pd_read(pd); + +err: + put_uobj_write(&obj->uobject); + return ret; +} + ssize_t ib_uverbs_modify_srq(struct ib_uverbs_file *file, const char __user *buf, int in_len, int out_len) @@ -2163,3 +2318,120 @@ ssize_t ib_uverbs_destroy_srq(struct ib_ return ret ? ret : in_len; } + +ssize_t ib_uverbs_open_xrc_domain(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_open_xrc_domain cmd; + struct ib_uverbs_open_xrc_domain_resp resp; + struct ib_udata udata; + struct ib_uobject *uobj; + struct ib_xrcd *xrcd; + int ret; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + /* file descriptors/inodes not yet implemented */ + if (cmd.fd != (u32) (-1)) + return -ENOSYS; + + INIT_UDATA(&udata, buf + sizeof cmd, + (unsigned long) cmd.response + sizeof resp, + in_len - sizeof cmd, out_len - sizeof resp); + + uobj = kmalloc(sizeof *uobj, GFP_KERNEL); + if (!uobj) + return -ENOMEM; + + init_uobj(uobj, 0, file->ucontext, &pd_lock_key); + down_write(&uobj->mutex); + + + xrcd = file->device->ib_dev->alloc_xrcd(file->device->ib_dev, + file->ucontext, &udata); + if (IS_ERR(xrcd)) { + ret = PTR_ERR(xrcd); + goto err; + } + + xrcd->fd = cmd.fd; + xrcd->flags = cmd.oflags; + xrcd->uobject = uobj; + xrcd->device = file->device->ib_dev; + atomic_set(&xrcd->usecnt, 0); + + uobj->object = xrcd; + ret = idr_add_uobj(&ib_uverbs_xrc_domain_idr, uobj); + if (ret) + goto err_idr; + + memset(&resp, 0, sizeof resp); + resp.xrcd_handle = uobj->id; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + ret = -EFAULT; + goto err_copy; + } + + mutex_lock(&file->mutex); + list_add_tail(&uobj->list, &file->ucontext->xrc_domain_list); + mutex_unlock(&file->mutex); + + uobj->live = 1; + + up_write(&uobj->mutex); + + return in_len; + +err_copy: + idr_remove_uobj(&ib_uverbs_pd_idr, uobj); + +err_idr: + ib_dealloc_xrcd(xrcd); + +err: + put_uobj_write(uobj); + return ret; +} + +ssize_t ib_uverbs_close_xrc_domain(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_close_xrc_domain cmd; + struct ib_uobject *uobj; + int ret; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + uobj = idr_write_uobj(&ib_uverbs_xrc_domain_idr, cmd.xrcd_handle, file->ucontext); + if (!uobj) + return -EINVAL; + + ret = ib_dealloc_xrcd(uobj->object); + if (!ret) + uobj->live = 0; + + put_uobj_write(uobj); + + if (ret) + return ret; + + idr_remove_uobj(&ib_uverbs_xrc_domain_idr, uobj); + + mutex_lock(&file->mutex); + list_del(&uobj->list); + mutex_unlock(&file->mutex); + + put_uobj(uobj); + + return in_len; +} + Index: infiniband/drivers/infiniband/core/verbs.c =================================================================== --- infiniband.orig/drivers/infiniband/core/verbs.c 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/drivers/infiniband/core/verbs.c 2008-06-23 13:59:52.000000000 +0300 @@ -234,6 +234,8 @@ struct ib_srq *ib_create_srq(struct ib_p srq->uobject = NULL; srq->event_handler = srq_init_attr->event_handler; srq->srq_context = srq_init_attr->srq_context; + srq->xrc_cq = NULL; + srq->xrcd = NULL; atomic_inc(&pd->usecnt); atomic_set(&srq->usecnt, 0); } @@ -263,16 +265,25 @@ EXPORT_SYMBOL(ib_query_srq); int ib_destroy_srq(struct ib_srq *srq) { struct ib_pd *pd; + struct ib_cq *xrc_cq; + struct ib_xrcd *xrcd; int ret; if (atomic_read(&srq->usecnt)) return -EBUSY; pd = srq->pd; + xrc_cq = srq->xrc_cq; + xrcd = srq->xrcd; ret = srq->device->destroy_srq(srq); - if (!ret) + if (!ret) { atomic_dec(&pd->usecnt); + if (xrc_cq) + atomic_dec(&xrc_cq->usecnt); + if (xrcd) + atomic_dec(&xrcd->usecnt); + } return ret; } @@ -297,6 +308,7 @@ struct ib_qp *ib_create_qp(struct ib_pd qp->event_handler = qp_init_attr->event_handler; qp->qp_context = qp_init_attr->qp_context; qp->qp_type = qp_init_attr->qp_type; + qp->xrcd = NULL; atomic_inc(&pd->usecnt); atomic_inc(&qp_init_attr->send_cq->usecnt); atomic_inc(&qp_init_attr->recv_cq->usecnt); @@ -327,6 +339,9 @@ static const struct { [IB_QPT_RC] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS), + [IB_QPT_XRC] = (IB_QP_PKEY_INDEX | + IB_QP_PORT | + IB_QP_ACCESS_FLAGS), [IB_QPT_SMI] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_PKEY_INDEX | @@ -349,6 +364,9 @@ static const struct { [IB_QPT_RC] = (IB_QP_PKEY_INDEX | IB_QP_PORT | IB_QP_ACCESS_FLAGS), + [IB_QPT_XRC] = (IB_QP_PKEY_INDEX | + IB_QP_PORT | + IB_QP_ACCESS_FLAGS), [IB_QPT_SMI] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_PKEY_INDEX | @@ -368,6 +386,12 @@ static const struct { IB_QP_RQ_PSN | IB_QP_MAX_DEST_RD_ATOMIC | IB_QP_MIN_RNR_TIMER), + [IB_QPT_XRC] = (IB_QP_AV | + IB_QP_PATH_MTU | + IB_QP_DEST_QPN | + IB_QP_RQ_PSN | + IB_QP_MAX_DEST_RD_ATOMIC | + IB_QP_MIN_RNR_TIMER), }, .opt_param = { [IB_QPT_UD] = (IB_QP_PKEY_INDEX | @@ -378,6 +402,9 @@ static const struct { [IB_QPT_RC] = (IB_QP_ALT_PATH | IB_QP_ACCESS_FLAGS | IB_QP_PKEY_INDEX), + [IB_QPT_XRC] = (IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX), [IB_QPT_SMI] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_PKEY_INDEX | @@ -398,6 +425,11 @@ static const struct { IB_QP_RNR_RETRY | IB_QP_SQ_PSN | IB_QP_MAX_QP_RD_ATOMIC), + [IB_QPT_XRC] = (IB_QP_TIMEOUT | + IB_QP_RETRY_CNT | + IB_QP_RNR_RETRY | + IB_QP_SQ_PSN | + IB_QP_MAX_QP_RD_ATOMIC), [IB_QPT_SMI] = IB_QP_SQ_PSN, [IB_QPT_GSI] = IB_QP_SQ_PSN, }, @@ -413,6 +445,11 @@ static const struct { IB_QP_ACCESS_FLAGS | IB_QP_MIN_RNR_TIMER | IB_QP_PATH_MIG_STATE), + [IB_QPT_XRC] = (IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_MIN_RNR_TIMER | + IB_QP_PATH_MIG_STATE), [IB_QPT_SMI] = (IB_QP_CUR_STATE | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_CUR_STATE | @@ -437,6 +474,11 @@ static const struct { IB_QP_ALT_PATH | IB_QP_PATH_MIG_STATE | IB_QP_MIN_RNR_TIMER), + [IB_QPT_XRC] = (IB_QP_CUR_STATE | + IB_QP_ACCESS_FLAGS | + IB_QP_ALT_PATH | + IB_QP_PATH_MIG_STATE | + IB_QP_MIN_RNR_TIMER), [IB_QPT_SMI] = (IB_QP_CUR_STATE | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_CUR_STATE | @@ -449,6 +491,7 @@ static const struct { [IB_QPT_UD] = IB_QP_EN_SQD_ASYNC_NOTIFY, [IB_QPT_UC] = IB_QP_EN_SQD_ASYNC_NOTIFY, [IB_QPT_RC] = IB_QP_EN_SQD_ASYNC_NOTIFY, + [IB_QPT_XRC] = IB_QP_EN_SQD_ASYNC_NOTIFY, [IB_QPT_SMI] = IB_QP_EN_SQD_ASYNC_NOTIFY, [IB_QPT_GSI] = IB_QP_EN_SQD_ASYNC_NOTIFY } @@ -471,6 +514,11 @@ static const struct { IB_QP_ACCESS_FLAGS | IB_QP_MIN_RNR_TIMER | IB_QP_PATH_MIG_STATE), + [IB_QPT_XRC] = (IB_QP_CUR_STATE | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_MIN_RNR_TIMER | + IB_QP_PATH_MIG_STATE), [IB_QPT_SMI] = (IB_QP_CUR_STATE | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_CUR_STATE | @@ -499,6 +547,18 @@ static const struct { IB_QP_PKEY_INDEX | IB_QP_MIN_RNR_TIMER | IB_QP_PATH_MIG_STATE), + [IB_QPT_XRC] = (IB_QP_PORT | + IB_QP_AV | + IB_QP_TIMEOUT | + IB_QP_RETRY_CNT | + IB_QP_RNR_RETRY | + IB_QP_MAX_QP_RD_ATOMIC | + IB_QP_MAX_DEST_RD_ATOMIC | + IB_QP_ALT_PATH | + IB_QP_ACCESS_FLAGS | + IB_QP_PKEY_INDEX | + IB_QP_MIN_RNR_TIMER | + IB_QP_PATH_MIG_STATE), [IB_QPT_SMI] = (IB_QP_PKEY_INDEX | IB_QP_QKEY), [IB_QPT_GSI] = (IB_QP_PKEY_INDEX | @@ -583,12 +643,14 @@ int ib_destroy_qp(struct ib_qp *qp) struct ib_pd *pd; struct ib_cq *scq, *rcq; struct ib_srq *srq; + struct ib_xrcd *xrcd; int ret; pd = qp->pd; scq = qp->send_cq; rcq = qp->recv_cq; srq = qp->srq; + xrcd = qp->xrcd; ret = qp->device->destroy_qp(qp); if (!ret) { @@ -597,6 +659,8 @@ int ib_destroy_qp(struct ib_qp *qp) atomic_dec(&rcq->usecnt); if (srq) atomic_dec(&srq->usecnt); + if (xrcd) + atomic_dec(&xrcd->usecnt); } return ret; @@ -904,3 +968,14 @@ int ib_detach_mcast(struct ib_qp *qp, un return qp->device->detach_mcast(qp, gid, lid); } EXPORT_SYMBOL(ib_detach_mcast); + +int ib_dealloc_xrcd(struct ib_xrcd *xrcd) +{ + if (atomic_read(&xrcd->usecnt)) + return -EBUSY; + + return xrcd->device->dealloc_xrcd(xrcd); +} +EXPORT_SYMBOL(ib_dealloc_xrcd); + + Index: infiniband/include/rdma/ib_verbs.h =================================================================== --- infiniband.orig/include/rdma/ib_verbs.h 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/include/rdma/ib_verbs.h 2008-06-23 13:59:52.000000000 +0300 @@ -103,6 +103,7 @@ enum ib_device_cap_flags { */ IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), + IB_DEVICE_XRC = (1<<20), IB_DEVICE_MEM_MGT_EXTENSIONS = (1<<21), }; @@ -499,6 +500,7 @@ enum ib_qp_type { IB_QPT_RC, IB_QPT_UC, IB_QPT_UD, + IB_QPT_XRC, IB_QPT_RAW_IPV6, IB_QPT_RAW_ETY }; @@ -517,6 +519,7 @@ struct ib_qp_init_attr { enum ib_sig_type sq_sig_type; enum ib_qp_type qp_type; enum ib_qp_create_flags create_flags; + struct ib_xrcd *xrc_domain; /* XRC qp's only */ u8 port_num; /* special QP types only */ }; @@ -762,6 +765,7 @@ struct ib_ucontext { struct list_head qp_list; struct list_head srq_list; struct list_head ah_list; + struct list_head xrc_domain_list; int closing; }; @@ -789,6 +793,18 @@ struct ib_pd { atomic_t usecnt; /* count all resources */ }; +struct ib_xrcd { + struct ib_device *device; + struct ib_uobject *uobject; + struct rb_node node; + u32 xrc_domain_num; + struct inode *inode; + int fd; + u32 flags; + atomic_t usecnt; /* count all resources */ +}; + + struct ib_ah { struct ib_device *device; struct ib_pd *pd; @@ -810,6 +826,8 @@ struct ib_cq { struct ib_srq { struct ib_device *device; struct ib_pd *pd; + struct ib_cq *xrc_cq; + struct ib_xrcd *xrcd; struct ib_uobject *uobject; void (*event_handler)(struct ib_event *, void *); void *srq_context; @@ -827,6 +845,7 @@ struct ib_qp { void *qp_context; u32 qp_num; enum ib_qp_type qp_type; + struct ib_xrcd *xrcd; /* XRC QPs only */ }; struct ib_mr { @@ -1076,6 +1095,15 @@ struct ib_device { struct ib_grh *in_grh, struct ib_mad *in_mad, struct ib_mad *out_mad); + struct ib_srq * (*create_xrc_srq)(struct ib_pd *pd, + struct ib_cq *xrc_cq, + struct ib_xrcd *xrcd, + struct ib_srq_init_attr *srq_init_attr, + struct ib_udata *udata); + struct ib_xrcd * (*alloc_xrcd)(struct ib_device *device, + struct ib_ucontext *context, + struct ib_udata *udata); + int (*dealloc_xrcd)(struct ib_xrcd *xrcd); struct ib_dma_mapping_ops *dma_ops; @@ -1976,4 +2004,11 @@ int ib_attach_mcast(struct ib_qp *qp, un */ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid); + +/** + * ib_dealloc_xrcd - Deallocates an extended reliably connected domain. + * @pd: The xrc domain to deallocate. + */ +int ib_dealloc_xrcd(struct ib_xrcd *xrcd); + #endif /* IB_VERBS_H */ Index: infiniband/drivers/infiniband/core/uverbs.h =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs.h 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs.h 2008-06-23 13:59:52.000000000 +0300 @@ -141,6 +141,7 @@ extern struct idr ib_uverbs_ah_idr; extern struct idr ib_uverbs_cq_idr; extern struct idr ib_uverbs_qp_idr; extern struct idr ib_uverbs_srq_idr; +extern struct idr ib_uverbs_xrc_domain_idr; void idr_remove_uobj(struct idr *idp, struct ib_uobject *uobj); @@ -194,5 +195,9 @@ IB_UVERBS_DECLARE_CMD(create_srq); IB_UVERBS_DECLARE_CMD(modify_srq); IB_UVERBS_DECLARE_CMD(query_srq); IB_UVERBS_DECLARE_CMD(destroy_srq); +IB_UVERBS_DECLARE_CMD(create_xrc_srq); +IB_UVERBS_DECLARE_CMD(open_xrc_domain); +IB_UVERBS_DECLARE_CMD(close_xrc_domain); + #endif /* UVERBS_H */ From jackm at dev.mellanox.co.il Mon Jun 23 06:01:37 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:37 +0300 Subject: [ofa-general] [PATCH 6/11] core: XRC - file descriptors associated with XRC domains Message-ID: <200806231601.38023.jackm@dev.mellanox.co.il> Add XRC support for working with file descriptors, to allow sharing XRC domains between processes. Signed-off-by: Jack Morgenstein Index: infiniband/drivers/infiniband/core/uverbs_cmd.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_cmd.c 2008-06-23 13:59:52.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs_cmd.c 2008-06-23 14:00:01.000000000 +0300 @@ -37,6 +37,7 @@ #include #include +#include #include "uverbs.h" @@ -254,14 +255,18 @@ static void put_srq_read(struct ib_srq * put_uobj_read(srq->uobject); } -static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, struct ib_ucontext *context) +static struct ib_xrcd *idr_read_xrcd(int xrcd_handle, + struct ib_ucontext *context, + struct ib_uobject **uobj) { - return idr_read_obj(&ib_uverbs_xrc_domain_idr, xrcd_handle, context, 0); + *uobj = idr_read_uobj(&ib_uverbs_xrc_domain_idr, xrcd_handle, + context, 0); + return *uobj ? (*uobj)->object : NULL; } -static void put_xrcd_read(struct ib_xrcd *xrcd) +static void put_xrcd_read(struct ib_uobject *uobj) { - put_uobj_read(xrcd->uobject); + put_uobj_read(uobj); } ssize_t ib_uverbs_get_context(struct ib_uverbs_file *file, @@ -1038,6 +1043,7 @@ ssize_t ib_uverbs_create_qp(struct ib_uv struct ib_qp *qp; struct ib_qp_init_attr attr; struct ib_xrcd *xrcd; + struct ib_uobject *xrcd_uobj; int ret; if (out_len < sizeof resp) @@ -1060,7 +1066,7 @@ ssize_t ib_uverbs_create_qp(struct ib_uv srq = (cmd.is_srq && cmd.qp_type != IB_QPT_XRC) ? idr_read_srq(cmd.srq_handle, file->ucontext) : NULL; xrcd = cmd.qp_type == IB_QPT_XRC ? - idr_read_xrcd(cmd.srq_handle, file->ucontext) : NULL; + idr_read_xrcd(cmd.srq_handle, file->ucontext, &xrcd_uobj) : NULL; pd = idr_read_pd(cmd.pd_handle, file->ucontext); scq = idr_read_cq(cmd.send_cq_handle, file->ucontext, 0); rcq = cmd.recv_cq_handle == cmd.send_cq_handle ? @@ -1143,7 +1149,7 @@ ssize_t ib_uverbs_create_qp(struct ib_uv if (srq) put_srq_read(srq); if (xrcd) - put_xrcd_read(xrcd); + put_xrcd_read(xrcd_uobj); mutex_lock(&file->mutex); list_add_tail(&obj->uevent.uobject.list, &file->ucontext->qp_list); @@ -1171,7 +1177,7 @@ err_put: if (srq) put_srq_read(srq); if (xrcd) - put_xrcd_read(xrcd); + put_xrcd_read(xrcd_uobj); put_uobj_write(&obj->uevent.uobject); return ret; @@ -2084,6 +2090,7 @@ ssize_t ib_uverbs_create_xrc_srq(struct struct ib_cq *xrc_cq; struct ib_xrcd *xrcd; struct ib_srq_init_attr attr; + struct ib_uobject *xrcd_uobj; int ret; if (out_len < sizeof resp) @@ -2115,7 +2122,7 @@ ssize_t ib_uverbs_create_xrc_srq(struct goto err_put_pd; } - xrcd = idr_read_xrcd(cmd.xrcd_handle, file->ucontext); + xrcd = idr_read_xrcd(cmd.xrcd_handle, file->ucontext, &xrcd_uobj); if (!xrcd) { ret = -EINVAL; goto err_put_cq; @@ -2166,7 +2173,7 @@ ssize_t ib_uverbs_create_xrc_srq(struct goto err_copy; } - put_xrcd_read(xrcd); + put_xrcd_read(xrcd_uobj); put_cq_read(xrc_cq); put_pd_read(pd); @@ -2187,7 +2194,7 @@ err_destroy: ib_destroy_srq(srq); err_put: - put_xrcd_read(xrcd); + put_xrcd_read(xrcd_uobj); err_put_cq: put_cq_read(xrc_cq); @@ -2319,6 +2326,117 @@ ssize_t ib_uverbs_destroy_srq(struct ib_ return ret ? ret : in_len; } +static struct inode * xrc_fd2inode(unsigned int fd) +{ + struct file * f = fget(fd); + + if (!f) + return NULL; + + return f->f_dentry->d_inode; +} + +struct xrcd_table_entry { + struct rb_node node; + struct inode * inode; + struct ib_xrcd *xrcd; +}; + +static int xrcd_table_insert(struct ib_device *dev, + struct inode *i_n, + struct ib_xrcd *xrcd) +{ + struct xrcd_table_entry *entry, *scan; + struct rb_node **p = &dev->ib_uverbs_xrcd_table.rb_node; + struct rb_node *parent = NULL; + + entry = kmalloc(sizeof(struct xrcd_table_entry), GFP_KERNEL); + if (!entry) + return -ENOMEM; + + entry->inode = i_n; + entry->xrcd = xrcd; + + while (*p) + { + parent = *p; + scan = rb_entry(parent, struct xrcd_table_entry, node); + + if (i_n < scan->inode) + p = &(*p)->rb_left; + else if (i_n > scan->inode) + p = &(*p)->rb_right; + else { + kfree(entry); + return -EEXIST; + } + } + + rb_link_node(&entry->node, parent, p); + rb_insert_color(&entry->node, &dev->ib_uverbs_xrcd_table); + return 0; +} + +static int insert_xrcd(struct ib_device *dev, struct inode *i_n, + struct ib_xrcd *xrcd) +{ + int ret; + + ret = xrcd_table_insert(dev, i_n, xrcd); + if (!ret) + igrab(i_n); + + return ret; +} + +static struct xrcd_table_entry * xrcd_table_search(struct ib_device *dev, + struct inode *i_n) +{ + struct xrcd_table_entry *scan; + struct rb_node **p = &dev->ib_uverbs_xrcd_table.rb_node; + struct rb_node *parent = NULL; + + while (*p) + { + parent = *p; + scan = rb_entry(parent, struct xrcd_table_entry, node); + + if (i_n < scan->inode) + p = &(*p)->rb_left; + else if (i_n > scan->inode) + p = &(*p)->rb_right; + else + return scan; + } + return NULL; +} + +static int find_xrcd(struct ib_device *dev, struct inode *i_n, + struct ib_xrcd **xrcd) +{ + struct xrcd_table_entry *entry; + + entry = xrcd_table_search(dev, i_n); + if (!entry) + return -EINVAL; + + *xrcd = entry->xrcd; + return 0; +} + + +static void xrcd_table_delete(struct ib_device *dev, + struct inode *i_n) +{ + struct xrcd_table_entry *entry = xrcd_table_search(dev, i_n); + + if (entry) { + iput(i_n); + rb_erase(&entry->node, &dev->ib_uverbs_xrcd_table); + kfree(entry); + } +} + ssize_t ib_uverbs_open_xrc_domain(struct ib_uverbs_file *file, const char __user *buf, int in_len, int out_len) @@ -2327,8 +2445,10 @@ ssize_t ib_uverbs_open_xrc_domain(struct struct ib_uverbs_open_xrc_domain_resp resp; struct ib_udata udata; struct ib_uobject *uobj; - struct ib_xrcd *xrcd; - int ret; + struct ib_xrcd *xrcd = NULL; + struct inode *inode = NULL; + int ret = 0; + int new_xrcd = 0; if (out_len < sizeof resp) return -ENOSPC; @@ -2336,35 +2456,55 @@ ssize_t ib_uverbs_open_xrc_domain(struct if (copy_from_user(&cmd, buf, sizeof cmd)) return -EFAULT; - /* file descriptors/inodes not yet implemented */ - if (cmd.fd != (u32) (-1)) - return -ENOSYS; - INIT_UDATA(&udata, buf + sizeof cmd, (unsigned long) cmd.response + sizeof resp, in_len - sizeof cmd, out_len - sizeof resp); + mutex_lock(&file->device->ib_dev->xrcd_table_mutex); + if (cmd.fd != (u32) (-1)) { + /* search for file descriptor */ + inode = xrc_fd2inode(cmd.fd); + if (!inode) { + ret = -EBADF; + goto err_table_mutex_unlock; + } + + ret = find_xrcd(file->device->ib_dev, inode, &xrcd); + if (ret && !(cmd.oflags & O_CREAT)) { + /* no file descriptor. Need CREATE flag */ + ret = -EAGAIN; + goto err_table_mutex_unlock; + } + + if (xrcd && cmd.oflags & O_EXCL){ + ret = -EINVAL; + goto err_table_mutex_unlock; + } + } + uobj = kmalloc(sizeof *uobj, GFP_KERNEL); - if (!uobj) - return -ENOMEM; + if (!uobj) { + ret = -ENOMEM; + goto err_table_mutex_unlock; + } init_uobj(uobj, 0, file->ucontext, &pd_lock_key); down_write(&uobj->mutex); - - xrcd = file->device->ib_dev->alloc_xrcd(file->device->ib_dev, - file->ucontext, &udata); - if (IS_ERR(xrcd)) { - ret = PTR_ERR(xrcd); - goto err; + if (!xrcd) { + xrcd = file->device->ib_dev->alloc_xrcd(file->device->ib_dev, + file->ucontext, &udata); + if (IS_ERR(xrcd)) { + ret = PTR_ERR(xrcd); + goto err; + } + xrcd->uobject = (cmd.fd == -1) ? uobj : NULL; + xrcd->inode = inode; + xrcd->device = file->device->ib_dev; + atomic_set(&xrcd->usecnt, 0); + new_xrcd = 1; } - xrcd->fd = cmd.fd; - xrcd->flags = cmd.oflags; - xrcd->uobject = uobj; - xrcd->device = file->device->ib_dev; - atomic_set(&xrcd->usecnt, 0); - uobj->object = xrcd; ret = idr_add_uobj(&ib_uverbs_xrc_domain_idr, uobj); if (ret) @@ -2373,6 +2513,16 @@ ssize_t ib_uverbs_open_xrc_domain(struct memset(&resp, 0, sizeof resp); resp.xrcd_handle = uobj->id; + if (inode) { + if (new_xrcd) { + /* create new inode/xrcd table entry */ + ret = insert_xrcd(file->device->ib_dev, inode, xrcd); + if (ret) + goto err_insert_xrcd; + } + atomic_inc(&xrcd->usecnt); + } + if (copy_to_user((void __user *) (unsigned long) cmd.response, &resp, sizeof resp)) { ret = -EFAULT; @@ -2387,16 +2537,29 @@ ssize_t ib_uverbs_open_xrc_domain(struct up_write(&uobj->mutex); + mutex_unlock(&file->device->ib_dev->xrcd_table_mutex); return in_len; err_copy: - idr_remove_uobj(&ib_uverbs_pd_idr, uobj); + + if (inode) { + if (new_xrcd) + xrcd_table_delete(file->device->ib_dev, inode); + atomic_dec(&xrcd->usecnt); + } + +err_insert_xrcd: + idr_remove_uobj(&ib_uverbs_xrc_domain_idr, uobj); err_idr: ib_dealloc_xrcd(xrcd); err: put_uobj_write(uobj); + +err_table_mutex_unlock: + + mutex_unlock(&file->device->ib_dev->xrcd_table_mutex); return ret; } @@ -2406,14 +2569,25 @@ ssize_t ib_uverbs_close_xrc_domain(struc { struct ib_uverbs_close_xrc_domain cmd; struct ib_uobject *uobj; - int ret; + struct ib_xrcd *xrcd = NULL; + struct inode *inode = NULL; + int ret = 0; if (copy_from_user(&cmd, buf, sizeof cmd)) return -EFAULT; + mutex_lock(&file->device->ib_dev->xrcd_table_mutex); uobj = idr_write_uobj(&ib_uverbs_xrc_domain_idr, cmd.xrcd_handle, file->ucontext); - if (!uobj) - return -EINVAL; + if (!uobj) { + ret = -EINVAL; + goto err_unlock_mutex; + } + + xrcd = (struct ib_xrcd *) (uobj->object); + inode = xrcd->inode; + + if (inode) + atomic_dec(&xrcd->usecnt); ret = ib_dealloc_xrcd(uobj->object); if (!ret) @@ -2421,8 +2595,11 @@ ssize_t ib_uverbs_close_xrc_domain(struc put_uobj_write(uobj); - if (ret) - return ret; + if (ret && !inode) + goto err_unlock_mutex; + + if (!ret && inode) + xrcd_table_delete(file->device->ib_dev, inode); idr_remove_uobj(&ib_uverbs_xrc_domain_idr, uobj); @@ -2432,6 +2609,27 @@ ssize_t ib_uverbs_close_xrc_domain(struc put_uobj(uobj); + mutex_unlock(&file->device->ib_dev->xrcd_table_mutex); return in_len; + +err_unlock_mutex: + mutex_unlock(&file->device->ib_dev->xrcd_table_mutex); + return ret; } +void ib_uverbs_dealloc_xrcd(struct ib_device *ib_dev, + struct ib_xrcd *xrcd) +{ + struct inode *inode = NULL; + int ret = 0; + + inode = xrcd->inode; + if (inode) + atomic_dec(&xrcd->usecnt); + + ret = ib_dealloc_xrcd(xrcd); + if (!ret && inode) + xrcd_table_delete(ib_dev, inode); +} + + Index: infiniband/include/rdma/ib_verbs.h =================================================================== --- infiniband.orig/include/rdma/ib_verbs.h 2008-06-23 13:59:52.000000000 +0300 +++ infiniband/include/rdma/ib_verbs.h 2008-06-23 14:00:01.000000000 +0300 @@ -50,6 +50,8 @@ #include #include +#include +#include union ib_gid { u8 raw[16]; @@ -796,11 +798,8 @@ struct ib_pd { struct ib_xrcd { struct ib_device *device; struct ib_uobject *uobject; - struct rb_node node; - u32 xrc_domain_num; struct inode *inode; - int fd; - u32 flags; + struct rb_node node; atomic_t usecnt; /* count all resources */ }; @@ -1125,6 +1124,8 @@ struct ib_device { __be64 node_guid; u8 node_type; u8 phys_port_cnt; + struct rb_root ib_uverbs_xrcd_table; + struct mutex xrcd_table_mutex; }; struct ib_client { Index: infiniband/drivers/infiniband/core/device.c =================================================================== --- infiniband.orig/drivers/infiniband/core/device.c 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/drivers/infiniband/core/device.c 2008-06-23 14:00:01.000000000 +0300 @@ -288,6 +288,8 @@ int ib_register_device(struct ib_device INIT_LIST_HEAD(&device->client_data_list); spin_lock_init(&device->event_handler_lock); spin_lock_init(&device->client_data_lock); + device->ib_uverbs_xrcd_table = RB_ROOT; + mutex_init(&device->xrcd_table_mutex); ret = read_port_table_lengths(device); if (ret) { Index: infiniband/drivers/infiniband/core/uverbs_main.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_main.c 2008-06-23 13:59:52.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs_main.c 2008-06-23 14:00:01.000000000 +0300 @@ -249,13 +249,15 @@ static int ib_uverbs_cleanup_ucontext(st kfree(uobj); } + mutex_lock(&file->device->ib_dev->xrcd_table_mutex); list_for_each_entry_safe(uobj, tmp, &context->xrc_domain_list, list) { struct ib_xrcd *xrcd = uobj->object; idr_remove_uobj(&ib_uverbs_xrc_domain_idr, uobj); - ib_dealloc_xrcd(xrcd); + ib_uverbs_dealloc_xrcd(file->device->ib_dev, xrcd); kfree(uobj); } + mutex_unlock(&file->device->ib_dev->xrcd_table_mutex); list_for_each_entry_safe(uobj, tmp, &context->pd_list, list) { struct ib_pd *pd = uobj->object; Index: infiniband/drivers/infiniband/core/uverbs.h =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs.h 2008-06-23 13:59:52.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs.h 2008-06-23 14:00:01.000000000 +0300 @@ -161,6 +161,8 @@ void ib_uverbs_qp_event_handler(struct i void ib_uverbs_srq_event_handler(struct ib_event *event, void *context_ptr); void ib_uverbs_event_handler(struct ib_event_handler *handler, struct ib_event *event); +void ib_uverbs_dealloc_xrcd(struct ib_device *ib_dev, + struct ib_xrcd *xrcd); #define IB_UVERBS_DECLARE_CMD(name) \ ssize_t ib_uverbs_##name(struct ib_uverbs_file *file, \ From jackm at dev.mellanox.co.il Mon Jun 23 06:01:40 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:40 +0300 Subject: [ofa-general] [PATCH 7/11] core: kernel-space application XRC QPs Message-ID: <200806231601.40952.jackm@dev.mellanox.co.il> IB/core: implement kernel-space XRC. Signed-off-by: Jack Morgenstein Index: infiniband/include/rdma/ib_verbs.h =================================================================== --- infiniband.orig/include/rdma/ib_verbs.h 2008-06-23 14:00:01.000000000 +0300 +++ infiniband/include/rdma/ib_verbs.h 2008-06-23 14:00:09.000000000 +0300 @@ -706,6 +706,7 @@ struct ib_send_wr { u32 rkey; } fast_reg; } wr; + u32 xrc_remote_srq_num; /* valid for XRC sends only */ }; struct ib_recv_wr { @@ -831,6 +832,7 @@ struct ib_srq { void (*event_handler)(struct ib_event *, void *); void *srq_context; atomic_t usecnt; + u32 xrc_srq_num; }; struct ib_qp { @@ -1286,8 +1288,28 @@ int ib_query_ah(struct ib_ah *ah, struct int ib_destroy_ah(struct ib_ah *ah); /** - * ib_create_srq - Creates a SRQ associated with the specified protection - * domain. + * ib_create_xrc_srq - Creates an XRC SRQ associated with the specified + * protection domain, cq, and xrc domain. + * @pd: The protection domain associated with the SRQ. + * @xrc_cq: The cq to be associated with the XRC SRQ. + * @xrcd: The XRC domain to be associated with the XRC SRQ. + * @srq_init_attr: A list of initial attributes required to create the + * XRC SRQ. If XRC SRQ creation succeeds, then the attributes are updated + * to the actual capabilities of the created XRC SRQ. + * + * srq_attr->max_wr and srq_attr->max_sge are read the determine the + * requested size of the XRC SRQ, and set to the actual values allocated + * on return. If ib_create_xrc_srq() succeeds, then max_wr and max_sge + * will always be at least as large as the requested values. + */ +struct ib_srq *ib_create_xrc_srq(struct ib_pd *pd, + struct ib_cq *xrc_cq, + struct ib_xrcd *xrcd, + struct ib_srq_init_attr *srq_init_attr); + +/** + * ib_create_srq - Creates an SRQ associated with the specified + * protection domain. * @pd: The protection domain associated with the SRQ. * @srq_init_attr: A list of initial attributes required to create the * SRQ. If SRQ creation succeeds, then the attributes are updated to @@ -2008,8 +2030,14 @@ int ib_detach_mcast(struct ib_qp *qp, un /** * ib_dealloc_xrcd - Deallocates an extended reliably connected domain. - * @pd: The xrc domain to deallocate. + * @xrcd: The xrc domain to deallocate. */ int ib_dealloc_xrcd(struct ib_xrcd *xrcd); +/** + * ib_alloc_xrcd - Allocates an extended reliably connected domain. + * @device: The device on which to allocate the xrcd. + */ +struct ib_xrcd * ib_alloc_xrcd(struct ib_device *device); + #endif /* IB_VERBS_H */ Index: infiniband/drivers/infiniband/core/verbs.c =================================================================== --- infiniband.orig/drivers/infiniband/core/verbs.c 2008-06-23 13:59:52.000000000 +0300 +++ infiniband/drivers/infiniband/core/verbs.c 2008-06-23 14:00:09.000000000 +0300 @@ -244,6 +244,36 @@ struct ib_srq *ib_create_srq(struct ib_p } EXPORT_SYMBOL(ib_create_srq); +struct ib_srq *ib_create_xrc_srq(struct ib_pd *pd, + struct ib_cq *xrc_cq, + struct ib_xrcd *xrcd, + struct ib_srq_init_attr *srq_init_attr) +{ + struct ib_srq *srq; + + if (!pd->device->create_xrc_srq) + return ERR_PTR(-ENOSYS); + + srq = pd->device->create_xrc_srq(pd, xrc_cq, xrcd, srq_init_attr, NULL); + + if (!IS_ERR(srq)) { + srq->device = pd->device; + srq->pd = pd; + srq->uobject = NULL; + srq->event_handler = srq_init_attr->event_handler; + srq->srq_context = srq_init_attr->srq_context; + srq->xrc_cq = xrc_cq; + srq->xrcd = xrcd; + atomic_inc(&pd->usecnt); + atomic_inc(&xrcd->usecnt); + atomic_inc(&xrc_cq->usecnt); + atomic_set(&srq->usecnt, 0); + } + + return srq; +} +EXPORT_SYMBOL(ib_create_xrc_srq); + int ib_modify_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr, enum ib_srq_attr_mask srq_attr_mask) @@ -308,12 +338,15 @@ struct ib_qp *ib_create_qp(struct ib_pd qp->event_handler = qp_init_attr->event_handler; qp->qp_context = qp_init_attr->qp_context; qp->qp_type = qp_init_attr->qp_type; - qp->xrcd = NULL; + qp->xrcd = qp->qp_type == IB_QPT_XRC ? + qp_init_attr->xrc_domain : NULL; atomic_inc(&pd->usecnt); atomic_inc(&qp_init_attr->send_cq->usecnt); atomic_inc(&qp_init_attr->recv_cq->usecnt); if (qp_init_attr->srq) atomic_inc(&qp_init_attr->srq->usecnt); + if (qp->qp_type == IB_QPT_XRC) + atomic_inc(&qp->xrcd->usecnt); } return qp; @@ -644,6 +677,7 @@ int ib_destroy_qp(struct ib_qp *qp) struct ib_cq *scq, *rcq; struct ib_srq *srq; struct ib_xrcd *xrcd; + enum ib_qp_type qp_type = qp->qp_type; int ret; pd = qp->pd; @@ -659,7 +693,7 @@ int ib_destroy_qp(struct ib_qp *qp) atomic_dec(&rcq->usecnt); if (srq) atomic_dec(&srq->usecnt); - if (xrcd) + if (qp_type == IB_QPT_XRC) atomic_dec(&xrcd->usecnt); } @@ -978,4 +1012,22 @@ int ib_dealloc_xrcd(struct ib_xrcd *xrcd } EXPORT_SYMBOL(ib_dealloc_xrcd); +struct ib_xrcd * ib_alloc_xrcd(struct ib_device *device) +{ + struct ib_xrcd *xrcd; + + if (!device->alloc_xrcd) + return ERR_PTR(-ENOSYS); + + xrcd = device->alloc_xrcd(device, NULL, NULL); + if (!IS_ERR(xrcd)) { + xrcd->device = device; + xrcd->inode = NULL; + xrcd->uobject = NULL; + atomic_set(&xrcd->usecnt, 0); + } + return xrcd; +} +EXPORT_SYMBOL(ib_alloc_xrcd); + From jackm at dev.mellanox.co.il Mon Jun 23 06:01:43 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:43 +0300 Subject: [ofa-general] [PATCH 8/11] core: XRC receive-only QPs Message-ID: <200806231601.43706.jackm@dev.mellanox.co.il> ib/core: Implement XRC receive-only QPs for userspace apps. Added creation of XRC receive-only QPs for userspace, which reside in kernel space (user cannot post-to or poll these QPs). Motivation: MPI community requires XRC receive QPs which will not be destroyed when the creating process terminates. Solution: Userspace requests that a QP be created in kernel space. Each userspace process using that QP (i.e. receiving packets on an XRC SRQ via the qp), registers with that QP (-- the creator is also registered, whether or not it is a user of the QP). When the last userspace user unregisters with the QP, it is destroyed. Unregistration is also part of userspace process cleanup, so there is no leakage. This patch implements the kernel procedures to implement the following (new) libibverbs API: ibv_create_xrc_rcv_qp ibv_modify_xrc_rcv_qp ibv_query_xrc_rcv_qp ibv_reg_xrc_rcv_qp ibv_unreg_xrc_rcv_qp In addition, the patch implements the foundation for distributing XRC-receive-only QP events to userspace processes registered with that QP. Finally, the patch modifies ib_uverbs_close_xrc_domain() to return BUSY if any resources are still in use by the process, so that the XRC rcv-only QP cleanup can operate properly. Signed-off-by: Jack Morgenstein Index: infiniband/include/rdma/ib_verbs.h =================================================================== --- infiniband.orig/include/rdma/ib_verbs.h 2008-06-23 14:08:12.000000000 +0300 +++ infiniband/include/rdma/ib_verbs.h 2008-06-23 14:08:14.000000000 +0300 @@ -294,6 +294,10 @@ enum ib_event_type { IB_EVENT_CLIENT_REREGISTER }; +enum ib_event_flags { + IB_XRC_QP_EVENT_FLAG = 0x80000000, +}; + struct ib_event { struct ib_device *device; union { @@ -301,6 +305,7 @@ struct ib_event { struct ib_qp *qp; struct ib_srq *srq; u8 port_num; + u32 xrc_qp_num; } element; enum ib_event_type event; }; @@ -509,6 +514,7 @@ enum ib_qp_type { enum ib_qp_create_flags { IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + QP_CREATE_XRC_RCV = 1 << 1, }; struct ib_qp_init_attr { @@ -769,6 +775,7 @@ struct ib_ucontext { struct list_head srq_list; struct list_head ah_list; struct list_head xrc_domain_list; + struct list_head xrc_reg_qp_list; int closing; }; @@ -790,6 +797,12 @@ struct ib_udata { size_t outlen; }; +struct ib_uxrc_rcv_object { + struct list_head list; /* link to context's list */ + u32 qp_num; + u32 domain_handle; +}; + struct ib_pd { struct ib_device *device; struct ib_uobject *uobject; @@ -1105,6 +1118,23 @@ struct ib_device { struct ib_ucontext *context, struct ib_udata *udata); int (*dealloc_xrcd)(struct ib_xrcd *xrcd); + int (*create_xrc_rcv_qp)(struct ib_qp_init_attr *init_attr, + u32* qp_num); + int (*modify_xrc_rcv_qp)(struct ib_xrcd *xrcd, + u32 qp_num, + struct ib_qp_attr *attr, + int attr_mask); + int (*query_xrc_rcv_qp)(struct ib_xrcd *xrcd, + u32 qp_num, + struct ib_qp_attr *attr, + int attr_mask, + struct ib_qp_init_attr *init_attr); + int (*reg_xrc_rcv_qp)(struct ib_xrcd *xrcd, + void *context, + u32 qp_num); + int (*unreg_xrc_rcv_qp)(struct ib_xrcd *xrcd, + void *context, + u32 qp_num); struct ib_dma_mapping_ops *dma_ops; Index: infiniband/drivers/infiniband/core/uverbs_main.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_main.c 2008-06-23 14:08:11.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs_main.c 2008-06-23 14:08:14.000000000 +0300 @@ -112,6 +112,11 @@ static ssize_t (*uverbs_cmd_table[])(str [IB_USER_VERBS_CMD_CREATE_XRC_SRQ] = ib_uverbs_create_xrc_srq, [IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN] = ib_uverbs_open_xrc_domain, [IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN] = ib_uverbs_close_xrc_domain, + [IB_USER_VERBS_CMD_CREATE_XRC_RCV_QP] = ib_uverbs_create_xrc_rcv_qp, + [IB_USER_VERBS_CMD_MODIFY_XRC_RCV_QP] = ib_uverbs_modify_xrc_rcv_qp, + [IB_USER_VERBS_CMD_QUERY_XRC_RCV_QP] = ib_uverbs_query_xrc_rcv_qp, + [IB_USER_VERBS_CMD_REG_XRC_RCV_QP] = ib_uverbs_reg_xrc_rcv_qp, + [IB_USER_VERBS_CMD_UNREG_XRC_RCV_QP] = ib_uverbs_unreg_xrc_rcv_qp, }; static struct vfsmount *uverbs_event_mnt; @@ -189,6 +194,7 @@ static int ib_uverbs_cleanup_ucontext(st struct ib_ucontext *context) { struct ib_uobject *uobj, *tmp; + struct ib_uxrc_rcv_object *xrc_qp_obj, *tmp1; if (!context) return 0; @@ -249,6 +255,13 @@ static int ib_uverbs_cleanup_ucontext(st kfree(uobj); } + list_for_each_entry_safe(xrc_qp_obj, tmp1, &context->xrc_reg_qp_list, list) { + list_del(&xrc_qp_obj->list); + ib_uverbs_cleanup_xrc_rcv_qp(file, xrc_qp_obj->domain_handle, + xrc_qp_obj->qp_num); + kfree(xrc_qp_obj); + } + mutex_lock(&file->device->ib_dev->xrcd_table_mutex); list_for_each_entry_safe(uobj, tmp, &context->xrc_domain_list, list) { struct ib_xrcd *xrcd = uobj->object; @@ -504,6 +517,12 @@ void ib_uverbs_event_handler(struct ib_e NULL, NULL); } +void ib_uverbs_xrc_rcv_qp_event_handler(struct ib_event *event, void *context_ptr) +{ + ib_uverbs_async_handler(context_ptr, event->element.xrc_qp_num, + event->event, NULL, NULL); +} + struct file *ib_uverbs_alloc_event_file(struct ib_uverbs_file *uverbs_file, int is_async, int *fd) { Index: infiniband/drivers/infiniband/core/uverbs_cmd.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_cmd.c 2008-06-23 14:08:11.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs_cmd.c 2008-06-23 14:51:55.000000000 +0300 @@ -313,6 +313,7 @@ ssize_t ib_uverbs_get_context(struct ib_ INIT_LIST_HEAD(&ucontext->srq_list); INIT_LIST_HEAD(&ucontext->ah_list); INIT_LIST_HEAD(&ucontext->xrc_domain_list); + INIT_LIST_HEAD(&ucontext->xrc_reg_qp_list); ucontext->closing = 0; resp.num_comp_vectors = file->device->num_comp_vectors; @@ -1078,6 +1079,7 @@ ssize_t ib_uverbs_create_qp(struct ib_uv goto err_put; } + attr.create_flags = 0; attr.event_handler = ib_uverbs_qp_event_handler; attr.qp_context = file; attr.send_cq = scq; @@ -2568,6 +2570,7 @@ ssize_t ib_uverbs_close_xrc_domain(struc int out_len) { struct ib_uverbs_close_xrc_domain cmd; + struct ib_uxrc_rcv_object *tmp; struct ib_uobject *uobj; struct ib_xrcd *xrcd = NULL; struct inode *inode = NULL; @@ -2583,6 +2586,18 @@ ssize_t ib_uverbs_close_xrc_domain(struc goto err_unlock_mutex; } + mutex_lock(&file->mutex); + list_for_each_entry(tmp, &file->ucontext->xrc_reg_qp_list, list) + if (cmd.xrcd_handle == tmp->domain_handle) { + ret = -EBUSY; + break; + } + mutex_unlock(&file->mutex); + if (ret) { + put_uobj_write(uobj); + goto err_unlock_mutex; + } + xrcd = (struct ib_xrcd *) (uobj->object); inode = xrcd->inode; @@ -2618,7 +2633,7 @@ err_unlock_mutex: } void ib_uverbs_dealloc_xrcd(struct ib_device *ib_dev, - struct ib_xrcd *xrcd) + struct ib_xrcd *xrcd) { struct inode *inode = NULL; int ret = 0; @@ -2632,4 +2647,352 @@ void ib_uverbs_dealloc_xrcd(struct ib_de xrcd_table_delete(ib_dev, inode); } +ssize_t ib_uverbs_create_xrc_rcv_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_create_xrc_rcv_qp cmd; + struct ib_uverbs_create_xrc_rcv_qp_resp resp; + struct ib_uxrc_rcv_object *obj; + struct ib_qp_init_attr init_attr; + struct ib_xrcd *xrcd; + struct ib_uobject *xrcd_uobj; + u32 qp_num; + int err; + + if (out_len < sizeof resp) + return -ENOSPC; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + obj = kmalloc(sizeof *obj, GFP_KERNEL); + if (!obj) + return -ENOMEM; + + xrcd = idr_read_xrcd(cmd.xrc_domain_handle, file->ucontext, &xrcd_uobj); + if (!xrcd) { + err = -EINVAL; + goto err_out; + } + + memset(&init_attr, 0, sizeof init_attr); + init_attr.event_handler = ib_uverbs_xrc_rcv_qp_event_handler; + init_attr.qp_context = file; + init_attr.srq = NULL; + init_attr.sq_sig_type = cmd.sq_sig_all ? IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR; + init_attr.qp_type = IB_QPT_XRC; + init_attr.xrc_domain = xrcd; + init_attr.create_flags = QP_CREATE_XRC_RCV; + + init_attr.cap.max_send_wr = 1; + init_attr.cap.max_recv_wr = 0; + init_attr.cap.max_send_sge = 1; + init_attr.cap.max_recv_sge = 0; + init_attr.cap.max_inline_data = 0; + + err = xrcd->device->create_xrc_rcv_qp(&init_attr, &qp_num); + if (err) + goto err_put; + + memset(&resp, 0, sizeof resp); + resp.qpn = qp_num; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) { + err = -EFAULT; + goto err_destroy; + } + + atomic_inc(&xrcd->usecnt); + put_xrcd_read(xrcd_uobj); + obj->qp_num = qp_num; + obj->domain_handle = cmd.xrc_domain_handle; + mutex_lock(&file->mutex); + list_add_tail(&obj->list, &file->ucontext->xrc_reg_qp_list); + mutex_unlock(&file->mutex); + + return in_len; + +err_destroy: + xrcd->device->unreg_xrc_rcv_qp(xrcd, file, qp_num); +err_put: + put_xrcd_read(xrcd_uobj); +err_out: + kfree(obj); + return err; +} + +ssize_t ib_uverbs_modify_xrc_rcv_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_modify_xrc_rcv_qp cmd; + struct ib_qp_attr *attr; + struct ib_xrcd *xrcd; + struct ib_uobject *xrcd_uobj; + int err; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + attr = kmalloc(sizeof *attr, GFP_KERNEL); + if (!attr) + return -ENOMEM; + + xrcd = idr_read_xrcd(cmd.xrc_domain_handle, file->ucontext, &xrcd_uobj); + if (!xrcd) { + kfree(attr); + return -EINVAL; + } + + memset(attr, 0, sizeof *attr); + attr->qp_state = cmd.qp_state; + attr->cur_qp_state = cmd.cur_qp_state; + attr->qp_access_flags = cmd.qp_access_flags; + attr->pkey_index = cmd.pkey_index; + attr->port_num = cmd.port_num; + attr->path_mtu = cmd.path_mtu; + attr->path_mig_state = cmd.path_mig_state; + attr->qkey = cmd.qkey; + attr->rq_psn = cmd.rq_psn; + attr->sq_psn = cmd.sq_psn; + attr->dest_qp_num = cmd.dest_qp_num; + attr->alt_pkey_index = cmd.alt_pkey_index; + attr->en_sqd_async_notify = cmd.en_sqd_async_notify; + attr->max_rd_atomic = cmd.max_rd_atomic; + attr->max_dest_rd_atomic = cmd.max_dest_rd_atomic; + attr->min_rnr_timer = cmd.min_rnr_timer; + attr->port_num = cmd.port_num; + attr->timeout = cmd.timeout; + attr->retry_cnt = cmd.retry_cnt; + attr->rnr_retry = cmd.rnr_retry; + attr->alt_port_num = cmd.alt_port_num; + attr->alt_timeout = cmd.alt_timeout; + + memcpy(attr->ah_attr.grh.dgid.raw, cmd.dest.dgid, 16); + attr->ah_attr.grh.flow_label = cmd.dest.flow_label; + attr->ah_attr.grh.sgid_index = cmd.dest.sgid_index; + attr->ah_attr.grh.hop_limit = cmd.dest.hop_limit; + attr->ah_attr.grh.traffic_class = cmd.dest.traffic_class; + attr->ah_attr.dlid = cmd.dest.dlid; + attr->ah_attr.sl = cmd.dest.sl; + attr->ah_attr.src_path_bits = cmd.dest.src_path_bits; + attr->ah_attr.static_rate = cmd.dest.static_rate; + attr->ah_attr.ah_flags = cmd.dest.is_global ? IB_AH_GRH : 0; + attr->ah_attr.port_num = cmd.dest.port_num; + + memcpy(attr->alt_ah_attr.grh.dgid.raw, cmd.alt_dest.dgid, 16); + attr->alt_ah_attr.grh.flow_label = cmd.alt_dest.flow_label; + attr->alt_ah_attr.grh.sgid_index = cmd.alt_dest.sgid_index; + attr->alt_ah_attr.grh.hop_limit = cmd.alt_dest.hop_limit; + attr->alt_ah_attr.grh.traffic_class = cmd.alt_dest.traffic_class; + attr->alt_ah_attr.dlid = cmd.alt_dest.dlid; + attr->alt_ah_attr.sl = cmd.alt_dest.sl; + attr->alt_ah_attr.src_path_bits = cmd.alt_dest.src_path_bits; + attr->alt_ah_attr.static_rate = cmd.alt_dest.static_rate; + attr->alt_ah_attr.ah_flags = cmd.alt_dest.is_global ? IB_AH_GRH : 0; + attr->alt_ah_attr.port_num = cmd.alt_dest.port_num; + + err = xrcd->device->modify_xrc_rcv_qp(xrcd, cmd.qp_num, attr, cmd.attr_mask); + put_xrcd_read(xrcd_uobj); + kfree(attr); + return err ? err : in_len; +} + +ssize_t ib_uverbs_query_xrc_rcv_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_query_xrc_rcv_qp cmd; + struct ib_uverbs_query_qp_resp resp; + struct ib_qp_attr *attr; + struct ib_qp_init_attr *init_attr; + struct ib_xrcd *xrcd; + struct ib_uobject *xrcd_uobj; + int ret; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + attr = kmalloc(sizeof *attr, GFP_KERNEL); + init_attr = kmalloc(sizeof *init_attr, GFP_KERNEL); + if (!attr || !init_attr) { + ret = -ENOMEM; + goto out; + } + xrcd = idr_read_xrcd(cmd.xrc_domain_handle, file->ucontext, &xrcd_uobj); + if (!xrcd) { + ret = -EINVAL; + goto out; + } + + ret = xrcd->device->query_xrc_rcv_qp(xrcd, cmd.qp_num, attr, + cmd.attr_mask, init_attr); + + put_xrcd_read(xrcd_uobj); + + if (ret) + goto out; + + memset(&resp, 0, sizeof resp); + resp.qp_state = attr->qp_state; + resp.cur_qp_state = attr->cur_qp_state; + resp.path_mtu = attr->path_mtu; + resp.path_mig_state = attr->path_mig_state; + resp.qkey = attr->qkey; + resp.rq_psn = attr->rq_psn; + resp.sq_psn = attr->sq_psn; + resp.dest_qp_num = attr->dest_qp_num; + resp.qp_access_flags = attr->qp_access_flags; + resp.pkey_index = attr->pkey_index; + resp.alt_pkey_index = attr->alt_pkey_index; + resp.sq_draining = attr->sq_draining; + resp.max_rd_atomic = attr->max_rd_atomic; + resp.max_dest_rd_atomic = attr->max_dest_rd_atomic; + resp.min_rnr_timer = attr->min_rnr_timer; + resp.port_num = attr->port_num; + resp.timeout = attr->timeout; + resp.retry_cnt = attr->retry_cnt; + resp.rnr_retry = attr->rnr_retry; + resp.alt_port_num = attr->alt_port_num; + resp.alt_timeout = attr->alt_timeout; + + memcpy(resp.dest.dgid, attr->ah_attr.grh.dgid.raw, 16); + resp.dest.flow_label = attr->ah_attr.grh.flow_label; + resp.dest.sgid_index = attr->ah_attr.grh.sgid_index; + resp.dest.hop_limit = attr->ah_attr.grh.hop_limit; + resp.dest.traffic_class = attr->ah_attr.grh.traffic_class; + resp.dest.dlid = attr->ah_attr.dlid; + resp.dest.sl = attr->ah_attr.sl; + resp.dest.src_path_bits = attr->ah_attr.src_path_bits; + resp.dest.static_rate = attr->ah_attr.static_rate; + resp.dest.is_global = !!(attr->ah_attr.ah_flags & IB_AH_GRH); + resp.dest.port_num = attr->ah_attr.port_num; + + memcpy(resp.alt_dest.dgid, attr->alt_ah_attr.grh.dgid.raw, 16); + resp.alt_dest.flow_label = attr->alt_ah_attr.grh.flow_label; + resp.alt_dest.sgid_index = attr->alt_ah_attr.grh.sgid_index; + resp.alt_dest.hop_limit = attr->alt_ah_attr.grh.hop_limit; + resp.alt_dest.traffic_class = attr->alt_ah_attr.grh.traffic_class; + resp.alt_dest.dlid = attr->alt_ah_attr.dlid; + resp.alt_dest.sl = attr->alt_ah_attr.sl; + resp.alt_dest.src_path_bits = attr->alt_ah_attr.src_path_bits; + resp.alt_dest.static_rate = attr->alt_ah_attr.static_rate; + resp.alt_dest.is_global = !!(attr->alt_ah_attr.ah_flags & IB_AH_GRH); + resp.alt_dest.port_num = attr->alt_ah_attr.port_num; + + resp.max_send_wr = init_attr->cap.max_send_wr; + resp.max_recv_wr = init_attr->cap.max_recv_wr; + resp.max_send_sge = init_attr->cap.max_send_sge; + resp.max_recv_sge = init_attr->cap.max_recv_sge; + resp.max_inline_data = init_attr->cap.max_inline_data; + resp.sq_sig_all = init_attr->sq_sig_type == IB_SIGNAL_ALL_WR; + + if (copy_to_user((void __user *) (unsigned long) cmd.response, + &resp, sizeof resp)) + ret = -EFAULT; + +out: + kfree(attr); + kfree(init_attr); + + return ret ? ret : in_len; +} + +ssize_t ib_uverbs_reg_xrc_rcv_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_reg_xrc_rcv_qp cmd; + struct ib_uxrc_rcv_object *qp_obj, *tmp; + struct ib_xrcd *xrcd; + struct ib_uobject *xrcd_uobj; + int ret; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + qp_obj = kmalloc(sizeof *qp_obj, GFP_KERNEL); + if (!qp_obj) + return -ENOMEM; + + xrcd = idr_read_xrcd(cmd.xrc_domain_handle, file->ucontext, &xrcd_uobj); + if (!xrcd) { + ret = -EINVAL; + goto err_out; + } + + ret = xrcd->device->reg_xrc_rcv_qp(xrcd, file, cmd.qp_num); + if (ret) + goto err_put; + + mutex_lock(&file->mutex); + list_for_each_entry(tmp, &file->ucontext->xrc_reg_qp_list, list) + if (cmd.qp_num == tmp->qp_num) { + kfree(qp_obj); + mutex_unlock(&file->mutex); + put_xrcd_read(xrcd_uobj); + return in_len; + } + qp_obj->qp_num = cmd.qp_num; + qp_obj->domain_handle = cmd.xrc_domain_handle; + list_add_tail(&qp_obj->list, &file->ucontext->xrc_reg_qp_list); + mutex_unlock(&file->mutex); + atomic_inc(&xrcd->usecnt); + put_xrcd_read(xrcd_uobj); + return in_len; + +err_put: + put_xrcd_read(xrcd_uobj); +err_out: + + kfree(qp_obj); + return ret; +} + +int ib_uverbs_cleanup_xrc_rcv_qp(struct ib_uverbs_file *file, + u32 domain_handle, u32 qp_num) +{ + struct ib_xrcd *xrcd; + struct ib_uobject *xrcd_uobj; + int err; + + xrcd = idr_read_xrcd(domain_handle, file->ucontext, &xrcd_uobj); + if (!xrcd) + return -EINVAL; + + err = xrcd->device->unreg_xrc_rcv_qp(xrcd, file, qp_num); + + if (!err) + atomic_dec(&xrcd->usecnt); + put_xrcd_read(xrcd_uobj); + return err; +} + +ssize_t ib_uverbs_unreg_xrc_rcv_qp(struct ib_uverbs_file *file, + const char __user *buf, int in_len, + int out_len) +{ + struct ib_uverbs_unreg_xrc_rcv_qp cmd; + struct ib_uxrc_rcv_object *qp_obj, *tmp; + int ret; + + if (copy_from_user(&cmd, buf, sizeof cmd)) + return -EFAULT; + + ret = ib_uverbs_cleanup_xrc_rcv_qp(file, cmd.xrc_domain_handle, cmd.qp_num); + if (ret) + return ret; + + mutex_lock(&file->mutex); + list_for_each_entry_safe(qp_obj, tmp, &file->ucontext->xrc_reg_qp_list, list) + if (cmd.qp_num == qp_obj->qp_num) { + list_del(&qp_obj->list); + kfree(qp_obj); + break; + } + mutex_unlock(&file->mutex); + return in_len; +} Index: infiniband/include/rdma/ib_user_verbs.h =================================================================== --- infiniband.orig/include/rdma/ib_user_verbs.h 2008-06-23 14:08:09.000000000 +0300 +++ infiniband/include/rdma/ib_user_verbs.h 2008-06-23 14:08:14.000000000 +0300 @@ -84,7 +84,12 @@ enum { IB_USER_VERBS_CMD_POST_SRQ_RECV, IB_USER_VERBS_CMD_CREATE_XRC_SRQ, IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN, - IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN + IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN, + IB_USER_VERBS_CMD_CREATE_XRC_RCV_QP, + IB_USER_VERBS_CMD_MODIFY_XRC_RCV_QP, + IB_USER_VERBS_CMD_QUERY_XRC_RCV_QP, + IB_USER_VERBS_CMD_REG_XRC_RCV_QP, + IB_USER_VERBS_CMD_UNREG_XRC_RCV_QP, }; /* @@ -718,6 +723,76 @@ struct ib_uverbs_close_xrc_domain { __u64 driver_data[0]; }; +struct ib_uverbs_create_xrc_rcv_qp { + __u64 response; + __u64 user_handle; + __u32 xrc_domain_handle; + __u32 max_send_wr; + __u32 max_recv_wr; + __u32 max_send_sge; + __u32 max_recv_sge; + __u32 max_inline_data; + __u8 sq_sig_all; + __u8 qp_type; + __u8 reserved[2]; + __u64 driver_data[0]; +}; + +struct ib_uverbs_create_xrc_rcv_qp_resp { + __u32 qpn; + __u32 reserved; +}; + +struct ib_uverbs_modify_xrc_rcv_qp { + __u32 xrc_domain_handle; + __u32 qp_num; + struct ib_uverbs_qp_dest dest; + struct ib_uverbs_qp_dest alt_dest; + __u32 attr_mask; + __u32 qkey; + __u32 rq_psn; + __u32 sq_psn; + __u32 dest_qp_num; + __u32 qp_access_flags; + __u16 pkey_index; + __u16 alt_pkey_index; + __u8 qp_state; + __u8 cur_qp_state; + __u8 path_mtu; + __u8 path_mig_state; + __u8 en_sqd_async_notify; + __u8 max_rd_atomic; + __u8 max_dest_rd_atomic; + __u8 min_rnr_timer; + __u8 port_num; + __u8 timeout; + __u8 retry_cnt; + __u8 rnr_retry; + __u8 alt_port_num; + __u8 alt_timeout; + __u8 reserved[2]; + __u64 driver_data[0]; +}; + +struct ib_uverbs_query_xrc_rcv_qp { + __u64 response; + __u32 xrc_domain_handle; + __u32 qp_num; + __u32 attr_mask; + __u64 driver_data[0]; +}; + +struct ib_uverbs_reg_xrc_rcv_qp { + __u32 xrc_domain_handle; + __u32 qp_num; + __u64 driver_data[0]; +}; + +struct ib_uverbs_unreg_xrc_rcv_qp { + __u32 xrc_domain_handle; + __u32 qp_num; + __u64 driver_data[0]; +}; #endif /* IB_USER_VERBS_H */ Index: infiniband/drivers/infiniband/core/uverbs.h =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs.h 2008-06-23 14:08:11.000000000 +0300 +++ infiniband/drivers/infiniband/core/uverbs.h 2008-06-23 14:08:14.000000000 +0300 @@ -161,8 +161,12 @@ void ib_uverbs_qp_event_handler(struct i void ib_uverbs_srq_event_handler(struct ib_event *event, void *context_ptr); void ib_uverbs_event_handler(struct ib_event_handler *handler, struct ib_event *event); +void ib_uverbs_xrc_rcv_qp_event_handler(struct ib_event *event, + void *context_ptr); void ib_uverbs_dealloc_xrcd(struct ib_device *ib_dev, struct ib_xrcd *xrcd); +int ib_uverbs_cleanup_xrc_rcv_qp(struct ib_uverbs_file *file, + u32 domain_handle, u32 qp_num); #define IB_UVERBS_DECLARE_CMD(name) \ ssize_t ib_uverbs_##name(struct ib_uverbs_file *file, \ @@ -200,6 +204,11 @@ IB_UVERBS_DECLARE_CMD(destroy_srq); IB_UVERBS_DECLARE_CMD(create_xrc_srq); IB_UVERBS_DECLARE_CMD(open_xrc_domain); IB_UVERBS_DECLARE_CMD(close_xrc_domain); +IB_UVERBS_DECLARE_CMD(create_xrc_rcv_qp); +IB_UVERBS_DECLARE_CMD(modify_xrc_rcv_qp); +IB_UVERBS_DECLARE_CMD(query_xrc_rcv_qp); +IB_UVERBS_DECLARE_CMD(reg_xrc_rcv_qp); +IB_UVERBS_DECLARE_CMD(unreg_xrc_rcv_qp); #endif /* UVERBS_H */ From jackm at dev.mellanox.co.il Mon Jun 23 06:01:54 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:54 +0300 Subject: [ofa-general] [PATCH 9/11] mlx4: XRC base implementation Message-ID: <200806231601.55148.jackm@dev.mellanox.co.il> mlx4: Implements XRC support. Rev 2: Do not allocate an RQ for an XRC QP. Signed-off-by: Jack Morgenstein Index: infiniband/include/linux/mlx4/device.h =================================================================== --- infiniband.orig/include/linux/mlx4/device.h 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/include/linux/mlx4/device.h 2008-06-23 14:07:54.000000000 +0300 @@ -56,6 +56,7 @@ enum { MLX4_DEV_CAP_FLAG_RC = 1 << 0, MLX4_DEV_CAP_FLAG_UC = 1 << 1, MLX4_DEV_CAP_FLAG_UD = 1 << 2, + MLX4_DEV_CAP_FLAG_XRC = 1 << 3, MLX4_DEV_CAP_FLAG_SRQ = 1 << 6, MLX4_DEV_CAP_FLAG_IPOIB_CSUM = 1 << 7, MLX4_DEV_CAP_FLAG_BAD_PKEY_CNTR = 1 << 8, @@ -181,6 +182,8 @@ struct mlx4_caps { int num_pds; int reserved_pds; int mtt_entry_sz; + int reserved_xrcds; + int max_xrcds; u32 max_msg_sz; u32 page_size_cap; u32 flags; @@ -356,6 +359,9 @@ static inline void *mlx4_buf_offset(stru int mlx4_pd_alloc(struct mlx4_dev *dev, u32 *pdn); void mlx4_pd_free(struct mlx4_dev *dev, u32 pdn); +int mlx4_xrcd_alloc(struct mlx4_dev *dev, u32 *xrcdn); +void mlx4_xrcd_free(struct mlx4_dev *dev, u32 xrcdn); + int mlx4_uar_alloc(struct mlx4_dev *dev, struct mlx4_uar *uar); void mlx4_uar_free(struct mlx4_dev *dev, struct mlx4_uar *uar); @@ -389,8 +395,8 @@ void mlx4_cq_free(struct mlx4_dev *dev, int mlx4_qp_alloc(struct mlx4_dev *dev, int sqpn, struct mlx4_qp *qp); void mlx4_qp_free(struct mlx4_dev *dev, struct mlx4_qp *qp); -int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, struct mlx4_mtt *mtt, - u64 db_rec, struct mlx4_srq *srq); +int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, u32 cqn, u16 xrcd, + struct mlx4_mtt *mtt, u64 db_rec, struct mlx4_srq *srq); void mlx4_srq_free(struct mlx4_dev *dev, struct mlx4_srq *srq); int mlx4_srq_arm(struct mlx4_dev *dev, struct mlx4_srq *srq, int limit_watermark); int mlx4_srq_query(struct mlx4_dev *dev, struct mlx4_srq *srq, int *limit_watermark); Index: infiniband/drivers/infiniband/hw/mlx4/main.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/main.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/main.c 2008-06-23 14:07:54.000000000 +0300 @@ -103,6 +103,8 @@ static int mlx4_ib_query_device(struct i props->device_cap_flags |= IB_DEVICE_UD_IP_CSUM; if (dev->dev->caps.max_gso_sz) props->device_cap_flags |= IB_DEVICE_UD_TSO; + if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC) + props->device_cap_flags |= IB_DEVICE_XRC; props->vendor_id = be32_to_cpup((__be32 *) (out_mad->data + 36)) & 0xffffff; @@ -446,6 +448,46 @@ static int mlx4_ib_mcg_detach(struct ib_ &to_mqp(ibqp)->mqp, gid->raw); } +static struct ib_xrcd *mlx4_ib_alloc_xrcd(struct ib_device *ibdev, + struct ib_ucontext *context, + struct ib_udata *udata) +{ + struct mlx4_ib_xrcd *xrcd; + struct mlx4_ib_dev *mdev = to_mdev(ibdev); + int err; + + if (!(mdev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) + return ERR_PTR(-ENOSYS); + + xrcd = kmalloc(sizeof *xrcd, GFP_KERNEL); + if (!xrcd) + return ERR_PTR(-ENOMEM); + + err = mlx4_xrcd_alloc(mdev->dev, &xrcd->xrcdn); + if (err) { + kfree(xrcd); + return ERR_PTR(err); + } + + if (context) + if (ib_copy_to_udata(udata, &xrcd->xrcdn, sizeof (__u32))) { + mlx4_xrcd_free(mdev->dev, xrcd->xrcdn); + kfree(xrcd); + return ERR_PTR(-EFAULT); + } + + return &xrcd->ibxrcd; +} + +static int mlx4_ib_dealloc_xrcd(struct ib_xrcd *xrcd) +{ + mlx4_xrcd_free(to_mdev(xrcd->device)->dev, to_mxrcd(xrcd)->xrcdn); + kfree(xrcd); + + return 0; +} + + static int init_node_data(struct mlx4_ib_dev *dev) { struct ib_smp *in_mad = NULL; @@ -632,6 +674,16 @@ static void *mlx4_ib_add(struct mlx4_dev ibdev->ib_dev.map_phys_fmr = mlx4_ib_map_phys_fmr; ibdev->ib_dev.unmap_fmr = mlx4_ib_unmap_fmr; ibdev->ib_dev.dealloc_fmr = mlx4_ib_fmr_dealloc; + if (dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC) { + ibdev->ib_dev.create_xrc_srq = mlx4_ib_create_xrc_srq; + ibdev->ib_dev.alloc_xrcd = mlx4_ib_alloc_xrcd; + ibdev->ib_dev.dealloc_xrcd = mlx4_ib_dealloc_xrcd; + ibdev->ib_dev.uverbs_cmd_mask |= + (1ull << IB_USER_VERBS_CMD_CREATE_XRC_SRQ) | + (1ull << IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN) | + (1ull << IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN); + } + if (init_node_data(ibdev)) goto err_map; Index: infiniband/drivers/infiniband/hw/mlx4/mlx4_ib.h =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-23 14:07:54.000000000 +0300 @@ -55,6 +55,11 @@ struct mlx4_ib_pd { u32 pdn; }; +struct mlx4_ib_xrcd { + struct ib_xrcd ibxrcd; + u32 xrcdn; +}; + struct mlx4_ib_cq_buf { struct mlx4_buf buf; struct mlx4_mtt mtt; @@ -124,6 +129,7 @@ struct mlx4_ib_qp { int buf_size; struct mutex mutex; u32 flags; + u16 xrcdn; u8 port; u8 alt_port; u8 atomic_rd_en; @@ -183,6 +189,11 @@ static inline struct mlx4_ib_pd *to_mpd( return container_of(ibpd, struct mlx4_ib_pd, ibpd); } +static inline struct mlx4_ib_xrcd *to_mxrcd(struct ib_xrcd *ibxrcd) +{ + return container_of(ibxrcd, struct mlx4_ib_xrcd, ibxrcd); +} + static inline struct mlx4_ib_cq *to_mcq(struct ib_cq *ibcq) { return container_of(ibcq, struct mlx4_ib_cq, ibcq); @@ -257,6 +268,11 @@ int mlx4_ib_destroy_ah(struct ib_ah *ah) struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd, struct ib_srq_init_attr *init_attr, struct ib_udata *udata); +struct ib_srq *mlx4_ib_create_xrc_srq(struct ib_pd *pd, + struct ib_cq *xrc_cq, + struct ib_xrcd *xrcd, + struct ib_srq_init_attr *init_attr, + struct ib_udata *udata); int mlx4_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr, enum ib_srq_attr_mask attr_mask, struct ib_udata *udata); int mlx4_ib_query_srq(struct ib_srq *srq, struct ib_srq_attr *srq_attr); Index: infiniband/drivers/net/mlx4/xrcd.c =================================================================== --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ infiniband/drivers/net/mlx4/xrcd.c 2008-06-23 14:07:54.000000000 +0300 @@ -0,0 +1,70 @@ +/* + * Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved. + * Copyright (c) 2007 Mellanox Technologies. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include +#include + +#include "mlx4.h" + +int mlx4_xrcd_alloc(struct mlx4_dev *dev, u32 *xrcdn) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + *xrcdn = mlx4_bitmap_alloc(&priv->xrcd_bitmap); + if (*xrcdn == -1) + return -ENOMEM; + + return 0; +} +EXPORT_SYMBOL_GPL(mlx4_xrcd_alloc); + +void mlx4_xrcd_free(struct mlx4_dev *dev, u32 xrcdn) +{ + mlx4_bitmap_free(&mlx4_priv(dev)->xrcd_bitmap, xrcdn); +} +EXPORT_SYMBOL_GPL(mlx4_xrcd_free); + +int __devinit mlx4_init_xrcd_table(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + return mlx4_bitmap_init(&priv->xrcd_bitmap, (1 << 16), + (1 << 16) - 1, dev->caps.reserved_xrcds + 1); +} + +void mlx4_cleanup_xrcd_table(struct mlx4_dev *dev) +{ + mlx4_bitmap_cleanup(&mlx4_priv(dev)->xrcd_bitmap); +} + + Index: infiniband/drivers/net/mlx4/mlx4.h =================================================================== --- infiniband.orig/drivers/net/mlx4/mlx4.h 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/mlx4.h 2008-06-23 14:07:54.000000000 +0300 @@ -264,6 +264,7 @@ struct mlx4_priv { struct mlx4_cmd cmd; struct mlx4_bitmap pd_bitmap; + struct mlx4_bitmap xrcd_bitmap; struct mlx4_uar_table uar_table; struct mlx4_mr_table mr_table; struct mlx4_cq_table cq_table; @@ -293,6 +294,7 @@ void mlx4_bitmap_cleanup(struct mlx4_bit int mlx4_reset(struct mlx4_dev *dev); int mlx4_init_pd_table(struct mlx4_dev *dev); +int mlx4_init_xrcd_table(struct mlx4_dev *dev); int mlx4_init_uar_table(struct mlx4_dev *dev); int mlx4_init_mr_table(struct mlx4_dev *dev); int mlx4_init_eq_table(struct mlx4_dev *dev); @@ -309,6 +311,7 @@ void mlx4_cleanup_cq_table(struct mlx4_d void mlx4_cleanup_qp_table(struct mlx4_dev *dev); void mlx4_cleanup_srq_table(struct mlx4_dev *dev); void mlx4_cleanup_mcg_table(struct mlx4_dev *dev); +void mlx4_cleanup_xrcd_table(struct mlx4_dev *dev); void mlx4_start_catas_poll(struct mlx4_dev *dev); void mlx4_stop_catas_poll(struct mlx4_dev *dev); Index: infiniband/drivers/net/mlx4/main.c =================================================================== --- infiniband.orig/drivers/net/mlx4/main.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/main.c 2008-06-23 14:07:54.000000000 +0300 @@ -160,6 +160,10 @@ static int mlx4_dev_cap(struct mlx4_dev dev->caps.flags = dev_cap->flags; dev->caps.stat_rate_support = dev_cap->stat_rate_support; dev->caps.max_gso_sz = dev_cap->max_gso_sz; + dev->caps.reserved_xrcds = (dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC) ? + dev_cap->reserved_xrcds : 0; + dev->caps.max_xrcds = (dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC) ? + dev_cap->max_xrcds : 0; return 0; } @@ -586,11 +590,18 @@ static int mlx4_setup_hca(struct mlx4_de goto err_kar_unmap; } + err = mlx4_init_xrcd_table(dev); + if (err) { + mlx4_err(dev, "Failed to initialize " + "extended reliably connected domain table, aborting.\n"); + goto err_pd_table_free; + } + err = mlx4_init_mr_table(dev); if (err) { mlx4_err(dev, "Failed to initialize " "memory region table, aborting.\n"); - goto err_pd_table_free; + goto err_xrcd_table_free; } err = mlx4_init_eq_table(dev); @@ -674,6 +685,9 @@ err_eq_table_free: err_mr_table_free: mlx4_cleanup_mr_table(dev); +err_xrcd_table_free: + mlx4_cleanup_xrcd_table(dev); + err_pd_table_free: mlx4_cleanup_pd_table(dev); @@ -849,6 +863,7 @@ err_cleanup: mlx4_cmd_use_polling(dev); mlx4_cleanup_eq_table(dev); mlx4_cleanup_mr_table(dev); + mlx4_cleanup_xrcd_table(dev); mlx4_cleanup_pd_table(dev); mlx4_cleanup_uar_table(dev); @@ -908,6 +923,7 @@ static void mlx4_remove_one(struct pci_d mlx4_cmd_use_polling(dev); mlx4_cleanup_eq_table(dev); mlx4_cleanup_mr_table(dev); + mlx4_cleanup_xrcd_table(dev); mlx4_cleanup_pd_table(dev); iounmap(priv->kar); Index: infiniband/drivers/net/mlx4/srq.c =================================================================== --- infiniband.orig/drivers/net/mlx4/srq.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/srq.c 2008-06-23 14:07:54.000000000 +0300 @@ -40,20 +40,20 @@ struct mlx4_srq_context { __be32 state_logsize_srqn; u8 logstride; - u8 reserved1[3]; - u8 pg_offset; - u8 reserved2[3]; - u32 reserved3; + u8 reserved1; + __be16 xrc_domain; + __be32 pg_offset_cqn; + u32 reserved2; u8 log_page_size; - u8 reserved4[2]; + u8 reserved3[2]; u8 mtt_base_addr_h; __be32 mtt_base_addr_l; __be32 pd; __be16 limit_watermark; __be16 wqe_cnt; - u16 reserved5; + u16 reserved4; __be16 wqe_counter; - u32 reserved6; + u32 reserved5; __be64 db_rec_addr; }; @@ -109,8 +109,8 @@ static int mlx4_QUERY_SRQ(struct mlx4_de MLX4_CMD_TIME_CLASS_A); } -int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, struct mlx4_mtt *mtt, - u64 db_rec, struct mlx4_srq *srq) +int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, u32 cqn, u16 xrcd, + struct mlx4_mtt *mtt, u64 db_rec, struct mlx4_srq *srq) { struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table; struct mlx4_cmd_mailbox *mailbox; @@ -148,6 +148,8 @@ int mlx4_srq_alloc(struct mlx4_dev *dev, srq_context->state_logsize_srqn = cpu_to_be32((ilog2(srq->max) << 24) | srq->srqn); srq_context->logstride = srq->wqe_shift - 4; + srq_context->xrc_domain = cpu_to_be16(xrcd); + srq_context->pg_offset_cqn = cpu_to_be32(cqn & 0xffffff); srq_context->log_page_size = mtt->page_shift - MLX4_ICM_PAGE_SHIFT; mtt_addr = mlx4_mtt_addr(dev, mtt); Index: infiniband/drivers/net/mlx4/fw.c =================================================================== --- infiniband.orig/drivers/net/mlx4/fw.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/fw.c 2008-06-23 14:07:54.000000000 +0300 @@ -160,6 +160,8 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev * #define QUERY_DEV_CAP_MAX_MCG_OFFSET 0x63 #define QUERY_DEV_CAP_RSVD_PD_OFFSET 0x64 #define QUERY_DEV_CAP_MAX_PD_OFFSET 0x65 +#define QUERY_DEV_CAP_RSVD_XRC_OFFSET 0x66 +#define QUERY_DEV_CAP_MAX_XRC_OFFSET 0x67 #define QUERY_DEV_CAP_RDMARC_ENTRY_SZ_OFFSET 0x80 #define QUERY_DEV_CAP_QPC_ENTRY_SZ_OFFSET 0x82 #define QUERY_DEV_CAP_AUX_ENTRY_SZ_OFFSET 0x84 @@ -270,6 +272,11 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev * MLX4_GET(field, outbox, QUERY_DEV_CAP_MAX_PD_OFFSET); dev_cap->max_pds = 1 << (field & 0x3f); + MLX4_GET(field, outbox, QUERY_DEV_CAP_RSVD_XRC_OFFSET); + dev_cap->reserved_xrcds = field >> 4; + MLX4_GET(field, outbox, QUERY_DEV_CAP_MAX_XRC_OFFSET); + dev_cap->max_xrcds = 1 << (field & 0x1f); + MLX4_GET(size, outbox, QUERY_DEV_CAP_RDMARC_ENTRY_SZ_OFFSET); dev_cap->rdmarc_entry_sz = size; MLX4_GET(size, outbox, QUERY_DEV_CAP_QPC_ENTRY_SZ_OFFSET); Index: infiniband/drivers/net/mlx4/fw.h =================================================================== --- infiniband.orig/drivers/net/mlx4/fw.h 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/fw.h 2008-06-23 14:07:54.000000000 +0300 @@ -82,6 +82,8 @@ struct mlx4_dev_cap { int max_mcgs; int reserved_pds; int max_pds; + int reserved_xrcds; + int max_xrcds; int qpc_entry_sz; int rdmarc_entry_sz; int altc_entry_sz; Index: infiniband/drivers/infiniband/hw/mlx4/qp.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/qp.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/qp.c 2008-06-23 14:07:54.000000000 +0300 @@ -282,14 +282,14 @@ static int send_wqe_overhead(enum ib_qp_ } static int set_rq_size(struct mlx4_ib_dev *dev, struct ib_qp_cap *cap, - int is_user, int has_srq, struct mlx4_ib_qp *qp) + int is_user, int has_srq_or_is_xrc, struct mlx4_ib_qp *qp) { /* Sanity check RQ size before proceeding */ if (cap->max_recv_wr > dev->dev->caps.max_wqes || cap->max_recv_sge > dev->dev->caps.max_rq_sg) return -EINVAL; - if (has_srq) { + if (has_srq_or_is_xrc) { /* QPs attached to an SRQ should have no RQ */ if (cap->max_recv_wr) return -EINVAL; @@ -468,7 +468,8 @@ static int create_qp_common(struct mlx4_ else qp->sq_signal_bits = 0; - err = set_rq_size(dev, &init_attr->cap, !!pd->uobject, !!init_attr->srq, qp); + err = set_rq_size(dev, &init_attr->cap, !!pd->uobject, + !!init_attr->srq || !!init_attr->xrc_domain , qp); if (err) goto err; @@ -502,7 +503,7 @@ static int create_qp_common(struct mlx4_ if (err) goto err_mtt; - if (!init_attr->srq) { + if (!init_attr->srq && init_attr->qp_type != IB_QPT_XRC) { err = mlx4_ib_db_map_user(to_mucontext(pd->uobject->context), ucmd.db_addr, &qp->db); if (err) @@ -518,7 +519,7 @@ static int create_qp_common(struct mlx4_ if (err) goto err; - if (!init_attr->srq) { + if (!init_attr->srq && init_attr->qp_type != IB_QPT_XRC) { err = mlx4_db_alloc(dev->dev, &qp->db, 0); if (err) goto err; @@ -553,6 +554,9 @@ static int create_qp_common(struct mlx4_ if (err) goto err_wrid; + if (init_attr->qp_type == IB_QPT_XRC) + qp->mqp.qpn |= (1 << 23); + /* * Hardware wants QPN written in big-endian order (after * shifting) for send doorbell. Precompute this value to save @@ -566,7 +570,7 @@ static int create_qp_common(struct mlx4_ err_wrid: if (pd->uobject) { - if (!init_attr->srq) + if (!init_attr->srq && init_attr->qp_type != IB_QPT_XRC) mlx4_ib_db_unmap_user(to_mucontext(pd->uobject->context), &qp->db); } else { @@ -584,7 +588,7 @@ err_buf: mlx4_buf_free(dev->dev, qp->buf_size, &qp->buf); err_db: - if (!pd->uobject && !init_attr->srq) + if (!pd->uobject && !init_attr->srq && init_attr->qp_type != IB_QPT_XRC) mlx4_db_free(dev->dev, &qp->db); err: @@ -662,7 +666,7 @@ static void destroy_qp_common(struct mlx mlx4_mtt_cleanup(dev->dev, &qp->mtt); if (is_user) { - if (!qp->ibqp.srq) + if (!qp->ibqp.srq && qp->ibqp.qp_type != IB_QPT_XRC) mlx4_ib_db_unmap_user(to_mucontext(qp->ibqp.uobject->context), &qp->db); ib_umem_release(qp->umem); @@ -670,7 +674,7 @@ static void destroy_qp_common(struct mlx kfree(qp->sq.wrid); kfree(qp->rq.wrid); mlx4_buf_free(dev->dev, qp->buf_size, &qp->buf); - if (!qp->ibqp.srq) + if (!qp->ibqp.srq && qp->ibqp.qp_type != IB_QPT_XRC) mlx4_db_free(dev->dev, &qp->db); } } @@ -692,6 +696,9 @@ struct ib_qp *mlx4_ib_create_qp(struct i return ERR_PTR(-EINVAL); switch (init_attr->qp_type) { + case IB_QPT_XRC: + if (!(dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) + return ERR_PTR(-ENOSYS); case IB_QPT_RC: case IB_QPT_UC: case IB_QPT_UD: @@ -706,6 +713,11 @@ struct ib_qp *mlx4_ib_create_qp(struct i return ERR_PTR(err); } + if (init_attr->qp_type == IB_QPT_XRC) + qp->xrcdn = to_mxrcd(init_attr->xrc_domain)->xrcdn; + else + qp->xrcdn = 0; + qp->ibqp.qp_num = qp->mqp.qpn; break; @@ -770,6 +782,7 @@ static int to_mlx4_st(enum ib_qp_type ty case IB_QPT_RC: return MLX4_QP_ST_RC; case IB_QPT_UC: return MLX4_QP_ST_UC; case IB_QPT_UD: return MLX4_QP_ST_UD; + case IB_QPT_XRC: return MLX4_QP_ST_XRC; case IB_QPT_SMI: case IB_QPT_GSI: return MLX4_QP_ST_MLX; default: return -1; @@ -920,8 +933,11 @@ static int __mlx4_ib_modify_qp(struct ib context->sq_size_stride = ilog2(qp->sq.wqe_cnt) << 3; context->sq_size_stride |= qp->sq.wqe_shift - 4; - if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) + if (cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) { context->sq_size_stride |= !!qp->sq_no_prefetch << 7; + if (ibqp->qp_type == IB_QPT_XRC) + context->xrcd = cpu_to_be32((u32) qp->xrcdn); + } if (qp->ibqp.uobject) context->usr_page = cpu_to_be32(to_mucontext(ibqp->uobject->context)->uar.index); @@ -1033,7 +1049,8 @@ static int __mlx4_ib_modify_qp(struct ib if (ibqp->srq) context->srqn = cpu_to_be32(1 << 24 | to_msrq(ibqp->srq)->msrq.srqn); - if (!ibqp->srq && cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) + if (!ibqp->srq && ibqp->qp_type != IB_QPT_XRC && + cur_state == IB_QPS_RESET && new_state == IB_QPS_INIT) context->db_rec_addr = cpu_to_be64(qp->db.dma); if (cur_state == IB_QPS_INIT && @@ -1123,7 +1140,7 @@ static int __mlx4_ib_modify_qp(struct ib qp->sq.head = 0; qp->sq.tail = 0; qp->sq_next_wqe = 0; - if (!ibqp->srq) + if (!ibqp->srq && ibqp->qp_type != IB_QPT_XRC) *qp->db.db = 0; } Index: infiniband/drivers/infiniband/hw/mlx4/srq.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/srq.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/srq.c 2008-06-23 14:07:54.000000000 +0300 @@ -66,13 +66,17 @@ static void mlx4_ib_srq_event(struct mlx } } -struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd, - struct ib_srq_init_attr *init_attr, - struct ib_udata *udata) +struct ib_srq *mlx4_ib_create_xrc_srq(struct ib_pd *pd, + struct ib_cq *xrc_cq, + struct ib_xrcd *xrcd, + struct ib_srq_init_attr *init_attr, + struct ib_udata *udata) { struct mlx4_ib_dev *dev = to_mdev(pd->device); struct mlx4_ib_srq *srq; struct mlx4_wqe_srq_next_seg *next; + u32 cqn; + u16 xrcdn; int desc_size; int buf_size; int err; @@ -166,7 +170,11 @@ struct ib_srq *mlx4_ib_create_srq(struct } } - err = mlx4_srq_alloc(dev->dev, to_mpd(pd)->pdn, &srq->mtt, + cqn = xrc_cq ? (u32) (to_mcq(xrc_cq)->mcq.cqn) : 0; + xrcdn = xrcd ? (u16) (to_mxrcd(xrcd)->xrcdn) : + (u16) dev->dev->caps.reserved_xrcds; + + err = mlx4_srq_alloc(dev->dev, to_mpd(pd)->pdn, cqn, xrcdn, &srq->mtt, srq->db.dma, &srq->msrq); if (err) goto err_wrid; @@ -234,6 +242,13 @@ int mlx4_ib_modify_srq(struct ib_srq *ib return 0; } +struct ib_srq *mlx4_ib_create_srq(struct ib_pd *pd, + struct ib_srq_init_attr *init_attr, + struct ib_udata *udata) +{ + return mlx4_ib_create_xrc_srq(pd, NULL, NULL, init_attr, udata); +} + int mlx4_ib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr) { struct mlx4_ib_dev *dev = to_mdev(ibsrq->device); Index: infiniband/include/linux/mlx4/qp.h =================================================================== --- infiniband.orig/include/linux/mlx4/qp.h 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/include/linux/mlx4/qp.h 2008-06-23 14:07:54.000000000 +0300 @@ -74,6 +74,7 @@ enum { MLX4_QP_ST_UC = 0x1, MLX4_QP_ST_RD = 0x2, MLX4_QP_ST_UD = 0x3, + MLX4_QP_ST_XRC = 0x6, MLX4_QP_ST_MLX = 0x7 }; @@ -136,7 +137,7 @@ struct mlx4_qp_context { __be32 ssn; __be32 params2; __be32 rnr_nextrecvpsn; - __be32 srcd; + __be32 xrcd; __be32 cqn_recv; __be64 db_rec_addr; __be32 qkey; Index: infiniband/drivers/net/mlx4/Makefile =================================================================== --- infiniband.orig/drivers/net/mlx4/Makefile 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/Makefile 2008-06-23 14:07:54.000000000 +0300 @@ -1,4 +1,4 @@ obj-$(CONFIG_MLX4_CORE) += mlx4_core.o mlx4_core-y := alloc.o catas.o cmd.o cq.o eq.o fw.o icm.o intf.o main.o mcg.o \ - mr.o pd.o profile.o qp.o reset.o srq.o + mr.o pd.o profile.o qp.o reset.o srq.o xrcd.o Index: infiniband/drivers/net/mlx4/qp.c =================================================================== --- infiniband.orig/drivers/net/mlx4/qp.c 2008-06-23 14:07:52.000000000 +0300 +++ infiniband/drivers/net/mlx4/qp.c 2008-06-23 14:07:54.000000000 +0300 @@ -263,10 +263,12 @@ int mlx4_init_qp_table(struct mlx4_dev * * We reserve 2 extra QPs per port for the special QPs. The * block of special QPs must be aligned to a multiple of 8, so * round up. + * We also reserve the MSB of the 24-bit QP number to indicate + * an XRC qp. */ dev->caps.sqp_start = ALIGN(dev->caps.reserved_qps, 8); err = mlx4_bitmap_init(&qp_table->bitmap, dev->caps.num_qps, - (1 << 24) - 1, dev->caps.sqp_start + 8); + (1 << 23) - 1, dev->caps.sqp_start + 8); if (err) return err; From jackm at dev.mellanox.co.il Mon Jun 23 06:01:57 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:57 +0300 Subject: [ofa-general] [PATCH 10/11] mlx4: XRC kernel-space app QP support Message-ID: <200806231601.58028.jackm@dev.mellanox.co.il> mlx4: Implement kernel-space XRC. Signed-off-by: Jack Morgenstein Index: infiniband/drivers/infiniband/hw/mlx4/cq.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/cq.c 2008-06-23 11:43:29.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/cq.c 2008-06-23 13:56:33.000000000 +0300 @@ -32,6 +32,7 @@ #include #include +#include #include "mlx4_ib.h" #include "user.h" @@ -536,9 +537,11 @@ static int mlx4_ib_poll_one(struct mlx4_ struct mlx4_qp *mqp; struct mlx4_ib_wq *wq; struct mlx4_ib_srq *srq; + struct mlx4_srq *msrq; int is_send; int is_error; u32 g_mlpath_rqpn; + int is_xrc_recv = 0; u16 wqe_ctr; repoll: @@ -580,7 +583,23 @@ repoll: goto repoll; } - if (!*cur_qp || + if ((be32_to_cpu(cqe->my_qpn) & (1 << 23)) && !is_send) { + /* + * We do not have to take the XRC SRQ table lock here, + * because CQs will be locked while XRC SRQs are removed + * from the table. + */ + msrq = __mlx4_srq_lookup(to_mdev(cq->ibcq.device)->dev, + be32_to_cpu(cqe->g_mlpath_rqpn) & + 0xffffff); + if (unlikely(!msrq)) { + printk(KERN_WARNING "CQ %06x with entry for unknown XRC SRQ %06x\n", + cq->mcq.cqn, be32_to_cpu(cqe->g_mlpath_rqpn) & 0xffffff); + return -EINVAL; + } + is_xrc_recv = 1; + srq = to_mibsrq(msrq); + } else if (!*cur_qp || (be32_to_cpu(cqe->my_qpn) & 0xffffff) != (*cur_qp)->mqp.qpn) { /* * We do not have to take the QP table lock here, @@ -598,7 +617,7 @@ repoll: *cur_qp = to_mibqp(mqp); } - wc->qp = &(*cur_qp)->ibqp; + wc->qp = is_xrc_recv ? NULL: &(*cur_qp)->ibqp; if (is_send) { wq = &(*cur_qp)->sq; @@ -608,6 +627,10 @@ repoll: } wc->wr_id = wq->wrid[wq->tail & (wq->wqe_cnt - 1)]; ++wq->tail; + } else if (is_xrc_recv) { + wqe_ctr = be16_to_cpu(cqe->wqe_index); + wc->wr_id = srq->wrid[wqe_ctr]; + mlx4_ib_free_srq_wqe(srq, wqe_ctr); } else if ((*cur_qp)->ibqp.srq) { srq = to_msrq((*cur_qp)->ibqp.srq); wqe_ctr = be16_to_cpu(cqe->wqe_index); @@ -736,6 +759,10 @@ void __mlx4_ib_cq_clean(struct mlx4_ib_c int nfreed = 0; struct mlx4_cqe *cqe, *dest; u8 owner_bit; + int is_xrc_srq = 0; + + if (srq && srq->ibsrq.xrc_cq) + is_xrc_srq = 1; /* * First we need to find the current producer index, so we @@ -754,7 +781,9 @@ void __mlx4_ib_cq_clean(struct mlx4_ib_c */ while ((int) --prod_index - (int) cq->mcq.cons_index >= 0) { cqe = get_cqe(cq, prod_index & cq->ibcq.cqe); - if ((be32_to_cpu(cqe->my_qpn) & 0xffffff) == qpn) { + if (((be32_to_cpu(cqe->my_qpn) & 0xffffff) == qpn) || + (is_xrc_srq && + (be32_to_cpu(cqe->g_mlpath_rqpn) & 0xffffff) == srq->msrq.srqn)) { if (srq && !(cqe->owner_sr_opcode & MLX4_CQE_IS_SEND_MASK)) mlx4_ib_free_srq_wqe(srq, be16_to_cpu(cqe->wqe_index)); ++nfreed; Index: infiniband/drivers/net/mlx4/mlx4.h =================================================================== --- infiniband.orig/drivers/net/mlx4/mlx4.h 2008-06-23 11:43:55.000000000 +0300 +++ infiniband/drivers/net/mlx4/mlx4.h 2008-06-23 13:56:33.000000000 +0300 @@ -221,7 +221,6 @@ struct mlx4_eq_table { struct mlx4_srq_table { struct mlx4_bitmap bitmap; spinlock_t lock; - struct radix_tree_root tree; struct mlx4_icm_table table; struct mlx4_icm_table cmpt_table; }; Index: infiniband/drivers/net/mlx4/srq.c =================================================================== --- infiniband.orig/drivers/net/mlx4/srq.c 2008-06-23 11:43:55.000000000 +0300 +++ infiniband/drivers/net/mlx4/srq.c 2008-06-23 13:56:33.000000000 +0300 @@ -64,7 +64,7 @@ void mlx4_srq_event(struct mlx4_dev *dev spin_lock(&srq_table->lock); - srq = radix_tree_lookup(&srq_table->tree, srqn & (dev->caps.num_srqs - 1)); + srq = radix_tree_lookup(&dev->srq_table_tree, srqn & (dev->caps.num_srqs - 1)); if (srq) atomic_inc(&srq->refcount); @@ -131,7 +131,7 @@ int mlx4_srq_alloc(struct mlx4_dev *dev, goto err_put; spin_lock_irq(&srq_table->lock); - err = radix_tree_insert(&srq_table->tree, srq->srqn, srq); + err = radix_tree_insert(&dev->srq_table_tree, srq->srqn, srq); spin_unlock_irq(&srq_table->lock); if (err) goto err_cmpt_put; @@ -170,7 +170,7 @@ int mlx4_srq_alloc(struct mlx4_dev *dev, err_radix: spin_lock_irq(&srq_table->lock); - radix_tree_delete(&srq_table->tree, srq->srqn); + radix_tree_delete(&dev->srq_table_tree, srq->srqn); spin_unlock_irq(&srq_table->lock); err_cmpt_put: @@ -186,18 +186,29 @@ err_out: } EXPORT_SYMBOL_GPL(mlx4_srq_alloc); -void mlx4_srq_free(struct mlx4_dev *dev, struct mlx4_srq *srq) +void mlx4_srq_invalidate(struct mlx4_dev *dev, struct mlx4_srq *srq) { - struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table; int err; err = mlx4_HW2SW_SRQ(dev, NULL, srq->srqn); if (err) mlx4_warn(dev, "HW2SW_SRQ failed (%d) for SRQN %06x\n", err, srq->srqn); +} +EXPORT_SYMBOL_GPL(mlx4_srq_invalidate); + +void mlx4_srq_remove(struct mlx4_dev *dev, struct mlx4_srq *srq) +{ + struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table; spin_lock_irq(&srq_table->lock); - radix_tree_delete(&srq_table->tree, srq->srqn); + radix_tree_delete(&dev->srq_table_tree, srq->srqn); spin_unlock_irq(&srq_table->lock); +} +EXPORT_SYMBOL_GPL(mlx4_srq_remove); + +void mlx4_srq_free(struct mlx4_dev *dev, struct mlx4_srq *srq) +{ + struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table; if (atomic_dec_and_test(&srq->refcount)) complete(&srq->free); @@ -243,7 +254,7 @@ int mlx4_init_srq_table(struct mlx4_dev int err; spin_lock_init(&srq_table->lock); - INIT_RADIX_TREE(&srq_table->tree, GFP_ATOMIC); + INIT_RADIX_TREE(&dev->srq_table_tree, GFP_ATOMIC); err = mlx4_bitmap_init(&srq_table->bitmap, dev->caps.num_srqs, dev->caps.num_srqs - 1, dev->caps.reserved_srqs); Index: infiniband/include/linux/mlx4/device.h =================================================================== --- infiniband.orig/include/linux/mlx4/device.h 2008-06-23 11:43:55.000000000 +0300 +++ infiniband/include/linux/mlx4/device.h 2008-06-23 13:56:33.000000000 +0300 @@ -326,6 +326,7 @@ struct mlx4_dev { unsigned long flags; struct mlx4_caps caps; struct radix_tree_root qp_table_tree; + struct radix_tree_root srq_table_tree; u32 rev_id; char board_id[MLX4_BOARD_ID_LEN]; }; Index: infiniband/include/linux/mlx4/srq.h =================================================================== --- infiniband.orig/include/linux/mlx4/srq.h 2008-06-23 11:41:51.000000000 +0300 +++ infiniband/include/linux/mlx4/srq.h 2008-06-23 13:56:33.000000000 +0300 @@ -33,10 +33,21 @@ #ifndef MLX4_SRQ_H #define MLX4_SRQ_H +#include +#include + struct mlx4_wqe_srq_next_seg { u16 reserved1; __be16 next_wqe_index; u32 reserved2[3]; }; +void mlx4_srq_invalidate(struct mlx4_dev *dev, struct mlx4_srq *srq); +void mlx4_srq_remove(struct mlx4_dev *dev, struct mlx4_srq *srq); + +static inline struct mlx4_srq *__mlx4_srq_lookup(struct mlx4_dev *dev, u32 srqn) +{ + return radix_tree_lookup(&dev->srq_table_tree, srqn & (dev->caps.num_srqs - 1)); +} + #endif /* MLX4_SRQ_H */ Index: infiniband/drivers/infiniband/hw/mlx4/srq.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/srq.c 2008-06-23 11:43:55.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/srq.c 2008-06-23 13:56:33.000000000 +0300 @@ -181,11 +181,13 @@ struct ib_srq *mlx4_ib_create_xrc_srq(st srq->msrq.event = mlx4_ib_srq_event; - if (pd->uobject) + if (pd->uobject) { if (ib_copy_to_udata(udata, &srq->msrq.srqn, sizeof (__u32))) { err = -EFAULT; goto err_wrid; } + } else + srq->ibsrq.xrc_srq_num = srq->msrq.srqn; init_attr->attr.max_wr = srq->msrq.max - 1; @@ -271,6 +273,18 @@ int mlx4_ib_destroy_srq(struct ib_srq *s { struct mlx4_ib_dev *dev = to_mdev(srq->device); struct mlx4_ib_srq *msrq = to_msrq(srq); + struct mlx4_ib_cq *cq; + + mlx4_srq_invalidate(dev->dev, &msrq->msrq); + + if (srq->xrc_cq && !srq->uobject) { + cq = to_mcq(srq->xrc_cq); + spin_lock_irq(&cq->lock); + __mlx4_ib_cq_clean(cq, -1, msrq); + mlx4_srq_remove(dev->dev, &msrq->msrq); + spin_unlock_irq(&cq->lock); + } else + mlx4_srq_remove(dev->dev, &msrq->msrq); mlx4_srq_free(dev->dev, &msrq->msrq); mlx4_mtt_cleanup(dev->dev, &msrq->mtt); Index: infiniband/drivers/infiniband/hw/mlx4/qp.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/qp.c 2008-06-23 11:43:55.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/qp.c 2008-06-23 13:56:33.000000000 +0300 @@ -261,6 +261,7 @@ static int send_wqe_overhead(enum ib_qp_ case IB_QPT_UC: return sizeof (struct mlx4_wqe_ctrl_seg) + sizeof (struct mlx4_wqe_raddr_seg); + case IB_QPT_XRC: case IB_QPT_RC: return sizeof (struct mlx4_wqe_ctrl_seg) + sizeof (struct mlx4_wqe_atomic_seg) + @@ -1499,6 +1500,10 @@ int mlx4_ib_post_send(struct ib_qp *ibqp size = sizeof *ctrl / 16; switch (ibqp->qp_type) { + case IB_QPT_XRC: + ctrl->srcrb_flags |= + cpu_to_be32(wr->xrc_remote_srq_num << 8); + /* fall thru */ case IB_QPT_RC: case IB_QPT_UC: switch (wr->opcode) { @@ -1813,7 +1818,8 @@ int mlx4_ib_query_qp(struct ib_qp *ibqp, qp_attr->qp_access_flags = to_ib_qp_access_flags(be32_to_cpu(context.params2)); - if (qp->ibqp.qp_type == IB_QPT_RC || qp->ibqp.qp_type == IB_QPT_UC) { + if (qp->ibqp.qp_type == IB_QPT_RC || qp->ibqp.qp_type == IB_QPT_UC || + qp->ibqp.qp_type == IB_QPT_XRC) { to_ib_ah_attr(dev->dev, &qp_attr->ah_attr, &context.pri_path); to_ib_ah_attr(dev->dev, &qp_attr->alt_ah_attr, &context.alt_path); qp_attr->alt_pkey_index = context.alt_path.pkey_index & 0x7f; From jackm at dev.mellanox.co.il Mon Jun 23 06:02:02 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:02:02 +0300 Subject: [ofa-general] [PATCH 11/11] mlx4: XRC receive-only QPs Message-ID: <200806231602.02721.jackm@dev.mellanox.co.il> mlx4: Implement XRC receive-only QP support. Support for XRC RCV-only QP (requested by userspace, but resides in kernel space). Signed-off-by: Jack Morgenstein Index: infiniband/drivers/infiniband/hw/mlx4/mlx4_ib.h =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-23 14:07:54.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-23 14:08:03.000000000 +0300 @@ -58,6 +58,8 @@ struct mlx4_ib_pd { struct mlx4_ib_xrcd { struct ib_xrcd ibxrcd; u32 xrcdn; + struct ib_pd *pd; + struct ib_cq *cq; }; struct mlx4_ib_cq_buf { @@ -106,7 +108,8 @@ struct mlx4_ib_wq { }; enum mlx4_ib_qp_flags { - MLX4_IB_QP_LSO = 1 << 0 + MLX4_IB_QP_LSO = 1 << 0, + MLX4_IB_XRC_RCV = 1 << 1 }; struct mlx4_ib_qp { @@ -129,6 +132,7 @@ struct mlx4_ib_qp { int buf_size; struct mutex mutex; u32 flags; + struct list_head xrc_reg_list; u16 xrcdn; u8 port; u8 alt_port; @@ -172,6 +176,7 @@ struct mlx4_ib_dev { spinlock_t sm_lock; struct mutex cap_mask_mutex; + struct mutex xrc_reg_mutex; }; static inline struct mlx4_ib_dev *to_mdev(struct ib_device *ibdev) @@ -309,6 +314,16 @@ int mlx4_ib_map_phys_fmr(struct ib_fmr * u64 iova); int mlx4_ib_unmap_fmr(struct list_head *fmr_list); int mlx4_ib_fmr_dealloc(struct ib_fmr *fmr); +int mlx4_ib_create_xrc_rcv_qp(struct ib_qp_init_attr *init_attr, + u32 *qp_num); +int mlx4_ib_modify_xrc_rcv_qp(struct ib_xrcd *xrcd, u32 qp_num, + struct ib_qp_attr *attr, int attr_mask); +int mlx4_ib_query_xrc_rcv_qp(struct ib_xrcd *xrcd, u32 qp_num, + struct ib_qp_attr *attr, int attr_mask, + struct ib_qp_init_attr *init_attr); +int mlx4_ib_reg_xrc_rcv_qp(struct ib_xrcd *xrcd, void * context, u32 qp_num); +int mlx4_ib_unreg_xrc_rcv_qp(struct ib_xrcd *xrcd, void * context, u32 qp_num); + static inline int mlx4_ib_ah_grh_present(struct mlx4_ib_ah *ah) { Index: infiniband/drivers/infiniband/hw/mlx4/qp.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/qp.c 2008-06-23 14:07:59.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/qp.c 2008-06-23 14:08:03.000000000 +0300 @@ -56,6 +56,12 @@ enum { MLX4_IB_UD_HEADER_SIZE = 72 }; + +struct mlx4_ib_xrc_reg_entry { + struct list_head list; + void *context; +}; + struct mlx4_ib_sqp { struct mlx4_ib_qp qp; int pkey_index; @@ -202,14 +208,15 @@ static inline unsigned pad_wraparound(st static void mlx4_ib_qp_event(struct mlx4_qp *qp, enum mlx4_event type) { struct ib_event event; - struct ib_qp *ibqp = &to_mibqp(qp)->ibqp; + struct mlx4_ib_qp *mqp = to_mibqp(qp); + struct ib_qp *ibqp = &mqp->ibqp; + struct mlx4_ib_xrc_reg_entry *ctx_entry; if (type == MLX4_EVENT_TYPE_PATH_MIG) to_mibqp(qp)->port = to_mibqp(qp)->alt_port; if (ibqp->event_handler) { event.device = ibqp->device; - event.element.qp = ibqp; switch (type) { case MLX4_EVENT_TYPE_PATH_MIG: event.event = IB_EVENT_PATH_MIG; @@ -241,7 +248,16 @@ static void mlx4_ib_qp_event(struct mlx4 return; } - ibqp->event_handler(&event, ibqp->qp_context); + if (!(ibqp->qp_type == IB_QPT_XRC && + mqp->flags & MLX4_IB_XRC_RCV)) { + event.element.qp = ibqp; + ibqp->event_handler(&event, ibqp->qp_context); + } else { + event.event |= IB_XRC_QP_EVENT_FLAG; + event.element.xrc_qp_num = ibqp->qp_num; + list_for_each_entry(ctx_entry, &mqp->xrc_reg_list, list) + ibqp->event_handler(&event, ctx_entry->context); + } } } @@ -469,6 +485,10 @@ static int create_qp_common(struct mlx4_ else qp->sq_signal_bits = 0; + if (init_attr->qp_type == IB_QPT_XRC && + init_attr->create_flags & QP_CREATE_XRC_RCV) + qp->flags |= MLX4_IB_XRC_RCV; + err = set_rq_size(dev, &init_attr->cap, !!pd->uobject, !!init_attr->srq || !!init_attr->xrc_domain , qp); if (err) @@ -708,6 +728,8 @@ struct ib_qp *mlx4_ib_create_qp(struct i if (!qp) return ERR_PTR(-ENOMEM); + memset(qp, 0, sizeof *qp); + INIT_LIST_HEAD(&qp->xrc_reg_list); err = create_qp_common(dev, pd, init_attr, udata, 0, qp); if (err) { kfree(qp); @@ -1872,3 +1894,260 @@ out: return err; } +int mlx4_ib_create_xrc_rcv_qp(struct ib_qp_init_attr *init_attr, + u32 *qp_num) +{ + struct mlx4_ib_dev *dev = to_mdev(init_attr->xrc_domain->device); + struct mlx4_ib_xrcd *xrcd = to_mxrcd(init_attr->xrc_domain); + struct ib_qp_init_attr ia = *init_attr; + struct mlx4_ib_qp *qp; + struct ib_qp *ibqp; + struct mlx4_ib_xrc_reg_entry *ctx_entry; + + if (!(dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) + return -ENOSYS; + + ctx_entry = kmalloc(sizeof *ctx_entry, GFP_KERNEL); + if (!ctx_entry) + return -ENOMEM; + + ia.qp_type = IB_QPT_XRC; + ia.create_flags = QP_CREATE_XRC_RCV; + ia.recv_cq = ia.send_cq = xrcd->cq; + + ibqp = mlx4_ib_create_qp(xrcd->pd, &ia, NULL); + if (IS_ERR(ibqp)) { + kfree(ctx_entry); + return PTR_ERR(ibqp); + } + + /* set the ibpq attributes which will be used by the mlx4 module */ + ibqp->device = init_attr->xrc_domain->device; + ibqp->pd = xrcd->pd; + ibqp->send_cq = ibqp->recv_cq = xrcd->cq; + ibqp->event_handler = init_attr->event_handler; + ibqp->qp_context = init_attr->qp_context; + ibqp->qp_type = init_attr->qp_type; + ibqp->xrcd = init_attr->xrc_domain; + + qp = to_mqp(ibqp); + + mutex_lock(&qp->mutex); + ctx_entry->context = init_attr->qp_context; + list_add_tail(&ctx_entry->list, &qp->xrc_reg_list); + mutex_unlock(&qp->mutex); + *qp_num = qp->mqp.qpn; + return 0; +} + +int mlx4_ib_modify_xrc_rcv_qp(struct ib_xrcd *ibxrcd, u32 qp_num, + struct ib_qp_attr *attr, int attr_mask) +{ + struct mlx4_ib_dev *dev = to_mdev(ibxrcd->device); + struct mlx4_ib_xrcd *xrcd = to_mxrcd(ibxrcd); + struct mlx4_qp *mqp; + int err; + + if (!(dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) + return -ENOSYS; + + mqp = __mlx4_qp_lookup(dev->dev, qp_num); + if (unlikely(!mqp)) { + printk(KERN_WARNING "mlx4_ib_reg_xrc_rcv_qp: unknown QPN %06x\n", + qp_num); + return -EINVAL; + } + + if (xrcd->xrcdn != to_mxrcd(to_mibqp(mqp)->ibqp.xrcd)->xrcdn) + return -EINVAL; + + err = mlx4_ib_modify_qp(&(to_mibqp(mqp)->ibqp), attr, attr_mask, NULL); + return err; +} + +int mlx4_ib_query_xrc_rcv_qp(struct ib_xrcd *ibxrcd, u32 qp_num, + struct ib_qp_attr *qp_attr, int qp_attr_mask, + struct ib_qp_init_attr *qp_init_attr) +{ + struct mlx4_ib_dev *dev = to_mdev(ibxrcd->device); + struct mlx4_ib_xrcd *xrcd = to_mxrcd(ibxrcd); + struct mlx4_ib_qp *qp; + struct mlx4_qp *mqp; + struct mlx4_qp_context context; + int mlx4_state; + int err; + + if (!(dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) + return -ENOSYS; + + mqp = __mlx4_qp_lookup(dev->dev, qp_num); + if (unlikely(!mqp)) { + printk(KERN_WARNING "mlx4_ib_reg_xrc_rcv_qp: unknown QPN %06x\n", + qp_num); + return -EINVAL; + } + + qp = to_mibqp(mqp); + if (xrcd->xrcdn != to_mxrcd(qp->ibqp.xrcd)->xrcdn) + return -EINVAL; + + if (qp->state == IB_QPS_RESET) { + qp_attr->qp_state = IB_QPS_RESET; + goto done; + } + + err = mlx4_qp_query(dev->dev, mqp, &context); + if (err) + return -EINVAL; + + mlx4_state = be32_to_cpu(context.flags) >> 28; + + qp_attr->qp_state = to_ib_qp_state(mlx4_state); + qp_attr->path_mtu = context.mtu_msgmax >> 5; + qp_attr->path_mig_state = + to_ib_mig_state((be32_to_cpu(context.flags) >> 11) & 0x3); + qp_attr->qkey = be32_to_cpu(context.qkey); + qp_attr->rq_psn = be32_to_cpu(context.rnr_nextrecvpsn) & 0xffffff; + qp_attr->sq_psn = be32_to_cpu(context.next_send_psn) & 0xffffff; + qp_attr->dest_qp_num = be32_to_cpu(context.remote_qpn) & 0xffffff; + qp_attr->qp_access_flags = + to_ib_qp_access_flags(be32_to_cpu(context.params2)); + + if (qp->ibqp.qp_type == IB_QPT_RC || qp->ibqp.qp_type == IB_QPT_UC || + qp->ibqp.qp_type == IB_QPT_XRC) { + to_ib_ah_attr(dev->dev, &qp_attr->ah_attr, &context.pri_path); + to_ib_ah_attr(dev->dev, &qp_attr->alt_ah_attr, &context.alt_path); + qp_attr->alt_pkey_index = context.alt_path.pkey_index & 0x7f; + qp_attr->alt_port_num = qp_attr->alt_ah_attr.port_num; + } + + qp_attr->pkey_index = context.pri_path.pkey_index & 0x7f; + if (qp_attr->qp_state == IB_QPS_INIT) + qp_attr->port_num = qp->port; + else + qp_attr->port_num = context.pri_path.sched_queue & 0x40 ? 2 : 1; + + /* qp_attr->en_sqd_async_notify is only applicable in modify qp */ + qp_attr->sq_draining = mlx4_state == MLX4_QP_STATE_SQ_DRAINING; + + qp_attr->max_rd_atomic = 1 << ((be32_to_cpu(context.params1) >> 21) & 0x7); + + qp_attr->max_dest_rd_atomic = + 1 << ((be32_to_cpu(context.params2) >> 21) & 0x7); + qp_attr->min_rnr_timer = + (be32_to_cpu(context.rnr_nextrecvpsn) >> 24) & 0x1f; + qp_attr->timeout = context.pri_path.ackto >> 3; + qp_attr->retry_cnt = (be32_to_cpu(context.params1) >> 16) & 0x7; + qp_attr->rnr_retry = (be32_to_cpu(context.params1) >> 13) & 0x7; + qp_attr->alt_timeout = context.alt_path.ackto >> 3; + +done: + qp_attr->cur_qp_state = qp_attr->qp_state; + qp_attr->cap.max_recv_wr = 0; + qp_attr->cap.max_recv_sge = 0; + qp_attr->cap.max_send_wr = 0; + qp_attr->cap.max_send_sge = 0; + qp_attr->cap.max_inline_data = 0; + qp_init_attr->cap = qp_attr->cap; + + return 0; +} + +int mlx4_ib_reg_xrc_rcv_qp(struct ib_xrcd *xrcd, void *context, u32 qp_num) +{ + + struct mlx4_ib_xrcd *mxrcd = to_mxrcd(xrcd); + + struct mlx4_qp *mqp; + struct mlx4_ib_qp *mibqp; + struct mlx4_ib_xrc_reg_entry *ctx_entry, *tmp; + int err = -EINVAL; + + mutex_lock(&to_mdev(xrcd->device)->xrc_reg_mutex); + mqp = __mlx4_qp_lookup(to_mdev(xrcd->device)->dev, qp_num); + if (unlikely(!mqp)) { + printk(KERN_WARNING "mlx4_ib_reg_xrc_rcv_qp: unknown QPN %06x\n", + qp_num); + goto err_out; + } + + mibqp = to_mibqp(mqp); + + if (mxrcd->xrcdn != to_mxrcd(mibqp->ibqp.xrcd)->xrcdn) + goto err_out; + + ctx_entry = kmalloc(sizeof *ctx_entry, GFP_KERNEL); + if (!ctx_entry) { + err = -ENOMEM; + goto err_out; + } + + mutex_lock(&mibqp->mutex); + list_for_each_entry(tmp, &mibqp->xrc_reg_list, list) + if (tmp->context == context) { + mutex_unlock(&mibqp->mutex); + kfree(ctx_entry); + mutex_unlock(&to_mdev(xrcd->device)->xrc_reg_mutex); + return 0; + } + + ctx_entry->context = context; + list_add_tail(&ctx_entry->list, &mibqp->xrc_reg_list); + mutex_unlock(&mibqp->mutex); + mutex_unlock(&to_mdev(xrcd->device)->xrc_reg_mutex); + return 0; + +err_out: + mutex_unlock(&to_mdev(xrcd->device)->xrc_reg_mutex); + return err; +} + +int mlx4_ib_unreg_xrc_rcv_qp(struct ib_xrcd *xrcd, void *context, u32 qp_num) +{ + + struct mlx4_ib_xrcd *mxrcd = to_mxrcd(xrcd); + + struct mlx4_qp *mqp; + struct mlx4_ib_qp *mibqp; + struct mlx4_ib_xrc_reg_entry *ctx_entry, *tmp; + int found = 0; + int err = -EINVAL; + + mutex_lock(&to_mdev(xrcd->device)->xrc_reg_mutex); + mqp = __mlx4_qp_lookup(to_mdev(xrcd->device)->dev, qp_num); + if (unlikely(!mqp)) { + printk(KERN_WARNING "mlx4_ib_unreg_xrc_rcv_qp: unknown QPN %06x\n", + qp_num); + goto err_out; + } + + mibqp = to_mibqp(mqp); + + if (mxrcd->xrcdn != (mibqp->xrcdn & 0xffff)) + goto err_out; + + mutex_lock(&mibqp->mutex); + list_for_each_entry_safe(ctx_entry, tmp, &mibqp->xrc_reg_list, list) + if (ctx_entry->context == context) { + found = 1; + list_del(&ctx_entry->list); + kfree(ctx_entry); + break; + } + + mutex_unlock(&mibqp->mutex); + if (!found) + goto err_out; + + /* destroy the QP if the registration list is empty */ + if (list_empty(&mibqp->xrc_reg_list)) + mlx4_ib_destroy_qp(&mibqp->ibqp); + + mutex_unlock(&to_mdev(xrcd->device)->xrc_reg_mutex); + return 0; + +err_out: + mutex_unlock(&to_mdev(xrcd->device)->xrc_reg_mutex); + return err; +} + Index: infiniband/drivers/infiniband/hw/mlx4/main.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/main.c 2008-06-23 14:07:54.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/main.c 2008-06-23 14:08:03.000000000 +0300 @@ -412,6 +412,7 @@ static struct ib_pd *mlx4_ib_alloc_pd(st if (!pd) return ERR_PTR(-ENOMEM); + memset(pd, 0, sizeof *pd); err = mlx4_pd_alloc(to_mdev(ibdev)->dev, &pd->pdn); if (err) { kfree(pd); @@ -448,12 +449,18 @@ static int mlx4_ib_mcg_detach(struct ib_ &to_mqp(ibqp)->mqp, gid->raw); } +static void mlx4_dummy_comp_handler(struct ib_cq *cq, void *cq_context) +{ +} + static struct ib_xrcd *mlx4_ib_alloc_xrcd(struct ib_device *ibdev, struct ib_ucontext *context, struct ib_udata *udata) { struct mlx4_ib_xrcd *xrcd; struct mlx4_ib_dev *mdev = to_mdev(ibdev); + struct ib_pd *pd; + struct ib_cq *cq; int err; if (!(mdev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) @@ -464,23 +471,51 @@ static struct ib_xrcd *mlx4_ib_alloc_xrc return ERR_PTR(-ENOMEM); err = mlx4_xrcd_alloc(mdev->dev, &xrcd->xrcdn); - if (err) { - kfree(xrcd); - return ERR_PTR(err); + if (err) + goto err_xrcd; + + pd = mlx4_ib_alloc_pd(ibdev,NULL,NULL); + if (IS_ERR(pd)) { + err = PTR_ERR(pd); + goto err_pd; } + pd->device = ibdev; + + cq = mlx4_ib_create_cq(ibdev, 1, 0, NULL, NULL); + if (IS_ERR(cq)) { + err = PTR_ERR(cq); + goto err_cq; + } + cq->device = ibdev; + cq->comp_handler = mlx4_dummy_comp_handler; if (context) if (ib_copy_to_udata(udata, &xrcd->xrcdn, sizeof (__u32))) { - mlx4_xrcd_free(mdev->dev, xrcd->xrcdn); - kfree(xrcd); - return ERR_PTR(-EFAULT); + err = -EFAULT; + goto err_copy; } + xrcd->cq = cq; + xrcd->pd = pd; return &xrcd->ibxrcd; + +err_copy: + mlx4_ib_destroy_cq(cq); +err_cq: + mlx4_ib_dealloc_pd(pd); +err_pd: + mlx4_xrcd_free(mdev->dev, xrcd->xrcdn); +err_xrcd: + kfree(xrcd); + return ERR_PTR(err); } static int mlx4_ib_dealloc_xrcd(struct ib_xrcd *xrcd) { + struct mlx4_ib_xrcd *mxrcd = to_mxrcd(xrcd); + + mlx4_ib_destroy_cq(mxrcd->cq); + mlx4_ib_dealloc_pd(mxrcd->pd); mlx4_xrcd_free(to_mdev(xrcd->device)->dev, to_mxrcd(xrcd)->xrcdn); kfree(xrcd); @@ -678,18 +713,28 @@ static void *mlx4_ib_add(struct mlx4_dev ibdev->ib_dev.create_xrc_srq = mlx4_ib_create_xrc_srq; ibdev->ib_dev.alloc_xrcd = mlx4_ib_alloc_xrcd; ibdev->ib_dev.dealloc_xrcd = mlx4_ib_dealloc_xrcd; + ibdev->ib_dev.create_xrc_rcv_qp = mlx4_ib_create_xrc_rcv_qp; + ibdev->ib_dev.modify_xrc_rcv_qp = mlx4_ib_modify_xrc_rcv_qp; + ibdev->ib_dev.query_xrc_rcv_qp = mlx4_ib_query_xrc_rcv_qp; + ibdev->ib_dev.reg_xrc_rcv_qp = mlx4_ib_reg_xrc_rcv_qp; + ibdev->ib_dev.unreg_xrc_rcv_qp = mlx4_ib_unreg_xrc_rcv_qp; ibdev->ib_dev.uverbs_cmd_mask |= (1ull << IB_USER_VERBS_CMD_CREATE_XRC_SRQ) | (1ull << IB_USER_VERBS_CMD_OPEN_XRC_DOMAIN) | - (1ull << IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN); + (1ull << IB_USER_VERBS_CMD_CLOSE_XRC_DOMAIN) | + (1ull << IB_USER_VERBS_CMD_CREATE_XRC_RCV_QP) | + (1ull << IB_USER_VERBS_CMD_MODIFY_XRC_RCV_QP) | + (1ull << IB_USER_VERBS_CMD_QUERY_XRC_RCV_QP) | + (1ull << IB_USER_VERBS_CMD_REG_XRC_RCV_QP) | + (1ull << IB_USER_VERBS_CMD_UNREG_XRC_RCV_QP); } - if (init_node_data(ibdev)) goto err_map; spin_lock_init(&ibdev->sm_lock); mutex_init(&ibdev->cap_mask_mutex); + mutex_init(&ibdev->xrc_reg_mutex); if (ib_register_device(&ibdev->ib_dev)) goto err_map; Index: infiniband/drivers/infiniband/hw/mlx4/cq.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mlx4/cq.c 2008-06-23 14:07:59.000000000 +0300 +++ infiniband/drivers/infiniband/hw/mlx4/cq.c 2008-06-23 14:08:03.000000000 +0300 @@ -178,6 +178,7 @@ struct ib_cq *mlx4_ib_create_cq(struct i if (!cq) return ERR_PTR(-ENOMEM); + memset(cq, 0, sizeof *cq); entries = roundup_pow_of_two(entries + 1); cq->ibcq.cqe = entries - 1; mutex_init(&cq->resize_mutex); From huwfgpwte at bpslaw.com Mon Jun 23 06:27:54 2008 From: huwfgpwte at bpslaw.com (Dominick Eaton) Date: Mon, 23 Jun 2008 14:27:54 +0100 Subject: [ofa-general] Luxury Message-ID: <01c8d53d$52dd1100$4477344e@huwfgpwte> We are now bringing forward the replicas of the watches,Accessories, boxsets, Cufflinks, Keychains, Lighters, Pens, Tiffany &Co Jewelry. A replica presupposes the exact copy of the original, that is thequality and all the other features the original possesses. For instance,youmight get a $18000 Rolex replica for a $229.00 price only! Our replica is ofsuch an amazing quality that it can be destinguished from the original only inthe jewellery workshop. You're anxiously wanting to be glamorous enough fordemanding high soiciety standarts, but your earnings are not that sky high? Youare willing to reward your lover with an outrageous gift but you don't wantthat to empty your wallet? Come, visit our shop! P.S When applying for a well-paid job, when flirting with agirl, when asking for a credit in the bank you get your chances doubled with ARolex President replica on your hand! PSS: The only missing thing here is a luxury suit! menireoplis poBqsyfpm7D7BqBmzlE -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackm at dev.mellanox.co.il Mon Jun 23 06:01:29 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 23 Jun 2008 16:01:29 +0300 Subject: [ofa-general] ***SPAM*** [PATCH 3/11] libmlx4: XRC base implementation Message-ID: <200806231601.29341.jackm@dev.mellanox.co.il> XRC implementation, consolidated: xrc ops were moved to their own structure at the end of struct ibv_context (to preserve binary compatibility). Check for ibv_context.xrc_ops member via AC_CHECK_MEMBER XRC QPs have MSB set in qp number, for identification in completion handling. Various bug fixes. (OFED 1.3 commit 39fe7f47e8fc07f356098df048d00740ba585fc5) Signed-off-by: Jack Morgenstein Index: libmlx4/configure.in =================================================================== --- libmlx4.orig/configure.in 2008-01-30 10:33:08.000000000 +0200 +++ libmlx4/configure.in 2008-06-23 09:04:50.000000000 +0300 @@ -42,6 +42,9 @@ AC_CHECK_HEADER(valgrind/memcheck.h, dnl Checks for typedefs, structures, and compiler characteristics. AC_C_CONST AC_CHECK_SIZEOF(long) +AC_CHECK_MEMBER(struct ibv_context.xrc_ops, + [AC_DEFINE([HAVE_IBV_XRC_OPS], 1, [Define to 1 if xrc_ops is a member of ibv_context])],, + [#include ]) dnl Checks for library functions AC_CHECK_FUNC(ibv_read_sysfs_file, [], Index: libmlx4/src/cq.c =================================================================== --- libmlx4.orig/src/cq.c 2008-06-03 15:41:58.000000000 +0300 +++ libmlx4/src/cq.c 2008-06-23 09:04:50.000000000 +0300 @@ -196,10 +196,12 @@ static int mlx4_poll_one(struct mlx4_cq struct mlx4_cqe *cqe; struct mlx4_srq *srq; uint32_t qpn; + uint32_t srqn; uint32_t g_mlpath_rqpn; uint16_t wqe_index; int is_error; int is_send; + int is_src_recv = 0; cqe = next_cqe_sw(cq); if (!cqe) @@ -221,20 +223,30 @@ static int mlx4_poll_one(struct mlx4_cq is_error = (cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) == MLX4_CQE_OPCODE_ERROR; - if (!*cur_qp || - (ntohl(cqe->my_qpn) & 0xffffff) != (*cur_qp)->ibv_qp.qp_num) { - /* - * We do not have to take the QP table lock here, - * because CQs will be locked while QPs are removed - * from the table. - */ - *cur_qp = mlx4_find_qp(to_mctx(cq->ibv_cq.context), - ntohl(cqe->my_qpn) & 0xffffff); - if (!*cur_qp) + if (qpn & MLX4_XRC_QPN_BIT && !is_send) { + srqn = ntohl(cqe->g_mlpath_rqpn) & 0xffffff; + /* + * We do not have to take the XRC SRQ table lock here, + * because CQs will be locked while XRC SRQs are removed + * from the table. + */ + srq = mlx4_find_xrc_srq(to_mctx(cq->ibv_cq.context), srqn); + if (!srq) return CQ_POLL_ERR; - } + is_src_recv = 1; + } else if (!*cur_qp || (qpn & 0xffffff) != (*cur_qp)->ibv_qp.qp_num) { + /* + * We do not have to take the QP table lock here, + * because CQs will be locked while QPs are removed + * from the table. + */ + *cur_qp = mlx4_find_qp(to_mctx(cq->ibv_cq.context), + qpn & 0xffffff); + if (!*cur_qp) + return CQ_POLL_ERR; + } - wc->qp_num = (*cur_qp)->ibv_qp.qp_num; + wc->qp_num = qpn & 0xffffff; if (is_send) { wq = &(*cur_qp)->sq; @@ -242,6 +254,10 @@ static int mlx4_poll_one(struct mlx4_cq wq->tail += (uint16_t) (wqe_index - (uint16_t) wq->tail); wc->wr_id = wq->wrid[wq->tail & (wq->wqe_cnt - 1)]; ++wq->tail; + } else if (is_src_recv) { + wqe_index = htons(cqe->wqe_index); + wc->wr_id = srq->wrid[wqe_index]; + mlx4_free_srq_wqe(srq, wqe_index); } else if ((*cur_qp)->ibv_qp.srq) { srq = to_msrq((*cur_qp)->ibv_qp.srq); wqe_index = htons(cqe->wqe_index); @@ -387,6 +403,10 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, uint32_t prod_index; uint8_t owner_bit; int nfreed = 0; + int is_xrc_srq = 0; + + if (srq && srq->ibv_srq.xrc_cq) + is_xrc_srq = 1; /* * First we need to find the current producer index, so we @@ -405,7 +425,12 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, */ while ((int) --prod_index - (int) cq->cons_index >= 0) { cqe = get_cqe(cq, prod_index & cq->ibv_cq.cqe); - if ((ntohl(cqe->my_qpn) & 0xffffff) == qpn) { + if (is_xrc_srq && + (ntohl(cqe->g_mlpath_rqpn & 0xffffff) == srq->srqn) && + !(cqe->owner_sr_opcode & MLX4_CQE_IS_SEND_MASK)) { + mlx4_free_srq_wqe(srq, ntohs(cqe->wqe_index)); + ++nfreed; + } else if ((ntohl(cqe->my_qpn) & 0xffffff) == qpn) { if (srq && !(cqe->owner_sr_opcode & MLX4_CQE_IS_SEND_MASK)) mlx4_free_srq_wqe(srq, ntohs(cqe->wqe_index)); ++nfreed; Index: libmlx4/src/mlx4-abi.h =================================================================== --- libmlx4.orig/src/mlx4-abi.h 2008-01-30 10:33:08.000000000 +0200 +++ libmlx4/src/mlx4-abi.h 2008-06-23 09:04:50.000000000 +0300 @@ -68,6 +68,14 @@ struct mlx4_resize_cq { __u64 buf_addr; }; +#ifdef HAVE_IBV_XRC_OPS +struct mlx4_create_xrc_srq { + struct ibv_create_xrc_srq ibv_cmd; + __u64 buf_addr; + __u64 db_addr; +}; +#endif + struct mlx4_create_srq { struct ibv_create_srq ibv_cmd; __u64 buf_addr; @@ -90,4 +98,12 @@ struct mlx4_create_qp { __u8 reserved[5]; }; +#ifdef HAVE_IBV_XRC_OPS +struct mlx4_open_xrc_domain_resp { + struct ibv_open_xrc_domain_resp ibv_resp; + __u32 xrcdn; + __u32 reserved; +}; +#endif + #endif /* MLX4_ABI_H */ Index: libmlx4/src/mlx4.c =================================================================== --- libmlx4.orig/src/mlx4.c 2008-06-03 15:39:26.000000000 +0300 +++ libmlx4/src/mlx4.c 2008-06-23 09:04:50.000000000 +0300 @@ -68,6 +68,14 @@ struct { HCA(MELLANOX, 0x673c), /* MT25408 "Hermon" QDR PCIe gen2 */ }; +#ifdef HAVE_IBV_XRC_OPS +static struct ibv_xrc_ops mlx4_xrc_ops = { + .create_xrc_srq = mlx4_create_xrc_srq, + .open_xrc_domain = mlx4_open_xrc_domain, + .close_xrc_domain = mlx4_close_xrc_domain, +}; +#endif + static struct ibv_context_ops mlx4_ctx_ops = { .query_device = mlx4_query_device, .query_port = mlx4_query_port, @@ -123,6 +131,15 @@ static struct ibv_context *mlx4_alloc_co for (i = 0; i < MLX4_QP_TABLE_SIZE; ++i) context->qp_table[i].refcnt = 0; + context->num_xrc_srqs = resp.qp_tab_size; + context->xrc_srq_table_shift = ffs(context->num_xrc_srqs) - 1 + - MLX4_XRC_SRQ_TABLE_BITS; + context->xrc_srq_table_mask = (1 << context->xrc_srq_table_shift) - 1; + + pthread_mutex_init(&context->xrc_srq_table_mutex, NULL); + for (i = 0; i < MLX4_XRC_SRQ_TABLE_SIZE; ++i) + context->xrc_srq_table[i].refcnt = 0; + for (i = 0; i < MLX4_NUM_DB_TYPE; ++i) context->db_list[i] = NULL; @@ -155,6 +172,9 @@ static struct ibv_context *mlx4_alloc_co pthread_spin_init(&context->uar_lock, PTHREAD_PROCESS_PRIVATE); context->ibv_ctx.ops = mlx4_ctx_ops; +#ifdef HAVE_IBV_XRC_OPS + context->ibv_ctx.xrc_ops = &mlx4_xrc_ops; +#endif return &context->ibv_ctx; Index: libmlx4/src/mlx4.h =================================================================== --- libmlx4.orig/src/mlx4.h 2008-06-03 15:41:58.000000000 +0300 +++ libmlx4/src/mlx4.h 2008-06-23 09:04:50.000000000 +0300 @@ -97,6 +97,16 @@ enum { MLX4_QP_TABLE_MASK = MLX4_QP_TABLE_SIZE - 1 }; +enum { + MLX4_XRC_SRQ_TABLE_BITS = 8, + MLX4_XRC_SRQ_TABLE_SIZE = 1 << MLX4_XRC_SRQ_TABLE_BITS, + MLX4_XRC_SRQ_TABLE_MASK = MLX4_XRC_SRQ_TABLE_SIZE - 1 +}; + +enum { + MLX4_XRC_QPN_BIT = (1 << 23) +}; + enum mlx4_db_type { MLX4_DB_TYPE_CQ, MLX4_DB_TYPE_RQ, @@ -157,6 +167,15 @@ struct mlx4_context { int qp_table_shift; int qp_table_mask; + struct { + struct mlx4_srq **table; + int refcnt; + } xrc_srq_table[MLX4_XRC_SRQ_TABLE_SIZE]; + pthread_mutex_t xrc_srq_table_mutex; + int num_xrc_srqs; + int xrc_srq_table_shift; + int xrc_srq_table_mask; + struct mlx4_db_page *db_list[MLX4_NUM_DB_TYPE]; pthread_mutex_t db_list_mutex; }; @@ -243,6 +262,11 @@ struct mlx4_ah { struct mlx4_av av; }; +struct mlx4_xrc_domain { + struct ibv_xrc_domain ibv_xrcd; + uint32_t xrcdn; +}; + static inline unsigned long align(unsigned long val, unsigned long align) { return (val + align - 1) & ~(align - 1); @@ -287,6 +311,13 @@ static inline struct mlx4_ah *to_mah(str return to_mxxx(ah, ah); } +#ifdef HAVE_IBV_XRC_OPS +static inline struct mlx4_xrc_domain *to_mxrcd(struct ibv_xrc_domain *ibxrcd) +{ + return to_mxxx(xrcd, xrc_domain); +} +#endif + int mlx4_alloc_buf(struct mlx4_buf *buf, size_t size, int page_size); void mlx4_free_buf(struct mlx4_buf *buf); @@ -333,6 +364,10 @@ void mlx4_free_srq_wqe(struct mlx4_srq * int mlx4_post_srq_recv(struct ibv_srq *ibsrq, struct ibv_recv_wr *wr, struct ibv_recv_wr **bad_wr); +struct mlx4_srq *mlx4_find_xrc_srq(struct mlx4_context *ctx, uint32_t xrc_srqn); +int mlx4_store_xrc_srq(struct mlx4_context *ctx, uint32_t xrc_srqn, + struct mlx4_srq *srq); +void mlx4_clear_xrc_srq(struct mlx4_context *ctx, uint32_t xrc_srqn); struct ibv_qp *mlx4_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *attr); int mlx4_query_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, @@ -363,5 +398,16 @@ int mlx4_alloc_av(struct mlx4_pd *pd, st void mlx4_free_av(struct mlx4_ah *ah); int mlx4_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); int mlx4_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); +#ifdef HAVE_IBV_XRC_OPS +struct ibv_srq *mlx4_create_xrc_srq(struct ibv_pd *pd, + struct ibv_xrc_domain *xrc_domain, + struct ibv_cq *xrc_cq, + struct ibv_srq_init_attr *attr); +struct ibv_xrc_domain *mlx4_open_xrc_domain(struct ibv_context *context, + int fd, int oflag); + +int mlx4_close_xrc_domain(struct ibv_xrc_domain *d); +#endif + #endif /* MLX4_H */ Index: libmlx4/src/qp.c =================================================================== --- libmlx4.orig/src/qp.c 2008-06-03 15:39:26.000000000 +0300 +++ libmlx4/src/qp.c 2008-06-23 09:04:50.000000000 +0300 @@ -224,7 +224,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, ctrl = wqe = get_send_wqe(qp, ind & (qp->sq.wqe_cnt - 1)); qp->sq.wrid[ind & (qp->sq.wqe_cnt - 1)] = wr->wr_id; - ctrl->srcrb_flags = + ctrl->xrcrb_flags = (wr->send_flags & IBV_SEND_SIGNALED ? htonl(MLX4_WQE_CTRL_CQ_UPDATE) : 0) | (wr->send_flags & IBV_SEND_SOLICITED ? @@ -241,6 +241,9 @@ int mlx4_post_send(struct ibv_qp *ibqp, size = sizeof *ctrl / 16; switch (ibqp->qp_type) { + case IBV_QPT_XRC: + ctrl->xrcrb_flags |= htonl(wr->xrc_remote_srq_num << 8); + /* fall thru */ case IBV_QPT_RC: case IBV_QPT_UC: switch (wr->opcode) { @@ -541,6 +544,7 @@ void mlx4_calc_sq_wqe_size(struct ibv_qp size += sizeof (struct mlx4_wqe_raddr_seg); break; + case IBV_QPT_XRC: case IBV_QPT_RC: size += sizeof (struct mlx4_wqe_raddr_seg); /* @@ -628,6 +632,7 @@ void mlx4_set_sq_sizes(struct mlx4_qp *q case IBV_QPT_UC: case IBV_QPT_RC: + case IBV_QPT_XRC: wqe_size -= sizeof (struct mlx4_wqe_raddr_seg); break; Index: libmlx4/src/srq.c =================================================================== --- libmlx4.orig/src/srq.c 2008-01-30 10:33:08.000000000 +0200 +++ libmlx4/src/srq.c 2008-06-23 09:04:50.000000000 +0300 @@ -167,3 +167,53 @@ int mlx4_alloc_srq_buf(struct ibv_pd *pd return 0; } + +struct mlx4_srq *mlx4_find_xrc_srq(struct mlx4_context *ctx, uint32_t xrc_srqn) +{ + int tind = (xrc_srqn & (ctx->num_xrc_srqs - 1)) >> ctx->xrc_srq_table_shift; + + if (ctx->xrc_srq_table[tind].refcnt) + return ctx->xrc_srq_table[tind].table[xrc_srqn & ctx->xrc_srq_table_mask]; + else + return NULL; +} + +int mlx4_store_xrc_srq(struct mlx4_context *ctx, uint32_t xrc_srqn, + struct mlx4_srq *srq) +{ + int tind = (xrc_srqn & (ctx->num_xrc_srqs - 1)) >> ctx->xrc_srq_table_shift; + int ret = 0; + + pthread_mutex_lock(&ctx->xrc_srq_table_mutex); + + if (!ctx->xrc_srq_table[tind].refcnt) { + ctx->xrc_srq_table[tind].table = calloc(ctx->xrc_srq_table_mask + 1, + sizeof (struct mlx4_srq *)); + if (!ctx->xrc_srq_table[tind].table) { + ret = -1; + goto out; + } + } + + ++ctx->xrc_srq_table[tind].refcnt; + ctx->xrc_srq_table[tind].table[xrc_srqn & ctx->xrc_srq_table_mask] = srq; + +out: + pthread_mutex_unlock(&ctx->xrc_srq_table_mutex); + return ret; +} + +void mlx4_clear_xrc_srq(struct mlx4_context *ctx, uint32_t xrc_srqn) +{ + int tind = (xrc_srqn & (ctx->num_xrc_srqs - 1)) >> ctx->xrc_srq_table_shift; + + pthread_mutex_lock(&ctx->xrc_srq_table_mutex); + + if (!--ctx->xrc_srq_table[tind].refcnt) + free(ctx->xrc_srq_table[tind].table); + else + ctx->xrc_srq_table[tind].table[xrc_srqn & ctx->xrc_srq_table_mask] = NULL; + + pthread_mutex_unlock(&ctx->xrc_srq_table_mutex); +} + Index: libmlx4/src/verbs.c =================================================================== --- libmlx4.orig/src/verbs.c 2008-06-03 15:41:58.000000000 +0300 +++ libmlx4/src/verbs.c 2008-06-23 09:04:50.000000000 +0300 @@ -368,18 +368,36 @@ int mlx4_query_srq(struct ibv_srq *srq, return ibv_cmd_query_srq(srq, attr, &cmd, sizeof cmd); } -int mlx4_destroy_srq(struct ibv_srq *srq) +int mlx4_destroy_srq(struct ibv_srq *ibsrq) { + struct mlx4_srq *srq = to_msrq(ibsrq); + struct mlx4_cq *mcq = NULL; int ret; - ret = ibv_cmd_destroy_srq(srq); - if (ret) + if (ibsrq->xrc_cq) { + /* is an xrc_srq */ + mcq = to_mcq(ibsrq->xrc_cq); + mlx4_cq_clean(mcq, 0, srq); + pthread_spin_lock(&mcq->lock); + mlx4_clear_xrc_srq(to_mctx(ibsrq->context), srq->srqn); + pthread_spin_unlock(&mcq->lock); + } + + ret = ibv_cmd_destroy_srq(ibsrq); + if (ret) { + if (ibsrq->xrc_cq) { + pthread_spin_lock(&mcq->lock); + mlx4_store_xrc_srq(to_mctx(ibsrq->context), + srq->srqn, srq); + pthread_spin_unlock(&mcq->lock); + } return ret; + } - mlx4_free_db(to_mctx(srq->context), MLX4_DB_TYPE_RQ, to_msrq(srq)->db); - mlx4_free_buf(&to_msrq(srq)->buf); - free(to_msrq(srq)->wrid); - free(to_msrq(srq)); + mlx4_free_db(to_mctx(ibsrq->context), MLX4_DB_TYPE_RQ, srq->db); + mlx4_free_buf(&srq->buf); + free(srq->wrid); + free(srq); return 0; } @@ -413,7 +431,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv qp->sq.wqe_cnt = align_queue_size(attr->cap.max_send_wr + qp->sq_spare_wqes); qp->rq.wqe_cnt = align_queue_size(attr->cap.max_recv_wr); - if (attr->srq) + if (attr->srq || attr->qp_type == IBV_QPT_XRC) attr->cap.max_recv_wr = qp->rq.wqe_cnt = 0; else { if (attr->cap.max_recv_sge < 1) @@ -431,7 +449,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv pthread_spin_init(&qp->rq.lock, PTHREAD_PROCESS_PRIVATE)) goto err_free; - if (!attr->srq) { + if (!attr->srq && attr->qp_type != IBV_QPT_XRC) { qp->db = mlx4_alloc_db(to_mctx(pd->context), MLX4_DB_TYPE_RQ); if (!qp->db) goto err_free; @@ -440,7 +458,7 @@ struct ibv_qp *mlx4_create_qp(struct ibv } cmd.buf_addr = (uintptr_t) qp->buf.buf; - if (attr->srq) + if (attr->srq || attr->qp_type == IBV_QPT_XRC) cmd.db_addr = 0; else cmd.db_addr = (uintptr_t) qp->db; @@ -477,7 +495,7 @@ err_destroy: ibv_cmd_destroy_qp(&qp->ibv_qp); err_rq_db: - if (!attr->srq) + if (!attr->srq && attr->qp_type != IBV_QPT_XRC) mlx4_free_db(to_mctx(pd->context), MLX4_DB_TYPE_RQ, qp->db); err_free: @@ -536,7 +554,7 @@ int mlx4_modify_qp(struct ibv_qp *qp, st mlx4_cq_clean(to_mcq(qp->send_cq), qp->qp_num, NULL); mlx4_init_qp_indices(to_mqp(qp)); - if (!qp->srq) + if (!qp->srq && qp->qp_type != IBV_QPT_XRC) *to_mqp(qp)->db = 0; } @@ -595,7 +613,7 @@ int mlx4_destroy_qp(struct ibv_qp *ibqp) mlx4_unlock_cqs(ibqp); - if (!ibqp->srq) + if (!ibqp->srq && ibqp->qp_type != IBV_QPT_XRC) mlx4_free_db(to_mctx(ibqp->context), MLX4_DB_TYPE_RQ, qp->db); free(qp->sq.wrid); if (qp->rq.wqe_cnt) @@ -653,3 +671,103 @@ int mlx4_detach_mcast(struct ibv_qp *qp, { return ibv_cmd_detach_mcast(qp, gid, lid); } + +#ifdef HAVE_IBV_XRC_OPS +struct ibv_srq *mlx4_create_xrc_srq(struct ibv_pd *pd, + struct ibv_xrc_domain *xrc_domain, + struct ibv_cq *xrc_cq, + struct ibv_srq_init_attr *attr) +{ + struct mlx4_create_xrc_srq cmd; + struct mlx4_create_srq_resp resp; + struct mlx4_srq *srq; + int ret; + + /* Sanity check SRQ size before proceeding */ + if (attr->attr.max_wr > 1 << 16 || attr->attr.max_sge > 64) + return NULL; + + srq = malloc(sizeof *srq); + if (!srq) + return NULL; + + if (pthread_spin_init(&srq->lock, PTHREAD_PROCESS_PRIVATE)) + goto err; + + srq->max = align_queue_size(attr->attr.max_wr + 1); + srq->max_gs = attr->attr.max_sge; + srq->counter = 0; + + if (mlx4_alloc_srq_buf(pd, &attr->attr, srq)) + goto err; + + srq->db = mlx4_alloc_db(to_mctx(pd->context), MLX4_DB_TYPE_RQ); + if (!srq->db) + goto err_free; + + *srq->db = 0; + + cmd.buf_addr = (uintptr_t) srq->buf.buf; + cmd.db_addr = (uintptr_t) srq->db; + + ret = ibv_cmd_create_xrc_srq(pd, &srq->ibv_srq, attr, + xrc_domain->handle, + xrc_cq->handle, + &cmd.ibv_cmd, sizeof cmd, + &resp.ibv_resp, sizeof resp); + if (ret) + goto err_db; + + srq->ibv_srq.xrc_srq_num = srq->srqn = resp.srqn; + + ret = mlx4_store_xrc_srq(to_mctx(pd->context), srq->ibv_srq.xrc_srq_num, srq); + if (ret) + goto err_destroy; + + return &srq->ibv_srq; + +err_destroy: + ibv_cmd_destroy_srq(&srq->ibv_srq); + +err_db: + mlx4_free_db(to_mctx(pd->context), MLX4_DB_TYPE_RQ, srq->db); + +err_free: + free(srq->wrid); + mlx4_free_buf(&srq->buf); + +err: + free(srq); + + return NULL; +} + +struct ibv_xrc_domain *mlx4_open_xrc_domain(struct ibv_context *context, + int fd, int oflag) +{ + int ret; + struct mlx4_open_xrc_domain_resp resp; + struct mlx4_xrc_domain *xrcd; + + xrcd = malloc(sizeof *xrcd); + if (!xrcd) + return NULL; + + ret = ibv_cmd_open_xrc_domain(context, fd, oflag, &xrcd->ibv_xrcd, + &resp.ibv_resp, sizeof resp); + if (ret) { + free(xrcd); + return NULL; + } + + xrcd->xrcdn = resp.xrcdn; + return &xrcd->ibv_xrcd; +} + +int mlx4_close_xrc_domain(struct ibv_xrc_domain *d) +{ + ibv_cmd_close_xrc_domain(d); + free(d); + return 0; +} +#endif Index: libmlx4/src/wqe.h =================================================================== --- libmlx4.orig/src/wqe.h 2008-01-30 10:33:08.000000000 +0200 +++ libmlx4/src/wqe.h 2008-06-23 09:04:50.000000000 +0300 @@ -65,7 +65,7 @@ struct mlx4_wqe_ctrl_seg { * [1] SE (solicited event) * [0] FL (force loopback) */ - uint32_t srcrb_flags; + uint32_t xrcrb_flags; /* * imm is immediate data for send/RDMA write w/ immediate; * also invalidation key for send with invalidate; input From hrosenstock at xsigo.com Mon Jun 23 06:53:46 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 23 Jun 2008 06:53:46 -0700 Subject: [ofa-general] Re: [PATCHv2] ibsim: Support short RMPP packets In-Reply-To: <20080620175833.GK3158@sashak.voltaire.com> References: <1213886129.13056.168.camel@hrosenstock-ws.xsigo.com> <20080620175833.GK3158@sashak.voltaire.com> Message-ID: <1214229226.13056.230.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-20 at 20:58 +0300, Sasha Khapyorsky wrote: > On 07:35 Thu 19 Jun , Hal Rosenstock wrote: > > ibsim: Support for short RMPP packets (up thru 256 bytes total) > > > > Changes from v1: > > - Added length to simulation header (by reducing context size so as not > > to break ABI) and encoded length on sends and decoded and checked > > lengths on receive > > RMPP supports transactions up to 2^^32 256 byte packets in size. Not quite 256 and depends on RMPP "mode" in use. > So length must be more that 32-bit (or later we will need to break ABI). OK but not sure this is really supported. Also, 64 bits is overkill for length. > Also it should transferred in network byte order. > > > - Restored short/partial packet read check in umad2sim_read > > - In umad2sim_write, fixed setting of umad->length > > - Eliminated sim_req256 typedef > > Entire patch still be huge for what it tries to accomplish, I think it > is mostly due to "to malloc()" and data types conversions and it still > unclear for me why it was needed if packets are limited in size anyway. Because there were additional changes to get ready for long RMPP but they can wait. > For me it looks that something simpler like a patch below will do the > same. What do you think? An additional change is needed to umad2sim/umad2sim.c: @@ -425,7 +425,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void cnt = count - umad_size(); memcpy(umad_get_mad(umad), req.mad, cnt); - return cnt + umad_size(); + return umad->length; } -- Hal > Sasha > > > diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c > index 675d95b..6c5add9 100644 > --- a/ibsim/sim_mad.c > +++ b/ibsim/sim_mad.c > @@ -1306,7 +1306,7 @@ int send_trap(Port * port, int trapnum) > req.sqp = 0; > req.dqp = 0; > req.status = 0; > - req.context = 0; > + req.length = htonll(sizeof(req.mad)); > > // find SM client > cl = find_client(destport, 0, 1, 0); > diff --git a/include/ibsim.h b/include/ibsim.h > index 84568e6..ba1e3dd 100644 > --- a/include/ibsim.h > +++ b/include/ibsim.h > @@ -67,7 +67,7 @@ struct sim_request { > uint32_t dqp; > uint32_t sqp; > uint32_t status; > - uint64_t context; > + uint64_t length; > char mad[256]; > }; > > diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c > index 4cbf8da..f8fa50a 100644 > --- a/umad2sim/umad2sim.c > +++ b/umad2sim/umad2sim.c > @@ -411,7 +411,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void *buf, size_t count) > umad->status = ntohl(req.status); > umad->timeout_ms = 0; > umad->retries = 0; > - umad->length = umad_size() + sizeof(req.mad); > + umad->length = umad_size() + ntohll(req.length); > > umad->addr.qpn = req.sqp; > umad->addr.qkey = 0; // agent->qkey; > @@ -470,13 +470,14 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, > req.dqp = umad->addr.qpn; > req.sqp = htonl(dev->agents[umad->agent_id].qpn); > req.status = 0; > - req.context = 0; > > cnt = count - umad_size(); > if (cnt > sizeof(req.mad)) > cnt = sizeof(req.mad); > memcpy(req.mad, umad_get_mad(umad), cnt); > > + req.length = htonll(cnt); > + > cnt = write(dev->sim_client.fd_pktout, (void *)&req, sizeof(req)); > if (cnt < 0) { > ERROR("umad2sim_write: cannot write\n"); > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Mon Jun 23 07:01:30 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 23 Jun 2008 07:01:30 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <20080620133953.GE3158@sashak.voltaire.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> Message-ID: <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> Hi Sasha, On Fri, 2008-06-20 at 16:39 +0300, Sasha Khapyorsky wrote: > Hi Hal, > > On 08:18 Wed 18 Jun , Hal Rosenstock wrote: > > pensm/osm_port_profile: No need to inline osm_port_prof_is_ignored_port > > and osm_port_prof_set_ignored_port functions > > What is the reason for doing this (and the next inline removing patch)? Inlining is only a hint to the compiler and given the over(ab)use of inline in OpenSM (by my count almost 500 instances) I doubt this has the intended effect. Are the inlines really needed in these two cases ? In general, I think OpenSM needs a more careful look as to what really needs inlining. > > Also, some cosmetic formatting changes > > Would be nice to not mix in one patch. Already done on list. -- Hal > Sasha From pw at osc.edu Mon Jun 23 07:27:11 2008 From: pw at osc.edu (Pete Wyckoff) Date: Mon, 23 Jun 2008 10:27:11 -0400 Subject: [ofa-general] Re: [PATCH] IB/mthca: Clear ICM pages before handing to FW In-Reply-To: <20080623063537.GB1473@mtls03> References: <20080622155559.GA1473@mtls03> <4e6a6b3c0806221114w72f2afd3mbe256ba8f485602a@mail.gmail.com> <20080623063537.GB1473@mtls03> Message-ID: <20080623142711.GA31085@osc.edu> eli at dev.mellanox.co.il wrote on Mon, 23 Jun 2008 09:35 +0300: > Looking at the implementation of kmap_atomic() quoted bellow: [..] > static inline void *kmap_atomic(struct page *page, enum km_type idx) > { > pagefault_disable(); > return page_address(page); > } > > I can't see page_address doing anything actively to map the page. It > just searches for the page and returns it's mapping. But I'm probably > missing something. You're looking at the no-highmem version. Check the one in arch/x86/mm/highmem_32.c for a more interesting example. -- Pete From orenmeron at dev.mellanox.co.il Mon Jun 23 07:56:19 2008 From: orenmeron at dev.mellanox.co.il (Oren Meron) Date: Mon, 23 Jun 2008 17:56:19 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: References: Message-ID: <485FB993.2030500@dev.mellanox.co.il> Or Gerlitz wrote: > Eli, > > Using the new connectx FW (2.5), I see performance drop to almost > zero with ipoib datagram mode. The code that runs on these systems > is ofed 1.3 and not mainline kernel, details below. > > Running netperf With connected mode (64k MTU) I get about 950MB/s > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to > see about 650MB/s and above with FW 2.3 and datagram mode. Not that > it could explain the drop, but the NIC reports to the OS stateless > offload support - /sys/class/net/ib1/features is 0x11423 > > I have opened the ipoib and mlx4 debug prints, and I don't see anything > special other then the dmesg get quite filled with > > ib1: TX ring full, stopping kernel net queue > > any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which > is about five times what I see with ipoib. > > > Or. > > git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel > commit 564e9e9383272f4311fd87ff4e5447cfcebad73a > > # uname -a > Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 5.1 (Tikanga) > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > Hi Or, Trying to reproduce it i encountered the opposite result: datagram mode produced perfectly normal results (450-600 MBytes/s), connected mode produced extremely low results (20-30 MBytes/s). run parameters: dell intel servers, sles10, fw 2.5.0, MLNX_OFED_LINUX-1.3.1 will look into it with eli. oren From raked at stiftung-warentest.de Mon Jun 23 08:22:09 2008 From: raked at stiftung-warentest.de (Colleen Reeder) Date: Mon, 23 Jun 2008 16:22:09 +0100 Subject: [ofa-general] How much would you pay for a higher education? zy y cazkkr Message-ID: <01c8d54d$49522f20$e3b058d9@raked> A Gen ue uine Un scb ivers vtl ity Deg qoc ree in 4-6 we hxj eks! Have you ever thought that the only thing stopping you from a great j knf ob and better p nw ay was a few let plg ters behind your name? Well now you can get them! BA___B cy Sc___MA___MSc___MBA___PhD Within 4-6 weeks! No St qv udy Re ci qui uxb red! 100% Ver vjm ifia ljk ble! These are real, ge afw nuine degrees that include Bac nqg helors, Ma ulb sters, MBA and Do hsz ctorate Deg mm rees. They are fully ve lym rifia yr ble and ce dps rtif qxq ied transcripts are available. Just ca uml ll the number below. You will thank me later… 1-5 un 20-2 hfm 03-0 mxa 21 lro 3Leave a message 24 hours a day, 7 days a week including Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From MinuteProfits at lists.openfabrics.org Mon Jun 23 09:20:53 2008 From: MinuteProfits at lists.openfabrics.org (MinuteProfits at lists.openfabrics.org) Date: 23 Jun 2008 09:20:53 -0700 Subject: [ofa-general] Profits in 15 Minutes ?! MinuteProfits! Message-ID: <20080623092052.6B035D518CC0FA98@from.header.has.no.domain> Hi! I thank you for taking the time to read this! How would you like to make $50 - $250 every hour - starting minutes from NOW ? Many people ask me everyday - "Sebastian, there are so many 'make money on the net' guides out there - which one shall I buy - which one really brings in money - I finally want to start making money on the net". Don't believe any hyped promises - you won't be making thousands of dollars every day and you won't be a millionaire by next year but my System is a surefire method for everyone who wants to start earning online ". If you are ready to finally start making money online this is the right System for you. For Full Details please read the attached .html file Unsubscribe: on attached html file click on contact form -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Mon Jun 23 09:33:32 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 09:33:32 -0700 Subject: [ofa-general] [PATCH 1/11] libibverbs: XRC base implementation In-Reply-To: <200806231601.19743.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:01:19 +0300") References: <200806231601.19743.jackm@dev.mellanox.co.il> Message-ID: > +struct ibv_srq *__ibv_create_xrc_srq(struct ibv_pd *pd, > +default_symver(__ibv_create_xrc_srq, ibv_create_xrc_srq); Am I missing something? Why do we need this default_symver() stuff when we're adding new functions that didn't exist in the 1.0 ABI? - R. From rdreier at cisco.com Mon Jun 23 09:34:14 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 09:34:14 -0700 Subject: [ofa-general] [PATCH 1/11] libibverbs: XRC base implementation In-Reply-To: <200806231601.19743.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:01:19 +0300") References: <200806231601.19743.jackm@dev.mellanox.co.il> Message-ID: I don't see any man pages for the new functions? From rdreier at cisco.com Mon Jun 23 09:36:57 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 09:36:57 -0700 Subject: [ofa-general] Re: [PATCH 11/11] mlx4: XRC receive-only QPs In-Reply-To: <200806231602.02721.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:02:02 +0300") References: <200806231602.02721.jackm@dev.mellanox.co.il> Message-ID: running checkpatch on the kernel patches in this series shows total: 30 errors, 89 warnings, 3781 lines checked it's a waste of everyone's time to send patches with trivial formatting problems where there are automated tools to catch them. From rdreier at cisco.com Mon Jun 23 09:39:45 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 09:39:45 -0700 Subject: [ofa-general] Re: [PATCH 11/11] mlx4: XRC receive-only QPs In-Reply-To: <200806231602.02721.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:02:02 +0300") References: <200806231602.02721.jackm@dev.mellanox.co.il> Message-ID: > @@ -178,6 +178,7 @@ struct ib_cq *mlx4_ib_create_cq(struct i > if (!cq) > return ERR_PTR(-ENOMEM); > > + memset(cq, 0, sizeof *cq); > entries = roundup_pow_of_two(entries + 1); > cq->ibcq.cqe = entries - 1; > mutex_init(&cq->resize_mutex); Why is this change necessary? If it is necessary it would be more sensible to use kzalloc() to allocate the structure (and get rid of the initialization to NULL of some members too) -- that would actually shrink the code rather than bloating it. - R. From ctung at NetEffect.com Mon Jun 23 10:46:20 2008 From: ctung at NetEffect.com (Chien Tung) Date: Mon, 23 Jun 2008 12:46:20 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA: Add protocol statisticsattributes to the RDMA sysfs. In-Reply-To: <485DED14.9010305@voltaire.com> References: <20080617213856.8122.96180.stgit@dell3.ogc.int><5E701717F2B2ED4EA60F87C8AA57B7CC08235AE8@venom2> <5E701717F2B2ED4EA60F87C8AA57B7CC08235DA9@venom2> <485DED14.9010305@voltaire.com> Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC08235F4F@venom2> > Chien Tung wrote: > >> Actually it seems that you attach a struct ib_device to > each netdev > >> (eg a two-port NIC gets two RDMA devices with iw_nes) so Steve's > >> scheme should work fine for you too? (I just looked on a > system with > >> a two-port NIC and I do see nes0 and nes1 devices) > > > > After some discussion with our architect, Steve's scheme > will work for us by limiting to one netdev per physical port. > Does "one netdev per port" means something (what?) will not > be supported when VLANs are (eg nes0.4001, nes1.4002, etc) > would be used? > > Or. I was not referring to VLAN netdevs. We can have multiple netdevs backed by HW resources per physical port. Currently we are only creating one per PCI function/physical port. You can create VLAN interfaces on top of those netdevs up to OS limit. Chien From swise at opengridcomputing.com Mon Jun 23 11:01:41 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 23 Jun 2008 13:01:41 -0500 Subject: [ofa-general] [PATCH for-2.6.27] Roland, Message-ID: <20080623180141.5813.50187.stgit@dell3.ogc.int> I think this is ready to merge. I've tested inv-local and fastreg with krping, and we're doing NFSRDMA changes now. Any bugs found I'll post as delta patches. Sound like a plan? Changes since last posting: - bug fix when 2 fragment fastreg wraps in the wq - fixed fw version checking to properly set MEM_MGT_EXTENSIONS bit only for fw versions 6.1 and beyond. - sparse fixes Steve ---- RDMA/cxgb3: MEM_MGT_EXTENSIONS support - set IB_DEVICE_MEM_MGT_EXTENSIONS capability bit if fw supports it. - set max_fast_reg_page_list_len device attribute. - add iwch_alloc_fast_reg_mr function. - add iwch_alloc_fastreg_pbl - add iwch_free_fastreg_pbl - adjust the WQ depth for kernel mode work queues to account for fastreg possibly taking 2 WR slots. - add fastreg_mr work request support. - add local_inv work request support. - add send_with_inv and send_with_se_inv work request support. - removed useless duplicate enums/defines for TPT/MW/MR stuff. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 15 ++- drivers/infiniband/hw/cxgb3/cxio_hal.h | 1 drivers/infiniband/hw/cxgb3/cxio_wr.h | 87 ++++++++++++++---- drivers/infiniband/hw/cxgb3/iwch_cq.c | 14 ++- drivers/infiniband/hw/cxgb3/iwch_provider.c | 98 +++++++++++++++++++- drivers/infiniband/hw/cxgb3/iwch_provider.h | 8 -- drivers/infiniband/hw/cxgb3/iwch_qp.c | 132 ++++++++++++++++++--------- 7 files changed, 277 insertions(+), 78 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 3f441fc..d3d8576 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -145,7 +145,9 @@ static int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev_p, u32 qpid) } wqe = (struct t3_modify_qp_wr *) skb_put(skb, sizeof(*wqe)); memset(wqe, 0, sizeof(*wqe)); - build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 3, 0, qpid, 7); + build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, + T3_COMPLETION_FLAG | T3_NOTIFY_FLAG, 0, qpid, 7, + T3_SOPEOP); wqe->flags = cpu_to_be32(MODQP_WRITE_EC); sge_cmd = qpid << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); @@ -558,7 +560,7 @@ static int cxio_hal_init_ctrl_qp(struct cxio_rdev *rdev_p) wqe = (struct t3_modify_qp_wr *) skb_put(skb, sizeof(*wqe)); memset(wqe, 0, sizeof(*wqe)); build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 0, 0, - T3_CTL_QP_TID, 7); + T3_CTL_QP_TID, 7, T3_SOPEOP); wqe->flags = cpu_to_be32(MODQP_WRITE_EC); sge_cmd = (3ULL << 56) | FW_RI_SGEEC_START << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); @@ -674,7 +676,7 @@ static int cxio_hal_ctrl_qp_write_mem(struct cxio_rdev *rdev_p, u32 addr, build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_BP, flag, Q_GENBIT(rdev_p->ctrl_qp.wptr, T3_CTRL_QP_SIZE_LOG2), T3_CTRL_QP_ID, - wr_len); + wr_len, T3_SOPEOP); if (flag == T3_COMPLETION_FLAG) ring_doorbell(rdev_p->ctrl_qp.doorbell, T3_CTRL_QP_ID); len -= 96; @@ -816,6 +818,13 @@ int cxio_deallocate_window(struct cxio_rdev *rdev_p, u32 stag) 0, 0); } +int cxio_allocate_stag(struct cxio_rdev *rdev_p, u32 * stag, u32 pdid, u32 pbl_size, u32 pbl_addr) +{ + *stag = T3_STAG_UNSET; + return __cxio_tpt_op(rdev_p, 0, stag, 0, pdid, TPT_NON_SHARED_MR, + 0, 0, 0ULL, 0, 0, pbl_size, pbl_addr); +} + int cxio_rdma_init(struct cxio_rdev *rdev_p, struct t3_rdma_init_attr *attr) { struct t3_rdma_init_wr *wqe; diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 6e128f6..4bb5926 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -165,6 +165,7 @@ int cxio_reregister_phys_mem(struct cxio_rdev *rdev, u32 * stag, u32 pdid, int cxio_dereg_mem(struct cxio_rdev *rdev, u32 stag, u32 pbl_size, u32 pbl_addr); int cxio_allocate_window(struct cxio_rdev *rdev, u32 * stag, u32 pdid); +int cxio_allocate_stag(struct cxio_rdev *rdev, u32 * stag, u32 pdid, u32 pbl_size, u32 pbl_addr); int cxio_deallocate_window(struct cxio_rdev *rdev, u32 stag); int cxio_rdma_init(struct cxio_rdev *rdev, struct t3_rdma_init_attr *attr); void cxio_register_ev_cb(cxio_hal_ev_callback_func_t ev_cb); diff --git a/drivers/infiniband/hw/cxgb3/cxio_wr.h b/drivers/infiniband/hw/cxgb3/cxio_wr.h index f1a25a8..7ae110f 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_wr.h +++ b/drivers/infiniband/hw/cxgb3/cxio_wr.h @@ -72,7 +72,8 @@ enum t3_wr_opcode { T3_WR_BIND = FW_WROPCODE_RI_BIND_MW, T3_WR_RCV = FW_WROPCODE_RI_RECEIVE, T3_WR_INIT = FW_WROPCODE_RI_RDMA_INIT, - T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP + T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP, + T3_WR_FASTREG = FW_WROPCODE_RI_FASTREGISTER_MR } __attribute__ ((packed)); enum t3_rdma_opcode { @@ -89,7 +90,8 @@ enum t3_rdma_opcode { T3_FAST_REGISTER, T3_LOCAL_INV, T3_QP_MOD, - T3_BYPASS + T3_BYPASS, + T3_RDMA_READ_REQ_WITH_INV, } __attribute__ ((packed)); static inline enum t3_rdma_opcode wr2opcode(enum t3_wr_opcode wrop) @@ -103,6 +105,7 @@ static inline enum t3_rdma_opcode wr2opcode(enum t3_wr_opcode wrop) case T3_WR_BIND: return T3_BIND_MW; case T3_WR_INIT: return T3_RDMA_INIT; case T3_WR_QP_MOD: return T3_QP_MOD; + case T3_WR_FASTREG: return T3_FAST_REGISTER; default: break; } return -1; @@ -170,11 +173,54 @@ struct t3_send_wr { struct t3_sge sgl[T3_MAX_SGE]; /* 4+ */ }; +#define T3_MAX_FASTREG_DEPTH 24 +#define T3_MAX_FASTREG_FRAG 10 + +struct t3_fastreg_wr { + struct fw_riwrh wrh; /* 0 */ + union t3_wrid wrid; /* 1 */ + __be32 stag; /* 2 */ + __be32 len; + __be32 va_base_hi; /* 3 */ + __be32 va_base_lo_fbo; + __be32 page_type_perms; /* 4 */ + __be32 reserved1; + __be64 pbl_addrs[0]; /* 5+ */ +}; + +/* + * If a fastreg wr spans multiple wqes, then the 2nd fragment look like this. + */ +struct t3_pbl_frag { + struct fw_riwrh wrh; /* 0 */ + __be64 pbl_addrs[14]; /* 1..14 */ +}; + +#define S_FR_PAGE_COUNT 24 +#define M_FR_PAGE_COUNT 0xff +#define V_FR_PAGE_COUNT(x) ((x) << S_FR_PAGE_COUNT) +#define G_FR_PAGE_COUNT(x) ((((x) >> S_FR_PAGE_COUNT)) & M_FR_PAGE_COUNT) + +#define S_FR_PAGE_SIZE 16 +#define M_FR_PAGE_SIZE 0x1f +#define V_FR_PAGE_SIZE(x) ((x) << S_FR_PAGE_SIZE) +#define G_FR_PAGE_SIZE(x) ((((x) >> S_FR_PAGE_SIZE)) & M_FR_PAGE_SIZE) + +#define S_FR_TYPE 8 +#define M_FR_TYPE 0x1 +#define V_FR_TYPE(x) ((x) << S_FR_TYPE) +#define G_FR_TYPE(x) ((((x) >> S_FR_TYPE)) & M_FR_TYPE) + +#define S_FR_PERMS 0 +#define M_FR_PERMS 0xff +#define V_FR_PERMS(x) ((x) << S_FR_PERMS) +#define G_FR_PERMS(x) ((((x) >> S_FR_PERMS)) & M_FR_PERMS) + struct t3_local_inv_wr { struct fw_riwrh wrh; /* 0 */ union t3_wrid wrid; /* 1 */ __be32 stag; /* 2 */ - __be32 reserved3; + __be32 reserved; }; struct t3_rdma_write_wr { @@ -201,18 +247,6 @@ struct t3_rdma_read_wr { __be64 local_to; /* 5 */ }; -enum t3_addr_type { - T3_VA_BASED_TO = 0x0, - T3_ZERO_BASED_TO = 0x1 -} __attribute__ ((packed)); - -enum t3_mem_perms { - T3_MEM_ACCESS_LOCAL_READ = 0x1, - T3_MEM_ACCESS_LOCAL_WRITE = 0x2, - T3_MEM_ACCESS_REM_READ = 0x4, - T3_MEM_ACCESS_REM_WRITE = 0x8 -} __attribute__ ((packed)); - struct t3_bind_mw_wr { struct fw_riwrh wrh; /* 0 */ union t3_wrid wrid; /* 1 */ @@ -336,6 +370,11 @@ struct t3_genbit { __be64 genbit; }; +struct t3_wq_in_err { + u64 flit[13]; + u64 err; +}; + enum rdma_init_wr_flags { MPA_INITIATOR = (1<<0), PRIV_QP = (1<<1), @@ -346,13 +385,16 @@ union t3_wr { struct t3_rdma_write_wr write; struct t3_rdma_read_wr read; struct t3_receive_wr recv; + struct t3_fastreg_wr fastreg; + struct t3_pbl_frag pbl_frag; struct t3_local_inv_wr local_inv; struct t3_bind_mw_wr bind; struct t3_bypass_wr bypass; struct t3_rdma_init_wr init; struct t3_modify_qp_wr qp_mod; struct t3_genbit genbit; - u64 flit[16]; + struct t3_wq_in_err wq_in_err; + __be64 flit[16]; }; #define T3_SQ_CQE_FLIT 13 @@ -366,12 +408,18 @@ static inline enum t3_wr_opcode fw_riwrh_opcode(struct fw_riwrh *wqe) return G_FW_RIWR_OP(be32_to_cpu(wqe->op_seop_flags)); } +enum t3_wr_hdr_bits { + T3_EOP = 1, + T3_SOP = 2, + T3_SOPEOP = T3_EOP|T3_SOP, +}; + static inline void build_fw_riwrh(struct fw_riwrh *wqe, enum t3_wr_opcode op, enum t3_wr_flags flags, u8 genbit, u32 tid, - u8 len) + u8 len, u8 sopeop) { wqe->op_seop_flags = cpu_to_be32(V_FW_RIWR_OP(op) | - V_FW_RIWR_SOPEOP(M_FW_RIWR_SOPEOP) | + V_FW_RIWR_SOPEOP(sopeop) | V_FW_RIWR_FLAGS(flags)); wmb(); wqe->gen_tid_len = cpu_to_be32(V_FW_RIWR_GEN(genbit) | @@ -404,6 +452,7 @@ enum tpt_addr_type { }; enum tpt_mem_perm { + TPT_MW_BIND = 0x10, TPT_LOCAL_READ = 0x8, TPT_LOCAL_WRITE = 0x4, TPT_REMOTE_READ = 0x2, @@ -659,7 +708,7 @@ struct t3_cq { static inline void cxio_set_wq_in_error(struct t3_wq *wq) { - wq->queue->flit[13] = 1; + wq->queue->wq_in_err.err = 1; } static inline struct t3_cqe *cxio_next_hw_cqe(struct t3_cq *cq) diff --git a/drivers/infiniband/hw/cxgb3/iwch_cq.c b/drivers/infiniband/hw/cxgb3/iwch_cq.c index 4ee8ccd..3d7a8e8 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cq.c +++ b/drivers/infiniband/hw/cxgb3/iwch_cq.c @@ -81,6 +81,7 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, wc->wr_id = cookie; wc->qp = &qhp->ibqp; wc->vendor_err = CQE_STATUS(cqe); + wc->wc_flags = 0; PDBG("%s qpid 0x%x type %d opcode %d status 0x%x wrid hi 0x%x " "lo 0x%x cookie 0x%llx\n", __func__, @@ -94,6 +95,10 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, else wc->byte_len = 0; wc->opcode = IB_WC_RECV; + if (CQE_OPCODE(cqe) == T3_SEND_WITH_INV || CQE_OPCODE(cqe) == T3_SEND_WITH_SE_INV) { + wc->ex.invalidate_rkey = CQE_WRID_STAG(cqe); + wc->wc_flags |= IB_WC_WITH_INVALIDATE; + } } else { switch (CQE_OPCODE(cqe)) { case T3_RDMA_WRITE: @@ -105,17 +110,20 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, break; case T3_SEND: case T3_SEND_WITH_SE: + case T3_SEND_WITH_INV: + case T3_SEND_WITH_SE_INV: wc->opcode = IB_WC_SEND; break; case T3_BIND_MW: wc->opcode = IB_WC_BIND_MW; break; - /* these aren't supported yet */ - case T3_SEND_WITH_INV: - case T3_SEND_WITH_SE_INV: case T3_LOCAL_INV: + wc->opcode = IB_WC_LOCAL_INV; + break; case T3_FAST_REGISTER: + wc->opcode = IB_WC_FAST_REG_MR; + break; default: printk(KERN_ERR MOD "Unexpected opcode %d " "in the CQE received for QPID=0x%0x\n", diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 8934178..089a7df 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -768,6 +768,67 @@ static int iwch_dealloc_mw(struct ib_mw *mw) return 0; } +static struct ib_mr *iwch_alloc_fast_reg_mr(struct ib_pd *pd, int pbl_depth) +{ + struct iwch_dev *rhp; + struct iwch_pd *php; + struct iwch_mr *mhp; + u32 mmid; + u32 stag = 0; + int ret; + + php = to_iwch_pd(pd); + rhp = php->rhp; + mhp = kzalloc(sizeof(*mhp), GFP_KERNEL); + if (!mhp) + return ERR_PTR(-ENOMEM); + + mhp->rhp = rhp; + ret = iwch_alloc_pbl(mhp, pbl_depth); + if (ret) { + kfree(mhp); + return ERR_PTR(ret); + } + mhp->attr.pbl_size = pbl_depth; + ret = cxio_allocate_stag(&rhp->rdev, &stag, php->pdid, mhp->attr.pbl_size, mhp->attr.pbl_addr); + if (ret) { + iwch_free_pbl(mhp); + kfree(mhp); + return ERR_PTR(ret); + } + mhp->attr.pdid = php->pdid; + mhp->attr.type = TPT_NON_SHARED_MR; + mhp->attr.stag = stag; + mhp->attr.state = 1; + mmid = (stag) >> 8; + mhp->ibmr.rkey = mhp->ibmr.lkey = stag; + insert_handle(rhp, &rhp->mmidr, mhp, mmid); + PDBG("%s mmid 0x%x mhp %p stag 0x%x\n", __func__, mmid, mhp, stag); + return &(mhp->ibmr); +} + +static struct ib_fast_reg_page_list *iwch_alloc_fastreg_pbl( + struct ib_device *device, + int page_list_len) +{ + struct ib_fast_reg_page_list *page_list; + + page_list = kmalloc(sizeof *page_list + page_list_len * sizeof(u64), + GFP_KERNEL); + if (!page_list) + return ERR_PTR(-ENOMEM); + + page_list->page_list = (u64 *)(page_list + 1); + page_list->max_page_list_len = page_list_len; + + return page_list; +} + +static void iwch_free_fastreg_pbl(struct ib_fast_reg_page_list *page_list) +{ + kfree(page_list); +} + static int iwch_destroy_qp(struct ib_qp *ib_qp) { struct iwch_dev *rhp; @@ -843,6 +904,15 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, */ sqsize = roundup_pow_of_two(attrs->cap.max_send_wr); wqsize = roundup_pow_of_two(rqsize + sqsize); + + /* + * Kernel users need more wq space for fastreg WRs which can take + * 2 WR fragments. + */ + ucontext = pd->uobject ? to_iwch_ucontext(pd->uobject->context) : NULL; + if (!ucontext && wqsize < (rqsize + (2 * sqsize))) + wqsize = roundup_pow_of_two(rqsize + + roundup_pow_of_two(attrs->cap.max_send_wr * 2)); PDBG("%s wqsize %d sqsize %d rqsize %d\n", __func__, wqsize, sqsize, rqsize); qhp = kzalloc(sizeof(*qhp), GFP_KERNEL); @@ -851,7 +921,6 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, qhp->wq.size_log2 = ilog2(wqsize); qhp->wq.rq_size_log2 = ilog2(rqsize); qhp->wq.sq_size_log2 = ilog2(sqsize); - ucontext = pd->uobject ? to_iwch_ucontext(pd->uobject->context) : NULL; if (cxio_create_qp(&rhp->rdev, !udata, &qhp->wq, ucontext ? &ucontext->uctx : &rhp->rdev.uctx)) { kfree(qhp); @@ -1048,6 +1117,7 @@ static int iwch_query_device(struct ib_device *ibdev, props->max_mr = dev->attr.max_mem_regs; props->max_pd = dev->attr.max_pds; props->local_ca_ack_delay = 0; + props->max_fast_reg_page_list_len = T3_MAX_FASTREG_DEPTH; return 0; } @@ -1088,6 +1158,23 @@ static ssize_t show_rev(struct device *dev, struct device_attribute *attr, return sprintf(buf, "%d\n", iwch_dev->rdev.t3cdev_p->type); } +static int fw_supports_fastreg(struct iwch_dev *iwch_dev) +{ + struct ethtool_drvinfo info; + struct net_device *lldev = iwch_dev->rdev.t3cdev_p->lldev; + char *cp, *next; + unsigned fw_maj, fw_min; + + lldev->ethtool_ops->get_drvinfo(lldev, &info); + next = info.fw_version+1; + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_maj); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_min); + printk("%s maj %u min %u\n", __func__, fw_maj, fw_min); + return fw_maj > 6 || (fw_maj == 6 && fw_min > 0); +} + static ssize_t show_fw_ver(struct device *dev, struct device_attribute *attr, char *buf) { struct iwch_dev *iwch_dev = container_of(dev, struct iwch_dev, @@ -1145,8 +1232,10 @@ int iwch_register_device(struct iwch_dev *dev) memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid)); memcpy(&dev->ibdev.node_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); dev->ibdev.owner = THIS_MODULE; - dev->device_cap_flags = - (IB_DEVICE_ZERO_STAG | IB_DEVICE_MEM_WINDOW); + dev->device_cap_flags = IB_DEVICE_ZERO_STAG | + IB_DEVICE_MEM_WINDOW; + if (fw_supports_fastreg(dev)) + dev->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; dev->ibdev.uverbs_cmd_mask = (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) | @@ -1198,6 +1287,9 @@ int iwch_register_device(struct iwch_dev *dev) dev->ibdev.alloc_mw = iwch_alloc_mw; dev->ibdev.bind_mw = iwch_bind_mw; dev->ibdev.dealloc_mw = iwch_dealloc_mw; + dev->ibdev.alloc_fast_reg_mr = iwch_alloc_fast_reg_mr; + dev->ibdev.alloc_fast_reg_page_list = iwch_alloc_fastreg_pbl; + dev->ibdev.free_fast_reg_page_list = iwch_free_fastreg_pbl; dev->ibdev.attach_mcast = iwch_multicast_attach; dev->ibdev.detach_mcast = iwch_multicast_detach; diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.h b/drivers/infiniband/hw/cxgb3/iwch_provider.h index 836163f..f5ceca0 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.h +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.h @@ -296,14 +296,6 @@ static inline u32 iwch_ib_to_tpt_access(int acc) TPT_LOCAL_READ; } -static inline u32 iwch_ib_to_mwbind_access(int acc) -{ - return (acc & IB_ACCESS_REMOTE_WRITE ? T3_MEM_ACCESS_REM_WRITE : 0) | - (acc & IB_ACCESS_REMOTE_READ ? T3_MEM_ACCESS_REM_READ : 0) | - (acc & IB_ACCESS_LOCAL_WRITE ? T3_MEM_ACCESS_LOCAL_WRITE : 0) | - T3_MEM_ACCESS_LOCAL_READ; -} - enum iwch_mmid_state { IWCH_STAG_STATE_VALID, IWCH_STAG_STATE_INVALID diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 9926137..02b3909 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -44,54 +44,39 @@ static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, switch (wr->opcode) { case IB_WR_SEND: - case IB_WR_SEND_WITH_IMM: if (wr->send_flags & IB_SEND_SOLICITED) wqe->send.rdmaop = T3_SEND_WITH_SE; else wqe->send.rdmaop = T3_SEND; wqe->send.rem_stag = 0; break; -#if 0 /* Not currently supported */ - case TYPE_SEND_INVALIDATE: - case TYPE_SEND_INVALIDATE_IMMEDIATE: - wqe->send.rdmaop = T3_SEND_WITH_INV; - wqe->send.rem_stag = cpu_to_be32(wr->wr.rdma.rkey); - break; - case TYPE_SEND_SE_INVALIDATE: - wqe->send.rdmaop = T3_SEND_WITH_SE_INV; - wqe->send.rem_stag = cpu_to_be32(wr->wr.rdma.rkey); + case IB_WR_SEND_WITH_INV: + if (wr->send_flags & IB_SEND_SOLICITED) + wqe->send.rdmaop = T3_SEND_WITH_SE_INV; + else + wqe->send.rdmaop = T3_SEND_WITH_INV; + wqe->send.rem_stag = cpu_to_be32(wr->ex.invalidate_rkey); break; -#endif default: - break; + return -EINVAL; } if (wr->num_sge > T3_MAX_SGE) return -EINVAL; wqe->send.reserved[0] = 0; wqe->send.reserved[1] = 0; wqe->send.reserved[2] = 0; - if (wr->opcode == IB_WR_SEND_WITH_IMM) { - plen = 4; - wqe->send.sgl[0].stag = wr->ex.imm_data; - wqe->send.sgl[0].len = __constant_cpu_to_be32(0); - wqe->send.num_sgle = __constant_cpu_to_be32(0); - *flit_cnt = 5; - } else { - plen = 0; - for (i = 0; i < wr->num_sge; i++) { - if ((plen + wr->sg_list[i].length) < plen) { - return -EMSGSIZE; - } - plen += wr->sg_list[i].length; - wqe->send.sgl[i].stag = - cpu_to_be32(wr->sg_list[i].lkey); - wqe->send.sgl[i].len = - cpu_to_be32(wr->sg_list[i].length); - wqe->send.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); + plen = 0; + for (i = 0; i < wr->num_sge; i++) { + if ((plen + wr->sg_list[i].length) < plen) { + return -EMSGSIZE; } - wqe->send.num_sgle = cpu_to_be32(wr->num_sge); - *flit_cnt = 4 + ((wr->num_sge) << 1); + plen += wr->sg_list[i].length; + wqe->send.sgl[i].stag = cpu_to_be32(wr->sg_list[i].lkey); + wqe->send.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); + wqe->send.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); } + wqe->send.num_sgle = cpu_to_be32(wr->num_sge); + *flit_cnt = 4 + ((wr->num_sge) << 1); wqe->send.plen = cpu_to_be32(plen); return 0; } @@ -155,6 +140,57 @@ static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } +static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, + u8 *flit_cnt, int *wr_cnt, struct t3_wq *wq) +{ + int i; + __be64 *p; + + if (wr->wr.fast_reg.page_list_len > T3_MAX_FASTREG_DEPTH) + return -EINVAL; + *wr_cnt = 1; + wqe->fastreg.stag = cpu_to_be32(wr->wr.fast_reg.rkey); + wqe->fastreg.len = cpu_to_be32(wr->wr.fast_reg.length); + wqe->fastreg.va_base_hi = cpu_to_be32(wr->wr.fast_reg.iova_start>>32); + wqe->fastreg.va_base_lo_fbo = + cpu_to_be32(wr->wr.fast_reg.iova_start&0xffffffff); + wqe->fastreg.page_type_perms = cpu_to_be32( + V_FR_PAGE_COUNT(wr->wr.fast_reg.page_list_len) | + V_FR_PAGE_SIZE(wr->wr.fast_reg.page_shift-12) | + V_FR_TYPE(TPT_VATO) | + V_FR_PERMS(iwch_ib_to_tpt_access(wr->wr.fast_reg.access_flags))); + p = &wqe->fastreg.pbl_addrs[0]; + for (i=0; iwr.fast_reg.page_list_len; i++, p++) { + + /* If we need a 2nd WR, then set it up */ + if (i == T3_MAX_FASTREG_FRAG) { + *wr_cnt = 2; + wqe = (union t3_wr *)(wq->queue + + Q_PTR2IDX((wq->wptr+1), wq->size_log2)); + build_fw_riwrh((void *)wqe, T3_WR_FASTREG, 0, + Q_GENBIT(wq->wptr + 1, wq->size_log2), + 0, 1 + wr->wr.fast_reg.page_list_len - T3_MAX_FASTREG_FRAG, + T3_EOP); + + p = &wqe->pbl_frag.pbl_addrs[0]; + } + *p = cpu_to_be64((u64)wr->wr.fast_reg.page_list->page_list[i]); + } + *flit_cnt = 5 + wr->wr.fast_reg.page_list_len; + if (*flit_cnt > 15) + *flit_cnt = 15; + return 0; +} + +static int iwch_build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, + u8 *flit_cnt) +{ + wqe->local_inv.stag = cpu_to_be32(wr->ex.invalidate_rkey); + wqe->local_inv.reserved = 0; + *flit_cnt = sizeof(struct t3_local_inv_wr) >> 3; + return 0; +} + /* * TBD: this is going to be moved to firmware. Missing pdid/qpid check for now. */ @@ -238,6 +274,7 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, u32 num_wrs; unsigned long flag; struct t3_swsq *sqp; + int wr_cnt = 1; qhp = to_iwch_qp(ibqp); spin_lock_irqsave(&qhp->lock, flag); @@ -262,15 +299,15 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, t3_wr_flags = 0; if (wr->send_flags & IB_SEND_SOLICITED) t3_wr_flags |= T3_SOLICITED_EVENT_FLAG; - if (wr->send_flags & IB_SEND_FENCE) - t3_wr_flags |= T3_READ_FENCE_FLAG; if (wr->send_flags & IB_SEND_SIGNALED) t3_wr_flags |= T3_COMPLETION_FLAG; sqp = qhp->wq.sq + Q_PTR2IDX(qhp->wq.sq_wptr, qhp->wq.sq_size_log2); switch (wr->opcode) { case IB_WR_SEND: - case IB_WR_SEND_WITH_IMM: + case IB_WR_SEND_WITH_INV: + if (wr->send_flags & IB_SEND_FENCE) + t3_wr_flags |= T3_READ_FENCE_FLAG; t3_wr_opcode = T3_WR_SEND; err = iwch_build_rdma_send(wqe, wr, &t3_wr_flit_cnt); break; @@ -289,6 +326,17 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, if (!qhp->wq.oldest_read) qhp->wq.oldest_read = sqp; break; + case IB_WR_FAST_REG_MR: + t3_wr_opcode = T3_WR_FASTREG; + err = iwch_build_fastreg(wqe, wr, &t3_wr_flit_cnt, + &wr_cnt, &qhp->wq); + break; + case IB_WR_LOCAL_INV: + if (wr->send_flags & IB_SEND_FENCE) + t3_wr_flags |= T3_LOCAL_FENCE_FLAG; + t3_wr_opcode = T3_WR_INV_STAG; + err = iwch_build_inv_stag(wqe, wr, &t3_wr_flit_cnt); + break; default: PDBG("%s post of type=%d TBD!\n", __func__, wr->opcode); @@ -307,14 +355,15 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, build_fw_riwrh((void *) wqe, t3_wr_opcode, t3_wr_flags, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), - 0, t3_wr_flit_cnt); + 0, t3_wr_flit_cnt, + (wr_cnt == 1) ? T3_SOPEOP : T3_SOP); PDBG("%s cookie 0x%llx wq idx 0x%x swsq idx %ld opcode %d\n", __func__, (unsigned long long) wr->wr_id, idx, Q_PTR2IDX(qhp->wq.sq_wptr, qhp->wq.sq_size_log2), sqp->opcode); wr = wr->next; num_wrs--; - ++(qhp->wq.wptr); + qhp->wq.wptr += wr_cnt; ++(qhp->wq.sq_wptr); } spin_unlock_irqrestore(&qhp->lock, flag); @@ -359,7 +408,7 @@ int iwch_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, wr->wr_id; build_fw_riwrh((void *) wqe, T3_WR_RCV, T3_COMPLETION_FLAG, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), - 0, sizeof(struct t3_receive_wr) >> 3); + 0, sizeof(struct t3_receive_wr) >> 3, T3_SOPEOP); PDBG("%s cookie 0x%llx idx 0x%x rq_wptr 0x%x rw_rptr 0x%x " "wqe %p \n", __func__, (unsigned long long) wr->wr_id, idx, qhp->wq.rq_wptr, qhp->wq.rq_rptr, wqe); @@ -419,10 +468,10 @@ int iwch_bind_mw(struct ib_qp *qp, sgl.lkey = mw_bind->mr->lkey; sgl.length = mw_bind->length; wqe->bind.reserved = 0; - wqe->bind.type = T3_VA_BASED_TO; + wqe->bind.type = TPT_VATO; /* TBD: check perms */ - wqe->bind.perms = iwch_ib_to_mwbind_access(mw_bind->mw_access_flags); + wqe->bind.perms = iwch_ib_to_tpt_access(mw_bind->mw_access_flags); wqe->bind.mr_stag = cpu_to_be32(mw_bind->mr->lkey); wqe->bind.mw_stag = cpu_to_be32(mw->rkey); wqe->bind.mw_len = cpu_to_be32(mw_bind->length); @@ -441,10 +490,9 @@ int iwch_bind_mw(struct ib_qp *qp, sqp->signaled = (mw_bind->send_flags & IB_SEND_SIGNALED); wqe->bind.mr_pbl_addr = cpu_to_be32(pbl_addr); wqe->bind.mr_pagesz = page_size; - wqe->flit[T3_SQ_COOKIE_FLIT] = mw_bind->wr_id; build_fw_riwrh((void *)wqe, T3_WR_BIND, t3_wr_flags, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), 0, - sizeof(struct t3_bind_mw_wr) >> 3); + sizeof(struct t3_bind_mw_wr) >> 3, T3_SOPEOP); ++(qhp->wq.wptr); ++(qhp->wq.sq_wptr); spin_unlock_irqrestore(&qhp->lock, flag); From swise at opengridcomputing.com Mon Jun 23 11:03:53 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 23 Jun 2008 13:03:53 -0500 Subject: [ofa-general] [PATCH REPOST for-2.6.27] RDMA/cxgb3: MEM_MGT_EXTENSIONS support Message-ID: <20080623180353.5877.94150.stgit@dell3.ogc.int> Roland, I think this is ready to merge. I've tested inv-local and fastreg with krping, and we're doing NFSRDMA changes now. Any bugs found I'll post as delta patches. Sound like a plan? Changes since last posting: - bug fix when 2 fragment fastreg wraps in the wq - fixed fw version checking to properly set MEM_MGT_EXTENSIONS bit only for fw versions 6.1 and beyond. - sparse fixes Steve ---- RDMA/cxgb3: MEM_MGT_EXTENSIONS support - set IB_DEVICE_MEM_MGT_EXTENSIONS capability bit if fw supports it. - set max_fast_reg_page_list_len device attribute. - add iwch_alloc_fast_reg_mr function. - add iwch_alloc_fastreg_pbl - add iwch_free_fastreg_pbl - adjust the WQ depth for kernel mode work queues to account for fastreg possibly taking 2 WR slots. - add fastreg_mr work request support. - add local_inv work request support. - add send_with_inv and send_with_se_inv work request support. - removed useless duplicate enums/defines for TPT/MW/MR stuff. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 15 ++- drivers/infiniband/hw/cxgb3/cxio_hal.h | 1 drivers/infiniband/hw/cxgb3/cxio_wr.h | 87 ++++++++++++++---- drivers/infiniband/hw/cxgb3/iwch_cq.c | 14 ++- drivers/infiniband/hw/cxgb3/iwch_provider.c | 98 +++++++++++++++++++- drivers/infiniband/hw/cxgb3/iwch_provider.h | 8 -- drivers/infiniband/hw/cxgb3/iwch_qp.c | 132 ++++++++++++++++++--------- 7 files changed, 277 insertions(+), 78 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 3f441fc..d3d8576 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -145,7 +145,9 @@ static int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev_p, u32 qpid) } wqe = (struct t3_modify_qp_wr *) skb_put(skb, sizeof(*wqe)); memset(wqe, 0, sizeof(*wqe)); - build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 3, 0, qpid, 7); + build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, + T3_COMPLETION_FLAG | T3_NOTIFY_FLAG, 0, qpid, 7, + T3_SOPEOP); wqe->flags = cpu_to_be32(MODQP_WRITE_EC); sge_cmd = qpid << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); @@ -558,7 +560,7 @@ static int cxio_hal_init_ctrl_qp(struct cxio_rdev *rdev_p) wqe = (struct t3_modify_qp_wr *) skb_put(skb, sizeof(*wqe)); memset(wqe, 0, sizeof(*wqe)); build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_QP_MOD, 0, 0, - T3_CTL_QP_TID, 7); + T3_CTL_QP_TID, 7, T3_SOPEOP); wqe->flags = cpu_to_be32(MODQP_WRITE_EC); sge_cmd = (3ULL << 56) | FW_RI_SGEEC_START << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); @@ -674,7 +676,7 @@ static int cxio_hal_ctrl_qp_write_mem(struct cxio_rdev *rdev_p, u32 addr, build_fw_riwrh((struct fw_riwrh *) wqe, T3_WR_BP, flag, Q_GENBIT(rdev_p->ctrl_qp.wptr, T3_CTRL_QP_SIZE_LOG2), T3_CTRL_QP_ID, - wr_len); + wr_len, T3_SOPEOP); if (flag == T3_COMPLETION_FLAG) ring_doorbell(rdev_p->ctrl_qp.doorbell, T3_CTRL_QP_ID); len -= 96; @@ -816,6 +818,13 @@ int cxio_deallocate_window(struct cxio_rdev *rdev_p, u32 stag) 0, 0); } +int cxio_allocate_stag(struct cxio_rdev *rdev_p, u32 * stag, u32 pdid, u32 pbl_size, u32 pbl_addr) +{ + *stag = T3_STAG_UNSET; + return __cxio_tpt_op(rdev_p, 0, stag, 0, pdid, TPT_NON_SHARED_MR, + 0, 0, 0ULL, 0, 0, pbl_size, pbl_addr); +} + int cxio_rdma_init(struct cxio_rdev *rdev_p, struct t3_rdma_init_attr *attr) { struct t3_rdma_init_wr *wqe; diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 6e128f6..4bb5926 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -165,6 +165,7 @@ int cxio_reregister_phys_mem(struct cxio_rdev *rdev, u32 * stag, u32 pdid, int cxio_dereg_mem(struct cxio_rdev *rdev, u32 stag, u32 pbl_size, u32 pbl_addr); int cxio_allocate_window(struct cxio_rdev *rdev, u32 * stag, u32 pdid); +int cxio_allocate_stag(struct cxio_rdev *rdev, u32 * stag, u32 pdid, u32 pbl_size, u32 pbl_addr); int cxio_deallocate_window(struct cxio_rdev *rdev, u32 stag); int cxio_rdma_init(struct cxio_rdev *rdev, struct t3_rdma_init_attr *attr); void cxio_register_ev_cb(cxio_hal_ev_callback_func_t ev_cb); diff --git a/drivers/infiniband/hw/cxgb3/cxio_wr.h b/drivers/infiniband/hw/cxgb3/cxio_wr.h index f1a25a8..7ae110f 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_wr.h +++ b/drivers/infiniband/hw/cxgb3/cxio_wr.h @@ -72,7 +72,8 @@ enum t3_wr_opcode { T3_WR_BIND = FW_WROPCODE_RI_BIND_MW, T3_WR_RCV = FW_WROPCODE_RI_RECEIVE, T3_WR_INIT = FW_WROPCODE_RI_RDMA_INIT, - T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP + T3_WR_QP_MOD = FW_WROPCODE_RI_MODIFY_QP, + T3_WR_FASTREG = FW_WROPCODE_RI_FASTREGISTER_MR } __attribute__ ((packed)); enum t3_rdma_opcode { @@ -89,7 +90,8 @@ enum t3_rdma_opcode { T3_FAST_REGISTER, T3_LOCAL_INV, T3_QP_MOD, - T3_BYPASS + T3_BYPASS, + T3_RDMA_READ_REQ_WITH_INV, } __attribute__ ((packed)); static inline enum t3_rdma_opcode wr2opcode(enum t3_wr_opcode wrop) @@ -103,6 +105,7 @@ static inline enum t3_rdma_opcode wr2opcode(enum t3_wr_opcode wrop) case T3_WR_BIND: return T3_BIND_MW; case T3_WR_INIT: return T3_RDMA_INIT; case T3_WR_QP_MOD: return T3_QP_MOD; + case T3_WR_FASTREG: return T3_FAST_REGISTER; default: break; } return -1; @@ -170,11 +173,54 @@ struct t3_send_wr { struct t3_sge sgl[T3_MAX_SGE]; /* 4+ */ }; +#define T3_MAX_FASTREG_DEPTH 24 +#define T3_MAX_FASTREG_FRAG 10 + +struct t3_fastreg_wr { + struct fw_riwrh wrh; /* 0 */ + union t3_wrid wrid; /* 1 */ + __be32 stag; /* 2 */ + __be32 len; + __be32 va_base_hi; /* 3 */ + __be32 va_base_lo_fbo; + __be32 page_type_perms; /* 4 */ + __be32 reserved1; + __be64 pbl_addrs[0]; /* 5+ */ +}; + +/* + * If a fastreg wr spans multiple wqes, then the 2nd fragment look like this. + */ +struct t3_pbl_frag { + struct fw_riwrh wrh; /* 0 */ + __be64 pbl_addrs[14]; /* 1..14 */ +}; + +#define S_FR_PAGE_COUNT 24 +#define M_FR_PAGE_COUNT 0xff +#define V_FR_PAGE_COUNT(x) ((x) << S_FR_PAGE_COUNT) +#define G_FR_PAGE_COUNT(x) ((((x) >> S_FR_PAGE_COUNT)) & M_FR_PAGE_COUNT) + +#define S_FR_PAGE_SIZE 16 +#define M_FR_PAGE_SIZE 0x1f +#define V_FR_PAGE_SIZE(x) ((x) << S_FR_PAGE_SIZE) +#define G_FR_PAGE_SIZE(x) ((((x) >> S_FR_PAGE_SIZE)) & M_FR_PAGE_SIZE) + +#define S_FR_TYPE 8 +#define M_FR_TYPE 0x1 +#define V_FR_TYPE(x) ((x) << S_FR_TYPE) +#define G_FR_TYPE(x) ((((x) >> S_FR_TYPE)) & M_FR_TYPE) + +#define S_FR_PERMS 0 +#define M_FR_PERMS 0xff +#define V_FR_PERMS(x) ((x) << S_FR_PERMS) +#define G_FR_PERMS(x) ((((x) >> S_FR_PERMS)) & M_FR_PERMS) + struct t3_local_inv_wr { struct fw_riwrh wrh; /* 0 */ union t3_wrid wrid; /* 1 */ __be32 stag; /* 2 */ - __be32 reserved3; + __be32 reserved; }; struct t3_rdma_write_wr { @@ -201,18 +247,6 @@ struct t3_rdma_read_wr { __be64 local_to; /* 5 */ }; -enum t3_addr_type { - T3_VA_BASED_TO = 0x0, - T3_ZERO_BASED_TO = 0x1 -} __attribute__ ((packed)); - -enum t3_mem_perms { - T3_MEM_ACCESS_LOCAL_READ = 0x1, - T3_MEM_ACCESS_LOCAL_WRITE = 0x2, - T3_MEM_ACCESS_REM_READ = 0x4, - T3_MEM_ACCESS_REM_WRITE = 0x8 -} __attribute__ ((packed)); - struct t3_bind_mw_wr { struct fw_riwrh wrh; /* 0 */ union t3_wrid wrid; /* 1 */ @@ -336,6 +370,11 @@ struct t3_genbit { __be64 genbit; }; +struct t3_wq_in_err { + u64 flit[13]; + u64 err; +}; + enum rdma_init_wr_flags { MPA_INITIATOR = (1<<0), PRIV_QP = (1<<1), @@ -346,13 +385,16 @@ union t3_wr { struct t3_rdma_write_wr write; struct t3_rdma_read_wr read; struct t3_receive_wr recv; + struct t3_fastreg_wr fastreg; + struct t3_pbl_frag pbl_frag; struct t3_local_inv_wr local_inv; struct t3_bind_mw_wr bind; struct t3_bypass_wr bypass; struct t3_rdma_init_wr init; struct t3_modify_qp_wr qp_mod; struct t3_genbit genbit; - u64 flit[16]; + struct t3_wq_in_err wq_in_err; + __be64 flit[16]; }; #define T3_SQ_CQE_FLIT 13 @@ -366,12 +408,18 @@ static inline enum t3_wr_opcode fw_riwrh_opcode(struct fw_riwrh *wqe) return G_FW_RIWR_OP(be32_to_cpu(wqe->op_seop_flags)); } +enum t3_wr_hdr_bits { + T3_EOP = 1, + T3_SOP = 2, + T3_SOPEOP = T3_EOP|T3_SOP, +}; + static inline void build_fw_riwrh(struct fw_riwrh *wqe, enum t3_wr_opcode op, enum t3_wr_flags flags, u8 genbit, u32 tid, - u8 len) + u8 len, u8 sopeop) { wqe->op_seop_flags = cpu_to_be32(V_FW_RIWR_OP(op) | - V_FW_RIWR_SOPEOP(M_FW_RIWR_SOPEOP) | + V_FW_RIWR_SOPEOP(sopeop) | V_FW_RIWR_FLAGS(flags)); wmb(); wqe->gen_tid_len = cpu_to_be32(V_FW_RIWR_GEN(genbit) | @@ -404,6 +452,7 @@ enum tpt_addr_type { }; enum tpt_mem_perm { + TPT_MW_BIND = 0x10, TPT_LOCAL_READ = 0x8, TPT_LOCAL_WRITE = 0x4, TPT_REMOTE_READ = 0x2, @@ -659,7 +708,7 @@ struct t3_cq { static inline void cxio_set_wq_in_error(struct t3_wq *wq) { - wq->queue->flit[13] = 1; + wq->queue->wq_in_err.err = 1; } static inline struct t3_cqe *cxio_next_hw_cqe(struct t3_cq *cq) diff --git a/drivers/infiniband/hw/cxgb3/iwch_cq.c b/drivers/infiniband/hw/cxgb3/iwch_cq.c index 4ee8ccd..3d7a8e8 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cq.c +++ b/drivers/infiniband/hw/cxgb3/iwch_cq.c @@ -81,6 +81,7 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, wc->wr_id = cookie; wc->qp = &qhp->ibqp; wc->vendor_err = CQE_STATUS(cqe); + wc->wc_flags = 0; PDBG("%s qpid 0x%x type %d opcode %d status 0x%x wrid hi 0x%x " "lo 0x%x cookie 0x%llx\n", __func__, @@ -94,6 +95,10 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, else wc->byte_len = 0; wc->opcode = IB_WC_RECV; + if (CQE_OPCODE(cqe) == T3_SEND_WITH_INV || CQE_OPCODE(cqe) == T3_SEND_WITH_SE_INV) { + wc->ex.invalidate_rkey = CQE_WRID_STAG(cqe); + wc->wc_flags |= IB_WC_WITH_INVALIDATE; + } } else { switch (CQE_OPCODE(cqe)) { case T3_RDMA_WRITE: @@ -105,17 +110,20 @@ static int iwch_poll_cq_one(struct iwch_dev *rhp, struct iwch_cq *chp, break; case T3_SEND: case T3_SEND_WITH_SE: + case T3_SEND_WITH_INV: + case T3_SEND_WITH_SE_INV: wc->opcode = IB_WC_SEND; break; case T3_BIND_MW: wc->opcode = IB_WC_BIND_MW; break; - /* these aren't supported yet */ - case T3_SEND_WITH_INV: - case T3_SEND_WITH_SE_INV: case T3_LOCAL_INV: + wc->opcode = IB_WC_LOCAL_INV; + break; case T3_FAST_REGISTER: + wc->opcode = IB_WC_FAST_REG_MR; + break; default: printk(KERN_ERR MOD "Unexpected opcode %d " "in the CQE received for QPID=0x%0x\n", diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 8934178..089a7df 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -768,6 +768,67 @@ static int iwch_dealloc_mw(struct ib_mw *mw) return 0; } +static struct ib_mr *iwch_alloc_fast_reg_mr(struct ib_pd *pd, int pbl_depth) +{ + struct iwch_dev *rhp; + struct iwch_pd *php; + struct iwch_mr *mhp; + u32 mmid; + u32 stag = 0; + int ret; + + php = to_iwch_pd(pd); + rhp = php->rhp; + mhp = kzalloc(sizeof(*mhp), GFP_KERNEL); + if (!mhp) + return ERR_PTR(-ENOMEM); + + mhp->rhp = rhp; + ret = iwch_alloc_pbl(mhp, pbl_depth); + if (ret) { + kfree(mhp); + return ERR_PTR(ret); + } + mhp->attr.pbl_size = pbl_depth; + ret = cxio_allocate_stag(&rhp->rdev, &stag, php->pdid, mhp->attr.pbl_size, mhp->attr.pbl_addr); + if (ret) { + iwch_free_pbl(mhp); + kfree(mhp); + return ERR_PTR(ret); + } + mhp->attr.pdid = php->pdid; + mhp->attr.type = TPT_NON_SHARED_MR; + mhp->attr.stag = stag; + mhp->attr.state = 1; + mmid = (stag) >> 8; + mhp->ibmr.rkey = mhp->ibmr.lkey = stag; + insert_handle(rhp, &rhp->mmidr, mhp, mmid); + PDBG("%s mmid 0x%x mhp %p stag 0x%x\n", __func__, mmid, mhp, stag); + return &(mhp->ibmr); +} + +static struct ib_fast_reg_page_list *iwch_alloc_fastreg_pbl( + struct ib_device *device, + int page_list_len) +{ + struct ib_fast_reg_page_list *page_list; + + page_list = kmalloc(sizeof *page_list + page_list_len * sizeof(u64), + GFP_KERNEL); + if (!page_list) + return ERR_PTR(-ENOMEM); + + page_list->page_list = (u64 *)(page_list + 1); + page_list->max_page_list_len = page_list_len; + + return page_list; +} + +static void iwch_free_fastreg_pbl(struct ib_fast_reg_page_list *page_list) +{ + kfree(page_list); +} + static int iwch_destroy_qp(struct ib_qp *ib_qp) { struct iwch_dev *rhp; @@ -843,6 +904,15 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, */ sqsize = roundup_pow_of_two(attrs->cap.max_send_wr); wqsize = roundup_pow_of_two(rqsize + sqsize); + + /* + * Kernel users need more wq space for fastreg WRs which can take + * 2 WR fragments. + */ + ucontext = pd->uobject ? to_iwch_ucontext(pd->uobject->context) : NULL; + if (!ucontext && wqsize < (rqsize + (2 * sqsize))) + wqsize = roundup_pow_of_two(rqsize + + roundup_pow_of_two(attrs->cap.max_send_wr * 2)); PDBG("%s wqsize %d sqsize %d rqsize %d\n", __func__, wqsize, sqsize, rqsize); qhp = kzalloc(sizeof(*qhp), GFP_KERNEL); @@ -851,7 +921,6 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, qhp->wq.size_log2 = ilog2(wqsize); qhp->wq.rq_size_log2 = ilog2(rqsize); qhp->wq.sq_size_log2 = ilog2(sqsize); - ucontext = pd->uobject ? to_iwch_ucontext(pd->uobject->context) : NULL; if (cxio_create_qp(&rhp->rdev, !udata, &qhp->wq, ucontext ? &ucontext->uctx : &rhp->rdev.uctx)) { kfree(qhp); @@ -1048,6 +1117,7 @@ static int iwch_query_device(struct ib_device *ibdev, props->max_mr = dev->attr.max_mem_regs; props->max_pd = dev->attr.max_pds; props->local_ca_ack_delay = 0; + props->max_fast_reg_page_list_len = T3_MAX_FASTREG_DEPTH; return 0; } @@ -1088,6 +1158,23 @@ static ssize_t show_rev(struct device *dev, struct device_attribute *attr, return sprintf(buf, "%d\n", iwch_dev->rdev.t3cdev_p->type); } +static int fw_supports_fastreg(struct iwch_dev *iwch_dev) +{ + struct ethtool_drvinfo info; + struct net_device *lldev = iwch_dev->rdev.t3cdev_p->lldev; + char *cp, *next; + unsigned fw_maj, fw_min; + + lldev->ethtool_ops->get_drvinfo(lldev, &info); + next = info.fw_version+1; + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_maj); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_min); + printk("%s maj %u min %u\n", __func__, fw_maj, fw_min); + return fw_maj > 6 || (fw_maj == 6 && fw_min > 0); +} + static ssize_t show_fw_ver(struct device *dev, struct device_attribute *attr, char *buf) { struct iwch_dev *iwch_dev = container_of(dev, struct iwch_dev, @@ -1145,8 +1232,10 @@ int iwch_register_device(struct iwch_dev *dev) memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid)); memcpy(&dev->ibdev.node_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); dev->ibdev.owner = THIS_MODULE; - dev->device_cap_flags = - (IB_DEVICE_ZERO_STAG | IB_DEVICE_MEM_WINDOW); + dev->device_cap_flags = IB_DEVICE_ZERO_STAG | + IB_DEVICE_MEM_WINDOW; + if (fw_supports_fastreg(dev)) + dev->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; dev->ibdev.uverbs_cmd_mask = (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) | @@ -1198,6 +1287,9 @@ int iwch_register_device(struct iwch_dev *dev) dev->ibdev.alloc_mw = iwch_alloc_mw; dev->ibdev.bind_mw = iwch_bind_mw; dev->ibdev.dealloc_mw = iwch_dealloc_mw; + dev->ibdev.alloc_fast_reg_mr = iwch_alloc_fast_reg_mr; + dev->ibdev.alloc_fast_reg_page_list = iwch_alloc_fastreg_pbl; + dev->ibdev.free_fast_reg_page_list = iwch_free_fastreg_pbl; dev->ibdev.attach_mcast = iwch_multicast_attach; dev->ibdev.detach_mcast = iwch_multicast_detach; diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.h b/drivers/infiniband/hw/cxgb3/iwch_provider.h index 836163f..f5ceca0 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.h +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.h @@ -296,14 +296,6 @@ static inline u32 iwch_ib_to_tpt_access(int acc) TPT_LOCAL_READ; } -static inline u32 iwch_ib_to_mwbind_access(int acc) -{ - return (acc & IB_ACCESS_REMOTE_WRITE ? T3_MEM_ACCESS_REM_WRITE : 0) | - (acc & IB_ACCESS_REMOTE_READ ? T3_MEM_ACCESS_REM_READ : 0) | - (acc & IB_ACCESS_LOCAL_WRITE ? T3_MEM_ACCESS_LOCAL_WRITE : 0) | - T3_MEM_ACCESS_LOCAL_READ; -} - enum iwch_mmid_state { IWCH_STAG_STATE_VALID, IWCH_STAG_STATE_INVALID diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 9926137..02b3909 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -44,54 +44,39 @@ static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, switch (wr->opcode) { case IB_WR_SEND: - case IB_WR_SEND_WITH_IMM: if (wr->send_flags & IB_SEND_SOLICITED) wqe->send.rdmaop = T3_SEND_WITH_SE; else wqe->send.rdmaop = T3_SEND; wqe->send.rem_stag = 0; break; -#if 0 /* Not currently supported */ - case TYPE_SEND_INVALIDATE: - case TYPE_SEND_INVALIDATE_IMMEDIATE: - wqe->send.rdmaop = T3_SEND_WITH_INV; - wqe->send.rem_stag = cpu_to_be32(wr->wr.rdma.rkey); - break; - case TYPE_SEND_SE_INVALIDATE: - wqe->send.rdmaop = T3_SEND_WITH_SE_INV; - wqe->send.rem_stag = cpu_to_be32(wr->wr.rdma.rkey); + case IB_WR_SEND_WITH_INV: + if (wr->send_flags & IB_SEND_SOLICITED) + wqe->send.rdmaop = T3_SEND_WITH_SE_INV; + else + wqe->send.rdmaop = T3_SEND_WITH_INV; + wqe->send.rem_stag = cpu_to_be32(wr->ex.invalidate_rkey); break; -#endif default: - break; + return -EINVAL; } if (wr->num_sge > T3_MAX_SGE) return -EINVAL; wqe->send.reserved[0] = 0; wqe->send.reserved[1] = 0; wqe->send.reserved[2] = 0; - if (wr->opcode == IB_WR_SEND_WITH_IMM) { - plen = 4; - wqe->send.sgl[0].stag = wr->ex.imm_data; - wqe->send.sgl[0].len = __constant_cpu_to_be32(0); - wqe->send.num_sgle = __constant_cpu_to_be32(0); - *flit_cnt = 5; - } else { - plen = 0; - for (i = 0; i < wr->num_sge; i++) { - if ((plen + wr->sg_list[i].length) < plen) { - return -EMSGSIZE; - } - plen += wr->sg_list[i].length; - wqe->send.sgl[i].stag = - cpu_to_be32(wr->sg_list[i].lkey); - wqe->send.sgl[i].len = - cpu_to_be32(wr->sg_list[i].length); - wqe->send.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); + plen = 0; + for (i = 0; i < wr->num_sge; i++) { + if ((plen + wr->sg_list[i].length) < plen) { + return -EMSGSIZE; } - wqe->send.num_sgle = cpu_to_be32(wr->num_sge); - *flit_cnt = 4 + ((wr->num_sge) << 1); + plen += wr->sg_list[i].length; + wqe->send.sgl[i].stag = cpu_to_be32(wr->sg_list[i].lkey); + wqe->send.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); + wqe->send.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); } + wqe->send.num_sgle = cpu_to_be32(wr->num_sge); + *flit_cnt = 4 + ((wr->num_sge) << 1); wqe->send.plen = cpu_to_be32(plen); return 0; } @@ -155,6 +140,57 @@ static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } +static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, + u8 *flit_cnt, int *wr_cnt, struct t3_wq *wq) +{ + int i; + __be64 *p; + + if (wr->wr.fast_reg.page_list_len > T3_MAX_FASTREG_DEPTH) + return -EINVAL; + *wr_cnt = 1; + wqe->fastreg.stag = cpu_to_be32(wr->wr.fast_reg.rkey); + wqe->fastreg.len = cpu_to_be32(wr->wr.fast_reg.length); + wqe->fastreg.va_base_hi = cpu_to_be32(wr->wr.fast_reg.iova_start>>32); + wqe->fastreg.va_base_lo_fbo = + cpu_to_be32(wr->wr.fast_reg.iova_start&0xffffffff); + wqe->fastreg.page_type_perms = cpu_to_be32( + V_FR_PAGE_COUNT(wr->wr.fast_reg.page_list_len) | + V_FR_PAGE_SIZE(wr->wr.fast_reg.page_shift-12) | + V_FR_TYPE(TPT_VATO) | + V_FR_PERMS(iwch_ib_to_tpt_access(wr->wr.fast_reg.access_flags))); + p = &wqe->fastreg.pbl_addrs[0]; + for (i=0; iwr.fast_reg.page_list_len; i++, p++) { + + /* If we need a 2nd WR, then set it up */ + if (i == T3_MAX_FASTREG_FRAG) { + *wr_cnt = 2; + wqe = (union t3_wr *)(wq->queue + + Q_PTR2IDX((wq->wptr+1), wq->size_log2)); + build_fw_riwrh((void *)wqe, T3_WR_FASTREG, 0, + Q_GENBIT(wq->wptr + 1, wq->size_log2), + 0, 1 + wr->wr.fast_reg.page_list_len - T3_MAX_FASTREG_FRAG, + T3_EOP); + + p = &wqe->pbl_frag.pbl_addrs[0]; + } + *p = cpu_to_be64((u64)wr->wr.fast_reg.page_list->page_list[i]); + } + *flit_cnt = 5 + wr->wr.fast_reg.page_list_len; + if (*flit_cnt > 15) + *flit_cnt = 15; + return 0; +} + +static int iwch_build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, + u8 *flit_cnt) +{ + wqe->local_inv.stag = cpu_to_be32(wr->ex.invalidate_rkey); + wqe->local_inv.reserved = 0; + *flit_cnt = sizeof(struct t3_local_inv_wr) >> 3; + return 0; +} + /* * TBD: this is going to be moved to firmware. Missing pdid/qpid check for now. */ @@ -238,6 +274,7 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, u32 num_wrs; unsigned long flag; struct t3_swsq *sqp; + int wr_cnt = 1; qhp = to_iwch_qp(ibqp); spin_lock_irqsave(&qhp->lock, flag); @@ -262,15 +299,15 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, t3_wr_flags = 0; if (wr->send_flags & IB_SEND_SOLICITED) t3_wr_flags |= T3_SOLICITED_EVENT_FLAG; - if (wr->send_flags & IB_SEND_FENCE) - t3_wr_flags |= T3_READ_FENCE_FLAG; if (wr->send_flags & IB_SEND_SIGNALED) t3_wr_flags |= T3_COMPLETION_FLAG; sqp = qhp->wq.sq + Q_PTR2IDX(qhp->wq.sq_wptr, qhp->wq.sq_size_log2); switch (wr->opcode) { case IB_WR_SEND: - case IB_WR_SEND_WITH_IMM: + case IB_WR_SEND_WITH_INV: + if (wr->send_flags & IB_SEND_FENCE) + t3_wr_flags |= T3_READ_FENCE_FLAG; t3_wr_opcode = T3_WR_SEND; err = iwch_build_rdma_send(wqe, wr, &t3_wr_flit_cnt); break; @@ -289,6 +326,17 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, if (!qhp->wq.oldest_read) qhp->wq.oldest_read = sqp; break; + case IB_WR_FAST_REG_MR: + t3_wr_opcode = T3_WR_FASTREG; + err = iwch_build_fastreg(wqe, wr, &t3_wr_flit_cnt, + &wr_cnt, &qhp->wq); + break; + case IB_WR_LOCAL_INV: + if (wr->send_flags & IB_SEND_FENCE) + t3_wr_flags |= T3_LOCAL_FENCE_FLAG; + t3_wr_opcode = T3_WR_INV_STAG; + err = iwch_build_inv_stag(wqe, wr, &t3_wr_flit_cnt); + break; default: PDBG("%s post of type=%d TBD!\n", __func__, wr->opcode); @@ -307,14 +355,15 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, build_fw_riwrh((void *) wqe, t3_wr_opcode, t3_wr_flags, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), - 0, t3_wr_flit_cnt); + 0, t3_wr_flit_cnt, + (wr_cnt == 1) ? T3_SOPEOP : T3_SOP); PDBG("%s cookie 0x%llx wq idx 0x%x swsq idx %ld opcode %d\n", __func__, (unsigned long long) wr->wr_id, idx, Q_PTR2IDX(qhp->wq.sq_wptr, qhp->wq.sq_size_log2), sqp->opcode); wr = wr->next; num_wrs--; - ++(qhp->wq.wptr); + qhp->wq.wptr += wr_cnt; ++(qhp->wq.sq_wptr); } spin_unlock_irqrestore(&qhp->lock, flag); @@ -359,7 +408,7 @@ int iwch_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, wr->wr_id; build_fw_riwrh((void *) wqe, T3_WR_RCV, T3_COMPLETION_FLAG, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), - 0, sizeof(struct t3_receive_wr) >> 3); + 0, sizeof(struct t3_receive_wr) >> 3, T3_SOPEOP); PDBG("%s cookie 0x%llx idx 0x%x rq_wptr 0x%x rw_rptr 0x%x " "wqe %p \n", __func__, (unsigned long long) wr->wr_id, idx, qhp->wq.rq_wptr, qhp->wq.rq_rptr, wqe); @@ -419,10 +468,10 @@ int iwch_bind_mw(struct ib_qp *qp, sgl.lkey = mw_bind->mr->lkey; sgl.length = mw_bind->length; wqe->bind.reserved = 0; - wqe->bind.type = T3_VA_BASED_TO; + wqe->bind.type = TPT_VATO; /* TBD: check perms */ - wqe->bind.perms = iwch_ib_to_mwbind_access(mw_bind->mw_access_flags); + wqe->bind.perms = iwch_ib_to_tpt_access(mw_bind->mw_access_flags); wqe->bind.mr_stag = cpu_to_be32(mw_bind->mr->lkey); wqe->bind.mw_stag = cpu_to_be32(mw->rkey); wqe->bind.mw_len = cpu_to_be32(mw_bind->length); @@ -441,10 +490,9 @@ int iwch_bind_mw(struct ib_qp *qp, sqp->signaled = (mw_bind->send_flags & IB_SEND_SIGNALED); wqe->bind.mr_pbl_addr = cpu_to_be32(pbl_addr); wqe->bind.mr_pagesz = page_size; - wqe->flit[T3_SQ_COOKIE_FLIT] = mw_bind->wr_id; build_fw_riwrh((void *)wqe, T3_WR_BIND, t3_wr_flags, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), 0, - sizeof(struct t3_bind_mw_wr) >> 3); + sizeof(struct t3_bind_mw_wr) >> 3, T3_SOPEOP); ++(qhp->wq.wptr); ++(qhp->wq.sq_wptr); spin_unlock_irqrestore(&qhp->lock, flag); From rdreier at cisco.com Mon Jun 23 11:10:27 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 11:10:27 -0700 Subject: [ofa-general] [PATCH for-2.6.27] Roland, In-Reply-To: <20080623180141.5813.50187.stgit@dell3.ogc.int> (Steve Wise's message of "Mon, 23 Jun 2008 13:01:41 -0500") References: <20080623180141.5813.50187.stgit@dell3.ogc.int> Message-ID: Actually I had already merged it, but I just replaced the patch with your new one. Sending future updates as deltas would be good tho. From swise at opengridcomputing.com Mon Jun 23 11:12:06 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 23 Jun 2008 13:12:06 -0500 Subject: [ofa-general] [PATCH for-2.6.27] Roland, In-Reply-To: References: <20080623180141.5813.50187.stgit@dell3.ogc.int> Message-ID: <485FE776.3020102@opengridcomputing.com> Roland Dreier wrote: > Actually I had already merged it, but I just replaced the patch with > your new one. Sending future updates as deltas would be good tho. > Ok. Cuz I fixed a bug in this last round. :) I'll post deltas from now on. Thanks! Steve. From patrick.latifi at qlogic.com Mon Jun 23 11:36:46 2008 From: patrick.latifi at qlogic.com (Patrick Marchand Latifi) Date: Mon, 23 Jun 2008 11:36:46 -0700 Subject: [ofa-general] [PATCH 4/7][v1.2] dapl scm: Adding socket cm provider for better scalability on large homogeneous clusters. In-Reply-To: References: Message-ID: <20080623183646.GG29683@ammonite.mv.qlogic.com> It seems like you have some inconsistency in how you handle the SGEs in dapls_ib_post_recv and dapls_ib_post_send between uDAPL v1 and uDAPL v2. For example in v1, you now leak the SGEs if the SGEs aren't allocated on the stack whereas in v2, you free them. Is that behavior intended ? Thanks for clarifying, -pat On Fri, Jun 20, 2008 at 11:50:11AM -0700, Davis, Arlin R wrote: > > Bring socket cm provider back to life with some changes: > > better threading support for exchanging QP information. > Avoid blocking during connect to support dynamic connection > model with MPI implementations. > > consumer control of ack timeout/retries. > > disconnect/reject capabilities via socket exchange. > > version support for wire protocol to insure compatibility > with peer scm provider. Add gids to exchange. > > validated with Intel MPI on a 14,000+ core fabric using IB DDR. > --- > Makefile.am | 134 ++++++- > dapl.spec.in | 10 +- > dapl/openib_scm/dapl_ib_cm.c | 833 > +++++++++++++++++++++++++--------------- > dapl/openib_scm/dapl_ib_cq.c | 19 +- > dapl/openib_scm/dapl_ib_dto.h | 212 ++++++----- > dapl/openib_scm/dapl_ib_mem.c | 137 ++++---- > dapl/openib_scm/dapl_ib_qp.c | 53 ++-- > dapl/openib_scm/dapl_ib_util.c | 152 ++++---- > dapl/openib_scm/dapl_ib_util.h | 44 ++- > 9 files changed, 1016 insertions(+), 578 deletions(-) > > diff --git a/Makefile.am b/Makefile.am > index 079ad7f..bc7926d 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -19,9 +19,11 @@ endif > > datlibdir = $(libdir) > dapllibcmadir = $(libdir) > +dapllibscmdir = $(libdir) > > datlib_LTLIBRARIES = dat/udat/libdat.la > dapllibcma_LTLIBRARIES = dapl/udapl/libdaplcma.la > +dapllibscm_LTLIBRARIES = dapl/udapl/libdaplscm.la > > dat_udat_libdat_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE $(OSFLAGS) > \ > -I$(srcdir)/dat/include/ > -I$(srcdir)/dat/udat/ \ > @@ -32,15 +34,22 @@ dapl_udapl_libdaplcma_la_CFLAGS = -Wall $(DBGFLAGS) > -D_GNU_SOURCE $(OSFLAGS) \ > -I$(srcdir)/dat/include/ > -I$(srcdir)/dapl/include/ \ > -I$(srcdir)/dapl/common > -I$(srcdir)/dapl/udapl/linux \ > -I$(srcdir)/dapl/openib_cma > + > +dapl_udapl_libdaplscm_la_CFLAGS = -Wall $(DBGFLAGS) -D_GNU_SOURCE > $(OSFLAGS) $(XFLAGS) \ > + -DOPENIB -DCQ_WAIT_OBJECT \ > + -I$(srcdir)/dat/include/ > -I$(srcdir)/dapl/include/ \ > + -I$(srcdir)/dapl/common > -I$(srcdir)/dapl/udapl/linux \ > + -I$(srcdir)/dapl/openib_scm > + > > if HAVE_LD_VERSION_SCRIPT > dat_version_script = > -Wl,--version-script=$(srcdir)/dat/udat/libdat.map > daplcma_version_script = > -Wl,--version-script=$(srcdir)/dapl/udapl/libdaplcma.map > - > + daplscm_version_script = > -Wl,--version-script=$(srcdir)/dapl/udapl/libdaplscm.map > else > - dat_version_script = > - daplcma_version_script = > - > + dat_version_script = > + daplofa_version_script = > + daplscm_version_script = > endif > > # > @@ -168,6 +177,115 @@ dapl_udapl_libdaplcma_la_LDFLAGS = -version-info > 1:2:0 $(daplcma_version_script) > -Wl,-init,dapl_init > -Wl,-fini,dapl_fini \ > -lpthread -libverbs -lrdmacm > > +# > +# uDAPL OpenFabrics Socket CM version: libdaplscm.so > +# > +dapl_udapl_libdaplscm_la_SOURCES = dapl/udapl/dapl_init.c \ > + dapl/udapl/dapl_evd_create.c \ > + dapl/udapl/dapl_evd_query.c \ > + dapl/udapl/dapl_cno_create.c \ > + dapl/udapl/dapl_cno_modify_agent.c \ > + dapl/udapl/dapl_cno_free.c \ > + dapl/udapl/dapl_cno_wait.c \ > + dapl/udapl/dapl_cno_query.c \ > + dapl/udapl/dapl_lmr_create.c \ > + dapl/udapl/dapl_evd_wait.c \ > + dapl/udapl/dapl_evd_disable.c \ > + dapl/udapl/dapl_evd_enable.c \ > + dapl/udapl/dapl_evd_modify_cno.c \ > + dapl/udapl/dapl_evd_set_unwaitable.c \ > + dapl/udapl/dapl_evd_clear_unwaitable.c \ > + dapl/udapl/linux/dapl_osd.c \ > + dapl/common/dapl_cookie.c \ > + dapl/common/dapl_cr_accept.c \ > + dapl/common/dapl_cr_query.c \ > + dapl/common/dapl_cr_reject.c \ > + dapl/common/dapl_cr_util.c \ > + dapl/common/dapl_cr_callback.c \ > + dapl/common/dapl_cr_handoff.c \ > + dapl/common/dapl_ep_connect.c \ > + dapl/common/dapl_ep_create.c \ > + dapl/common/dapl_ep_disconnect.c \ > + dapl/common/dapl_ep_dup_connect.c \ > + dapl/common/dapl_ep_free.c \ > + dapl/common/dapl_ep_reset.c \ > + dapl/common/dapl_ep_get_status.c \ > + dapl/common/dapl_ep_modify.c \ > + dapl/common/dapl_ep_post_rdma_read.c \ > + dapl/common/dapl_ep_post_rdma_write.c \ > + dapl/common/dapl_ep_post_recv.c \ > + dapl/common/dapl_ep_post_send.c \ > + dapl/common/dapl_ep_query.c \ > + dapl/common/dapl_ep_util.c \ > + dapl/common/dapl_evd_dequeue.c \ > + dapl/common/dapl_evd_free.c \ > + dapl/common/dapl_evd_post_se.c \ > + dapl/common/dapl_evd_resize.c \ > + dapl/common/dapl_evd_util.c \ > + dapl/common/dapl_evd_cq_async_error_callb.c \ > + dapl/common/dapl_evd_qp_async_error_callb.c \ > + dapl/common/dapl_evd_un_async_error_callb.c \ > + dapl/common/dapl_evd_connection_callb.c \ > + dapl/common/dapl_evd_dto_callb.c \ > + dapl/common/dapl_get_consumer_context.c \ > + dapl/common/dapl_get_handle_type.c \ > + dapl/common/dapl_hash.c \ > + dapl/common/dapl_hca_util.c \ > + dapl/common/dapl_ia_close.c \ > + dapl/common/dapl_ia_open.c \ > + dapl/common/dapl_ia_query.c \ > + dapl/common/dapl_ia_util.c \ > + dapl/common/dapl_llist.c \ > + dapl/common/dapl_lmr_free.c \ > + dapl/common/dapl_lmr_query.c \ > + dapl/common/dapl_lmr_util.c \ > + dapl/common/dapl_lmr_sync_rdma_read.c \ > + dapl/common/dapl_lmr_sync_rdma_write.c \ > + dapl/common/dapl_mr_util.c \ > + dapl/common/dapl_provider.c \ > + dapl/common/dapl_sp_util.c \ > + dapl/common/dapl_psp_create.c \ > + dapl/common/dapl_psp_create_any.c \ > + dapl/common/dapl_psp_free.c \ > + dapl/common/dapl_psp_query.c \ > + dapl/common/dapl_pz_create.c \ > + dapl/common/dapl_pz_free.c \ > + dapl/common/dapl_pz_query.c \ > + dapl/common/dapl_pz_util.c \ > + dapl/common/dapl_rmr_create.c \ > + dapl/common/dapl_rmr_free.c \ > + dapl/common/dapl_rmr_bind.c \ > + dapl/common/dapl_rmr_query.c \ > + dapl/common/dapl_rmr_util.c \ > + dapl/common/dapl_rsp_create.c \ > + dapl/common/dapl_rsp_free.c \ > + dapl/common/dapl_rsp_query.c \ > + dapl/common/dapl_cno_util.c \ > + dapl/common/dapl_set_consumer_context.c \ > + dapl/common/dapl_ring_buffer_util.c \ > + dapl/common/dapl_name_service.c \ > + dapl/common/dapl_timer_util.c \ > + dapl/common/dapl_ep_create_with_srq.c \ > + dapl/common/dapl_ep_recv_query.c \ > + dapl/common/dapl_ep_set_watermark.c \ > + dapl/common/dapl_srq_create.c \ > + dapl/common/dapl_srq_free.c \ > + dapl/common/dapl_srq_query.c \ > + dapl/common/dapl_srq_resize.c \ > + dapl/common/dapl_srq_post_recv.c \ > + dapl/common/dapl_srq_set_lw.c \ > + dapl/common/dapl_srq_util.c \ > + dapl/common/dapl_debug.c \ > + dapl/openib_scm/dapl_ib_util.c \ > + dapl/openib_scm/dapl_ib_cq.c \ > + dapl/openib_scm/dapl_ib_qp.c \ > + dapl/openib_scm/dapl_ib_cm.c \ > + dapl/openib_scm/dapl_ib_mem.c > + > +dapl_udapl_libdaplscm_la_LDFLAGS = -version-info 1:2:0 > $(daplscm_version_script) \ > + -Wl,-init,dapl_init > -Wl,-fini,dapl_fini \ > + -lpthread -libverbs > + > libdatincludedir = $(includedir)/dat > > libdatinclude_HEADERS = dat/include/dat/dat.h \ > @@ -230,6 +348,7 @@ EXTRA_DIST = dat/common/dat_dictionary.h \ > dapl/openib_scm/dapl_ib_util.h \ > dat/udat/libdat.map \ > dapl/udapl/libdaplcma.map \ > + dapl/udapl/libdaplscm.map \ > dapl.spec.in \ > $(man_MANS) \ > test/dapltest/include/dapl_bpool.h \ > @@ -271,9 +390,10 @@ install-exec-hook: > fi; \ > echo OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 > dapl.1.2 '"ib0 0" ""' >> $(sysconfdir)/dat.conf; \ > echo OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 > dapl.1.2 '"ib1 0" ""' >> $(sysconfdir)/dat.conf; \ > - echo OpenIB-cma-2 u1.2 nonthreadsafe default libdaplcma.so.1 > dapl.1.2 '"ib2 0" ""' >> $(sysconfdir)/dat.conf; \ > - echo OpenIB-cma-3 u1.2 nonthreadsafe default libdaplcma.so.1 > dapl.1.2 '"ib3 0" ""' >> $(sysconfdir)/dat.conf; \ > - echo OpenIB-bond u1.2 nonthreadsafe default libdaplcma.so.1 > dapl.1.2 '"bond0 0" ""' >> $(sysconfdir)/dat.conf; > + echo OpenIB-mthca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mthca0 1" ""' >> $(sysconfdir)/dat.conf; \ > + echo OpenIB-mthca0-2 u1.2 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mthca0 2" ""' >> $(sysconfdir)/dat.conf; \ > + echo OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mlx4_0 1" ""' >> $(sysconfdir)/dat.conf; \ > + echo OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mlx4_0 2" ""' >> $(sysconfdir)/dat.conf; > > uninstall-hook: > if test -e $(sysconfdir)/dat.conf; then \ > diff --git a/dapl.spec.in b/dapl.spec.in > index 2153e27..3e64bdc 100644 > --- a/dapl.spec.in > +++ b/dapl.spec.in > @@ -95,10 +95,10 @@ if [ -e %{_sysconfdir}/dat.conf ]; then > fi > echo OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 > '"ib0 0" ""' >> %{_sysconfdir}/dat.conf > echo OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 > '"ib1 0" ""' >> %{_sysconfdir}/dat.conf > -echo OpenIB-cma-2 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 > '"ib2 0" ""' >> %{_sysconfdir}/dat.conf > -echo OpenIB-cma-3 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 > '"ib3 0" ""' >> %{_sysconfdir}/dat.conf > -echo OpenIB-bond u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 > '"bond0 0" ""' >> %{_sysconfdir}/dat.conf > - > +echo OpenIB-mthca0-1 u2.0 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mthca0 1" ""' >> %{_sysconfdir}/dat.conf > +echo OpenIB-mthca0-2 u2.0 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mthca0 2" ""' >> %{_sysconfdir}/dat.conf > +echo OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mlx4_0 1" ""' >> %{_sysconfdir}/dat.conf > +echo OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 > dapl.1.2 '"mlx4_0 2" ""' >> %{_sysconfdir}/dat.conf > > %postun > /sbin/ldconfig > @@ -130,7 +130,7 @@ fi > > %changelog > * Tue May 20 2008 Arlin Davis - 1.2.7 > -- DAT/DAPL Version 1.2.7 Release 1, OFED 1.3.1 GA > +- DAT/DAPL Version 1.2.7 Release 1, OFED 1.3.1 GA > > * Thu May 1 2008 Arlin Davis - 1.2.6 > - DAT/DAPL Version 1.2.6 Release 1, OFED 1.3.1 > diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c > index f534e8d..9e686d6 100644 > --- a/dapl/openib_scm/dapl_ib_cm.c > +++ b/dapl/openib_scm/dapl_ib_cm.c > @@ -57,76 +57,171 @@ > #include > #include > #include > -#include > +#include > +#include > > -/* prototypes */ > -static uint16_t dapli_get_lid( struct ibv_device *dev, int port ); > +#include > +#include > +#include > > -static DAT_RETURN dapli_socket_connect ( DAPL_EP *ep_ptr, > - DAT_IA_ADDRESS_PTR r_addr, > - DAT_CONN_QUAL r_qual, > - DAT_COUNT p_size, > - DAT_PVOID p_data > ); > +#if __BYTE_ORDER == __LITTLE_ENDIAN > +static inline uint64_t cpu_to_be64(uint64_t x) {return bswap_64(x);} > +#elif __BYTE_ORDER == __BIG_ENDIAN > +static inline uint64_t cpu_to_be64(uint64_t x) {return x;} > +#endif > > -static DAT_RETURN dapli_socket_listen ( DAPL_IA > *ia_ptr, > - DAT_CONN_QUAL > serviceID, > - DAPL_SP *sp_ptr > ); > +extern int g_scm_pipe[2]; > > -static DAT_RETURN dapli_socket_accept( ib_cm_srvc_handle_t cm_ptr ); > +static struct ib_cm_handle *dapli_cm_create(void) > +{ > + struct ib_cm_handle *cm_ptr; > > -static DAT_RETURN dapli_socket_accept_final( DAPL_EP *ep_ptr, > - DAPL_CR *cr_ptr, > - DAT_COUNT p_size, > - DAT_PVOID p_data > ); > + /* Allocate CM, init lock, and initialize */ > + if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) > + return NULL; > + > + if (dapl_os_lock_init(&cm_ptr->lock)) > + goto bail; > > -/* XXX temporary hack to get lid */ > -static uint16_t dapli_get_lid(IN struct ibv_device *dev, IN int port) > + (void)dapl_os_memzero(cm_ptr, sizeof(*cm_ptr)); > + cm_ptr->dst.ver = htons(DSCM_VER); > + cm_ptr->socket = -1; > + return cm_ptr; > +bail: > + dapl_os_free(cm_ptr, sizeof(*cm_ptr)); > + return NULL; > +} > + > +/* mark for destroy, remove all references, schedule cleanup */ > +static void dapli_cm_destroy(struct ib_cm_handle *cm_ptr) > { > - char path[128]; > - char val[16]; > - char name[256]; > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " cm_destroy: cm %p ep %p\n", cm_ptr,cm_ptr->ep); > + > + /* cleanup, never made it to work queue */ > + if (cm_ptr->state == SCM_INIT) { > + if (cm_ptr->socket >= 0) > + close(cm_ptr->socket); > + dapl_os_free(cm_ptr, sizeof(*cm_ptr)); > + return; > + } > > - if (sysfs_get_mnt_path(path, sizeof path)) { > - fprintf(stderr, "Couldn't find sysfs mount.\n"); > - return 0; > + dapl_os_lock(&cm_ptr->lock); > + cm_ptr->state = SCM_DESTROY; > + if (cm_ptr->ep) { > + cm_ptr->ep->cm_handle = IB_INVALID_HANDLE; > + cm_ptr->ep->qp_handle = IB_INVALID_HANDLE; > } > - sprintf(name, "%s/class/infiniband/%s/ports/%d/lid", path, > - ibv_get_device_name(dev), port); > > - if (sysfs_read_attribute_value(name, val, sizeof val)) { > - fprintf(stderr, "Couldn't read LID at %s\n", name); > - return 0; > + /* close socket if still active */ > + if (cm_ptr->socket >= 0) { > + close(cm_ptr->socket); > + cm_ptr->socket = -1; > } > - return strtol(val, NULL, 0); > + dapl_os_unlock(&cm_ptr->lock); > + > + /* wakeup work thread */ > + write(g_scm_pipe[1], "w", sizeof "w"); > +} > + > +/* queue socket for processing CM work */ > +static void dapli_cm_queue(struct ib_cm_handle *cm_ptr) > +{ > + /* add to work queue for cr thread processing */ > + dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry); > + dapl_os_lock(&cm_ptr->hca->ib_trans.lock); > + dapl_llist_add_tail(&cm_ptr->hca->ib_trans.list, > + (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr); > + dapl_os_unlock(&cm_ptr->hca->ib_trans.lock); > + > + /* wakeup CM work thread */ > + write(g_scm_pipe[1], "w", sizeof "w"); > +} > + > +static uint16_t dapli_get_lid(IN struct ibv_context *ctx, IN uint8_t > port) > +{ > + struct ibv_port_attr port_attr; > + > + if(ibv_query_port(ctx, port,&port_attr)) > + return(0xffff); > + else > + return(port_attr.lid); > } > > /* > - * ACTIVE: Create socket, connect, and exchange QP information > + * ACTIVE/PASSIVE: called from CR thread or consumer via ep_disconnect > */ > static DAT_RETURN > -dapli_socket_connect ( DAPL_EP *ep_ptr, > - DAT_IA_ADDRESS_PTR r_addr, > - DAT_CONN_QUAL r_qual, > - DAT_COUNT p_size, > - DAT_PVOID p_data ) > +dapli_socket_disconnect(ib_cm_handle_t cm_ptr) > { > - ib_cm_handle_t cm_ptr; > - DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; > + DAPL_EP *ep_ptr = cm_ptr->ep; > + DAT_UINT32 disc_data = htonl(0xdead); > + > + if (ep_ptr == NULL) > + return DAT_SUCCESS; > + > + dapl_os_lock(&cm_ptr->lock); > + if ((cm_ptr->state == SCM_INIT) || > + (cm_ptr->state == SCM_DISCONNECTED)) { > + dapl_os_unlock(&cm_ptr->lock); > + return DAT_SUCCESS; > + } else { > + /* send disc date, close socket, schedule destroy */ > + if (cm_ptr->socket >= 0) { > + write(cm_ptr->socket, &disc_data, > sizeof(disc_data)); > + close(cm_ptr->socket); > + cm_ptr->socket = -1; > + } > + cm_ptr->state = SCM_DISCONNECTED; > + write(g_scm_pipe[1], "w", sizeof "w"); > + } > + dapl_os_unlock(&cm_ptr->lock); > + > + > + if (ep_ptr->cr_ptr) { > + dapls_cr_callback(cm_ptr, > + IB_CME_DISCONNECTED, > + NULL, > + ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr); > + } else { > + dapl_evd_connection_callback(ep_ptr->cm_handle, > + IB_CME_DISCONNECTED, > + NULL, > + ep_ptr); > + } > + > + /* remove reference from endpoint */ > + ep_ptr->cm_handle = NULL; > + > + /* schedule destroy */ > + > + > + return DAT_SUCCESS; > +} > + > + > +/* > + * ACTIVE: Create socket, connect, defer exchange QP information to CR > thread > + * to avoid blocking. > + */ > +DAT_RETURN > +dapli_socket_connect(DAPL_EP *ep_ptr, > + DAT_IA_ADDRESS_PTR r_addr, > + DAT_CONN_QUAL r_qual, > + DAT_COUNT p_size, > + DAT_PVOID p_data) > +{ > + ib_cm_handle_t cm_ptr; > int len, opt = 1; > struct iovec iovec[2]; > - short rtu_data = htons(0x0E0F); > - > - dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d\n", r_qual); > + DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; > + > + dapl_dbg_log(DAPL_DBG_TYPE_EP, " connect: r_qual %d > p_size=%d\n", > + r_qual,p_size); > > - /* > - * Allocate CM and initialize > - */ > - if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL ) { > + cm_ptr = dapli_cm_create(); > + if (cm_ptr == NULL) > return DAT_INSUFFICIENT_RESOURCES; > - } > - > - (void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) ); > - cm_ptr->socket = -1; > > /* create, connect, sockopt, and exchange QP information */ > if ((cm_ptr->socket = socket(AF_INET,SOCK_STREAM,0)) < 0 ) { > @@ -136,197 +231,252 @@ dapli_socket_connect ( DAPL_EP > *ep_ptr, > > ((struct sockaddr_in*)r_addr)->sin_port = htons(r_qual); > > - if ( connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0 ) { > + if (connect(cm_ptr->socket, r_addr, sizeof(*r_addr)) < 0) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > " connect: %s on r_qual %d\n", > strerror(errno), (unsigned int)r_qual); > - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); > + dapli_cm_destroy(cm_ptr); > return DAT_INVALID_ADDRESS; > } > > setsockopt(cm_ptr->socket,IPPROTO_TCP,TCP_NODELAY,&opt,sizeof(opt)); > - > + > + dapl_dbg_log(DAPL_DBG_TYPE_EP, " socket connected!\n"); > + > + > /* Send QP info, IA address, and private data */ > - cm_ptr->dst.qpn = ep_ptr->qp_handle->qp_num; > - cm_ptr->dst.port = ia_ptr->hca_ptr->port_num; > - cm_ptr->dst.lid = dapli_get_lid( > ia_ptr->hca_ptr->ib_trans.ib_dev, > - ia_ptr->hca_ptr->port_num ); > + cm_ptr->dst.qpn = htonl(ep_ptr->qp_handle->qp_num); > + cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); > + cm_ptr->dst.lid = > + htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, > + > (uint8_t)ia_ptr->hca_ptr->port_num)); > + if (cm_ptr->dst.lid == 0xffff) > + goto bail; > + > + /* in network order */ > + if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, > + (uint8_t)ia_ptr->hca_ptr->port_num, > + 0, > + &cm_ptr->dst.gid)) > + goto bail; > + > cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; > - cm_ptr->dst.p_size = p_size; > + cm_ptr->dst.p_size = htonl(p_size); > iovec[0].iov_base = &cm_ptr->dst; > iovec[0].iov_len = sizeof(ib_qp_cm_t); > - if ( p_size ) { > + if (p_size) { > iovec[1].iov_base = p_data; > iovec[1].iov_len = p_size; > } > - len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) ); > - if ( len != (p_size + sizeof(ib_qp_cm_t)) ) { > + > + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket connected, write QP and > private data\n"); > + len = writev(cm_ptr->socket, iovec, (p_size ? 2:1)); > + if (len != (p_size + sizeof(ib_qp_cm_t))) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > " connect write: ERR %s, wcnt=%d\n", > strerror(errno), len); > goto bail; > } > - dapl_dbg_log(DAPL_DBG_TYPE_EP, > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > " connect: SRC port=0x%x lid=0x%x, qpn=0x%x, > psize=%d\n", > - cm_ptr->dst.port, cm_ptr->dst.lid, > - cm_ptr->dst.qpn, cm_ptr->dst.p_size ); > + ntohs(cm_ptr->dst.port), ntohs(cm_ptr->dst.lid), > + ntohl(cm_ptr->dst.qpn), ntohl(cm_ptr->dst.p_size)); > > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " connect SRC GID subnet %016llx id %016llx\n", > + (unsigned long long) > + > cpu_to_be64(cm_ptr->dst.gid.global.subnet_prefix), > + (unsigned long long) > + > cpu_to_be64(cm_ptr->dst.gid.global.interface_id)); > + > + /* queue up to work thread to avoid blocking consumer */ > + cm_ptr->state = SCM_CONN_PENDING; > + cm_ptr->hca = ia_ptr->hca_ptr; > + cm_ptr->ep = ep_ptr; > + dapli_cm_queue(cm_ptr); > + return DAT_SUCCESS; > +bail: > + /* close socket, free cm structure */ > + dapli_cm_destroy(cm_ptr); > + return DAT_INTERNAL_ERROR; > +} > + > + > +/* > + * ACTIVE: exchange QP information, called from CR thread > + */ > +void > +dapli_socket_connect_rtu(ib_cm_handle_t cm_ptr) > +{ > + DAPL_EP *ep_ptr = cm_ptr->ep; > + int len; > + struct iovec iovec[2]; > + short rtu_data = htons(0x0E0F); > + ib_cm_events_t event = IB_CME_DESTINATION_REJECT; > > /* read DST information into cm_ptr, overwrite SRC info */ > - len = readv( cm_ptr->socket, iovec, 1 ); > - if ( len != sizeof(ib_qp_cm_t) ) { > + dapl_dbg_log(DAPL_DBG_TYPE_EP," connect_rtu: recv peer QP > data\n"); > + > + iovec[0].iov_base = &cm_ptr->dst; > + iovec[0].iov_len = sizeof(ib_qp_cm_t); > + len = readv(cm_ptr->socket, iovec, 1); > + if (len != sizeof(ib_qp_cm_t) || ntohs(cm_ptr->dst.ver) != > DSCM_VER) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " connect read: ERR %s, rcnt=%d\n", > - strerror(errno), len); > + " connect_rtu read: ERR %s, rcnt=%d, > ver=%d\n", > + strerror(errno), len, cm_ptr->dst.ver); > + goto bail; > + } > + /* check for consumer reject */ > + if (cm_ptr->dst.rej) { > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " connect_rtu read: PEER REJ > reason=0x%x\n", > + ntohs(cm_ptr->dst.rej)); > + event = IB_CME_DESTINATION_REJECT_PRIVATE_DATA; > goto bail; > } > + > + /* convert peer response values to host order */ > + cm_ptr->dst.port = ntohs(cm_ptr->dst.port); > + cm_ptr->dst.lid = ntohs(cm_ptr->dst.lid); > + cm_ptr->dst.qpn = ntohl(cm_ptr->dst.qpn); > + cm_ptr->dst.p_size = ntohl(cm_ptr->dst.p_size); > + > + /* save remote address information */ > + dapl_os_memcpy( &ep_ptr->remote_ia_address, > + &cm_ptr->dst.ia_address, > + sizeof(ep_ptr->remote_ia_address)); > + > dapl_dbg_log(DAPL_DBG_TYPE_EP, > - " connect: DST port=0x%x lid=0x%x, qpn=0x%x, > psize=%d\n", > + " connect_rtu: DST %s port=0x%x lid=0x%x, qpn=0x%x, > psize=%d\n", > + inet_ntoa(((struct sockaddr_in > *)&cm_ptr->dst.ia_address)->sin_addr), > cm_ptr->dst.port, cm_ptr->dst.lid, > - cm_ptr->dst.qpn, cm_ptr->dst.p_size ); > + cm_ptr->dst.qpn, cm_ptr->dst.p_size); > > /* validate private data size before reading */ > - if ( cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE ) { > + if (cm_ptr->dst.p_size > IB_MAX_REP_PDATA_SIZE) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " connect read: psize (%d) wrong\n", > + " connect_rtu read: psize (%d) wrong\n", > cm_ptr->dst.p_size ); > goto bail; > } > > /* read private data into cm_handle if any present */ > - if ( cm_ptr->dst.p_size ) { > + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket connected, read private > data\n"); > + if (cm_ptr->dst.p_size) { > iovec[0].iov_base = cm_ptr->p_data; > iovec[0].iov_len = cm_ptr->dst.p_size; > - len = readv( cm_ptr->socket, iovec, 1 ); > - if ( len != cm_ptr->dst.p_size ) { > + len = readv(cm_ptr->socket, iovec, 1); > + if (len != cm_ptr->dst.p_size) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " connect read pdata: ERR %s, > rcnt=%d\n", > + " connect_rtu read pdata: ERR %s, > rcnt=%d\n", > strerror(errno), len); > goto bail; > } > } > > /* modify QP to RTR and then to RTS with remote info */ > - if ( dapls_modify_qp_state( ep_ptr->qp_handle, > - IBV_QPS_RTR, &cm_ptr->dst ) != > DAT_SUCCESS ) > + if (dapls_modify_qp_state(ep_ptr->qp_handle, > + IBV_QPS_RTR, &cm_ptr->dst) != > DAT_SUCCESS) > goto bail; > > - if ( dapls_modify_qp_state( ep_ptr->qp_handle, > - IBV_QPS_RTS, &cm_ptr->dst ) != > DAT_SUCCESS ) > + if (dapls_modify_qp_state(ep_ptr->qp_handle, > + IBV_QPS_RTS, &cm_ptr->dst) != > DAT_SUCCESS) > goto bail; > > ep_ptr->qp_state = IB_QP_STATE_RTS; > > + dapl_dbg_log(DAPL_DBG_TYPE_EP," connect_rtu: send RTU\n"); > + > /* complete handshake after final QP state change */ > - write(cm_ptr->socket, &rtu_data, sizeof(rtu_data) ); > + write(cm_ptr->socket, &rtu_data, sizeof(rtu_data)); > > /* init cm_handle and post the event with private data */ > ep_ptr->cm_handle = cm_ptr; > - dapl_dbg_log( DAPL_DBG_TYPE_EP," ACTIVE: connected!\n" ); > - dapl_evd_connection_callback( ep_ptr->cm_handle, > - IB_CME_CONNECTED, > - cm_ptr->p_data, > - ep_ptr ); > - return DAT_SUCCESS; > - > + cm_ptr->state = SCM_CONNECTED; > + dapl_dbg_log(DAPL_DBG_TYPE_EP," ACTIVE: connected!\n"); > + dapl_evd_connection_callback(cm_ptr, > + IB_CME_CONNECTED, > + cm_ptr->p_data, > + ep_ptr); > + return; > bail: > /* close socket, free cm structure and post error event */ > - if ( cm_ptr->socket >= 0 ) > - close(cm_ptr->socket); > - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); > - dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */ > - > - dapl_evd_connection_callback( ep_ptr->cm_handle, > - IB_CME_LOCAL_FAILURE, > - NULL, > - ep_ptr ); > - return DAT_INTERNAL_ERROR; > + dapli_cm_destroy(cm_ptr); > + dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ > + dapl_evd_connection_callback(NULL, event, NULL, ep_ptr); > } > > - > /* > * PASSIVE: Create socket, listen, accept, exchange QP information > */ > -static DAT_RETURN > -dapli_socket_listen ( DAPL_IA *ia_ptr, > - DAT_CONN_QUAL serviceID, > - DAPL_SP *sp_ptr ) > +DAT_RETURN > +dapli_socket_listen(DAPL_IA *ia_ptr, > + DAT_CONN_QUAL serviceID, > + DAPL_SP *sp_ptr ) > { > struct sockaddr_in addr; > ib_cm_srvc_handle_t cm_ptr = NULL; > int opt = 1; > DAT_RETURN dat_status = DAT_SUCCESS; > > - dapl_dbg_log ( DAPL_DBG_TYPE_EP, > - " listen(ia_ptr %p ServiceID %d sp_ptr %p)\n", > - ia_ptr, serviceID, sp_ptr); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " listen(ia_ptr %p ServiceID %d sp_ptr %p)\n", > + ia_ptr, serviceID, sp_ptr); > > - /* Allocate CM and initialize */ > - if ((cm_ptr = dapl_os_alloc(sizeof(*cm_ptr))) == NULL) > + cm_ptr = dapli_cm_create(); > + if (cm_ptr == NULL) > return DAT_INSUFFICIENT_RESOURCES; > > - (void) dapl_os_memzero( cm_ptr, sizeof( *cm_ptr ) ); > - > - cm_ptr->socket = cm_ptr->l_socket = -1; > cm_ptr->sp = sp_ptr; > - cm_ptr->hca_ptr = ia_ptr->hca_ptr; > + cm_ptr->hca = ia_ptr->hca_ptr; > > /* bind, listen, set sockopt, accept, exchange data */ > - if ((cm_ptr->l_socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) { > + if ((cm_ptr->socket = socket(AF_INET, SOCK_STREAM, 0)) < 0) { > dapl_dbg_log (DAPL_DBG_TYPE_ERR, > "socket for listen returned %d\n", > errno); > dat_status = DAT_INSUFFICIENT_RESOURCES; > goto bail; > } > > - > setsockopt(cm_ptr->l_socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt)); > + > setsockopt(cm_ptr->socket,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof(opt)); > addr.sin_port = htons(serviceID); > addr.sin_family = AF_INET; > addr.sin_addr.s_addr = INADDR_ANY; > > - if (( bind( cm_ptr->l_socket,(struct sockaddr*)&addr, > sizeof(addr) ) < 0) || > - (listen( cm_ptr->l_socket, 128 ) < 0) ) { > - > + if ((bind(cm_ptr->socket,(struct sockaddr*)&addr, sizeof(addr)) > < 0) || > + (listen(cm_ptr->socket, 128) < 0)) { > dapl_dbg_log( DAPL_DBG_TYPE_CM, > " listen: ERROR %s on conn_qual 0x%x\n", > strerror(errno),serviceID); > - > - if ( errno == EADDRINUSE ) > + if (errno == EADDRINUSE) > dat_status = DAT_CONN_QUAL_IN_USE; > else > dat_status = DAT_CONN_QUAL_UNAVAILABLE; > - > goto bail; > } > > /* set cm_handle for this service point, save listen socket */ > sp_ptr->cm_srvc_handle = cm_ptr; > > - /* add to SP->CR thread list */ > - dapl_llist_init_entry((DAPL_LLIST_ENTRY*)&cm_ptr->entry); > - dapl_os_lock( &cm_ptr->hca_ptr->ib_trans.lock ); > - dapl_llist_add_tail(&cm_ptr->hca_ptr->ib_trans.list, > - (DAPL_LLIST_ENTRY*)&cm_ptr->entry, cm_ptr); > - dapl_os_unlock(&cm_ptr->hca_ptr->ib_trans.lock); > + /* queue up listen socket to process inbound CR's */ > + cm_ptr->state = SCM_LISTEN; > + dapli_cm_queue(cm_ptr); > + > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " listen: qual 0x%x cr %p s_fd %d\n", > + ntohs(serviceID), cm_ptr, cm_ptr->socket ); > > - dapl_dbg_log( DAPL_DBG_TYPE_CM, > - " listen: qual 0x%x cr %p s_fd %d\n", > - ntohs(serviceID), cm_ptr, cm_ptr->l_socket ); > - > return dat_status; > bail: > dapl_dbg_log( DAPL_DBG_TYPE_CM, > " listen: ERROR on conn_qual 0x%x\n",serviceID); > > - if ( cm_ptr->l_socket >= 0 ) > - close( cm_ptr->l_socket ); > - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); > + dapli_cm_destroy(cm_ptr); > return dat_status; > } > > > /* > - * PASSIVE: send local QP information, private data, and wait for > - * active side to respond with QP RTS/RTR status > + * PASSIVE: accept socket, receive peer QP information, private data, > post cr_event > */ > -static DAT_RETURN > +DAT_RETURN > dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) > { > ib_cm_handle_t acm_ptr; > @@ -334,6 +484,8 @@ dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) > int len; > DAT_RETURN dat_status = DAT_SUCCESS; > > + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket_accept\n"); > + > /* Allocate accept CM and initialize */ > if ((acm_ptr = dapl_os_alloc(sizeof(*acm_ptr))) == NULL) > return DAT_INSUFFICIENT_RESOURCES; > @@ -342,155 +494,221 @@ dapli_socket_accept(ib_cm_srvc_handle_t cm_ptr) > > acm_ptr->socket = -1; > acm_ptr->sp = cm_ptr->sp; > - acm_ptr->hca_ptr = cm_ptr->hca_ptr; > + acm_ptr->hca = cm_ptr->hca; > > len = sizeof(acm_ptr->dst.ia_address); > - acm_ptr->socket = accept(cm_ptr->l_socket, > + acm_ptr->socket = accept(cm_ptr->socket, > (struct > sockaddr*)&acm_ptr->dst.ia_address, > - (socklen_t*)&len ); > + (socklen_t*)&len); > > - if ( acm_ptr->socket < 0 ) { > + if (acm_ptr->socket < 0) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > " accept: ERR %s on FD %d l_cr %p\n", > - strerror(errno),cm_ptr->l_socket,cm_ptr); > + strerror(errno),cm_ptr->socket,cm_ptr); > dat_status = DAT_INTERNAL_ERROR; > goto bail; > } > > + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket accepted, read QP > data\n"); > + > /* read in DST QP info, IA address. check for private data */ > - len = read( acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t) > ); > - if ( len != sizeof(ib_qp_cm_t) ) { > + len = read(acm_ptr->socket, &acm_ptr->dst, sizeof(ib_qp_cm_t)); > + if (len != sizeof(ib_qp_cm_t) || > + ntohs(acm_ptr->dst.ver) != DSCM_VER) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " accept read: ERR %s, rcnt=%d\n", > - strerror(errno), len); > + " accept read: ERR %s, rcnt=%d, ver=%d\n", > + strerror(errno), len, acm_ptr->dst.ver); > dat_status = DAT_INTERNAL_ERROR; > goto bail; > - > } > + > + /* convert accepted values to host order */ > + acm_ptr->dst.port = ntohs(acm_ptr->dst.port); > + acm_ptr->dst.lid = ntohs(acm_ptr->dst.lid); > + acm_ptr->dst.qpn = ntohl(acm_ptr->dst.qpn); > + acm_ptr->dst.p_size = ntohl(acm_ptr->dst.p_size); > + > dapl_dbg_log(DAPL_DBG_TYPE_EP, > - " accept: DST port=0x%x lid=0x%x, qpn=0x%x, psize=%d\n", > - acm_ptr->dst.port, acm_ptr->dst.lid, > - acm_ptr->dst.qpn, acm_ptr->dst.p_size ); > + " accept: DST %s port=0x%x lid=0x%x, qpn=0x%x, > psize=%d\n", > + inet_ntoa(((struct sockaddr_in > *)&acm_ptr->dst.ia_address)->sin_addr), > + acm_ptr->dst.port, acm_ptr->dst.lid, > + acm_ptr->dst.qpn, acm_ptr->dst.p_size); > > /* validate private data size before reading */ > - if ( acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE ) { > + if (acm_ptr->dst.p_size > IB_MAX_REQ_PDATA_SIZE) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " accept read: psize (%d) wrong\n", > - acm_ptr->dst.p_size ); > + " accept read: psize (%d) wrong\n", > + acm_ptr->dst.p_size); > dat_status = DAT_INTERNAL_ERROR; > goto bail; > } > > + dapl_dbg_log(DAPL_DBG_TYPE_EP," socket accepted, read private > data\n"); > + > /* read private data into cm_handle if any present */ > - if ( acm_ptr->dst.p_size ) { > + if (acm_ptr->dst.p_size) { > len = read( acm_ptr->socket, > - acm_ptr->p_data, acm_ptr->dst.p_size ); > - if ( len != acm_ptr->dst.p_size ) { > + acm_ptr->p_data, acm_ptr->dst.p_size); > + if (len != acm_ptr->dst.p_size) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " accept read pdata: ERR %s, rcnt=%d\n", > - strerror(errno), len ); > + " accept read pdata: ERR %s, > rcnt=%d\n", > + strerror(errno), len); > dat_status = DAT_INTERNAL_ERROR; > goto bail; > } > - dapl_dbg_log(DAPL_DBG_TYPE_EP, > - " accept: psize=%d read\n", > - acm_ptr->dst.p_size); > + dapl_dbg_log(DAPL_DBG_TYPE_EP," accept: psize=%d > read\n",len); > p_data = acm_ptr->p_data; > } > > - /* trigger CR event and return SUCCESS */ > - dapls_cr_callback( acm_ptr, > - IB_CME_CONNECTION_REQUEST_PENDING, > - p_data, > - acm_ptr->sp ); > + acm_ptr->state = SCM_ACCEPTING; > > + /* trigger CR event and return SUCCESS */ > + dapls_cr_callback(acm_ptr, > + IB_CME_CONNECTION_REQUEST_PENDING, > + p_data, > + acm_ptr->sp ); > return DAT_SUCCESS; > - > bail: > - if ( acm_ptr->socket >=0 ) > - close( acm_ptr->socket ); > - dapl_os_free( acm_ptr, sizeof( *acm_ptr ) ); > + dapli_cm_destroy(acm_ptr); > return DAT_INTERNAL_ERROR; > } > > - > -static DAT_RETURN > -dapli_socket_accept_final( DAPL_EP *ep_ptr, > - DAPL_CR *cr_ptr, > - DAT_COUNT p_size, > - DAT_PVOID p_data ) > +/* > + * PASSIVE: consumer accept, send local QP information, private data, > + * queue on work thread to receive RTU information to avoid blocking > + * user thread. > + */ > +DAT_RETURN > +dapli_socket_accept_usr(DAPL_EP *ep_ptr, > + DAPL_CR *cr_ptr, > + DAT_COUNT p_size, > + DAT_PVOID p_data) > { > DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; > - ib_cm_handle_t cm_ptr = cr_ptr->ib_cm_handle; > - ib_qp_cm_t qp_cm; > + ib_cm_handle_t cm_ptr = cr_ptr->ib_cm_handle; > struct iovec iovec[2]; > int len; > - short rtu_data = 0; > > - if (p_size > IB_MAX_REP_PDATA_SIZE) > + if (p_size > IB_MAX_REP_PDATA_SIZE) > return DAT_LENGTH_ERROR; > > /* must have a accepted socket */ > - if ( cm_ptr->socket < 0 ) > + if (cm_ptr->socket < 0) > return DAT_INTERNAL_ERROR; > > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " accept_usr: remote port=0x%x lid=0x%x" > + " qpn=0x%x psize=%d\n", > + cm_ptr->dst.port, cm_ptr->dst.lid, > + cm_ptr->dst.qpn, cm_ptr->dst.p_size); > + > /* modify QP to RTR and then to RTS with remote info already > read */ > - if ( dapls_modify_qp_state( ep_ptr->qp_handle, > - IBV_QPS_RTR, &cm_ptr->dst ) != > DAT_SUCCESS ) > + if (dapls_modify_qp_state(ep_ptr->qp_handle, > + IBV_QPS_RTR, &cm_ptr->dst) != > DAT_SUCCESS) > goto bail; > > - if ( dapls_modify_qp_state( ep_ptr->qp_handle, > - IBV_QPS_RTS, &cm_ptr->dst ) != > DAT_SUCCESS ) > + if (dapls_modify_qp_state(ep_ptr->qp_handle, > + IBV_QPS_RTS, &cm_ptr->dst) != > DAT_SUCCESS) > goto bail; > > ep_ptr->qp_state = IB_QP_STATE_RTS; > > - /* Send QP info, IA address, and private data */ > - qp_cm.qpn = ep_ptr->qp_handle->qp_num; > - qp_cm.port = ia_ptr->hca_ptr->port_num; > - qp_cm.lid = dapli_get_lid( ia_ptr->hca_ptr->ib_trans.ib_dev, > - ia_ptr->hca_ptr->port_num ); > - qp_cm.ia_address = ia_ptr->hca_ptr->hca_address; > - qp_cm.p_size = p_size; > - iovec[0].iov_base = &qp_cm; > + /* save remote address information */ > + dapl_os_memcpy( &ep_ptr->remote_ia_address, > + &cm_ptr->dst.ia_address, > + sizeof(ep_ptr->remote_ia_address)); > + > + /* send our QP info, IA address, and private data */ > + cm_ptr->dst.qpn = htonl(ep_ptr->qp_handle->qp_num); > + cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); > + cm_ptr->dst.lid = > htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, > + > (uint8_t)ia_ptr->hca_ptr->port_num)); > + if (cm_ptr->dst.lid == 0xffff) > + goto bail; > + > + /* in network order */ > + if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, > + (uint8_t)ia_ptr->hca_ptr->port_num, > + 0, > + &cm_ptr->dst.gid)) > + goto bail; > + > + cm_ptr->dst.ia_address = ia_ptr->hca_ptr->hca_address; > + cm_ptr->dst.p_size = htonl(p_size); > + iovec[0].iov_base = &cm_ptr->dst; > iovec[0].iov_len = sizeof(ib_qp_cm_t); > if (p_size) { > iovec[1].iov_base = p_data; > iovec[1].iov_len = p_size; > } > - len = writev( cm_ptr->socket, iovec, (p_size ? 2:1) ); > + len = writev(cm_ptr->socket, iovec, (p_size ? 2:1)); > if (len != (p_size + sizeof(ib_qp_cm_t))) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " accept_final: ERR %s, wcnt=%d\n", > + " accept_rtu: ERR %s, wcnt=%d\n", > strerror(errno), len); > goto bail; > } > - dapl_dbg_log(DAPL_DBG_TYPE_EP, > - " accept_final: SRC port=0x%x lid=0x%x, qpn=0x%x, > psize=%d\n", > - qp_cm.port, qp_cm.lid, qp_cm.qpn, qp_cm.p_size ); > - > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " accept_usr: local port=0x%x lid=0x%x" > + " qpn=0x%x psize=%d\n", > + ntohs(cm_ptr->dst.port), ntohs(cm_ptr->dst.lid), > + ntohl(cm_ptr->dst.qpn), ntohl(cm_ptr->dst.p_size)); > > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " accept_usr SRC GID subnet %016llx id %016llx\n", > + (unsigned long long) > + > cpu_to_be64(cm_ptr->dst.gid.global.subnet_prefix), > + (unsigned long long) > + > cpu_to_be64(cm_ptr->dst.gid.global.interface_id)); > + > + /* save state and reference to EP, queue for RTU data */ > + cm_ptr->ep = ep_ptr; > + cm_ptr->hca = ia_ptr->hca_ptr; > + cm_ptr->state = SCM_ACCEPTED; > + > + /* restore remote address information for query */ > + dapl_os_memcpy( &cm_ptr->dst.ia_address, > + &ep_ptr->remote_ia_address, > + sizeof(cm_ptr->dst.ia_address)); > + > + dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: accepted!\n" ); > + dapli_cm_queue(cm_ptr); > + return DAT_SUCCESS; > +bail: > + dapl_dbg_log(DAPL_DBG_TYPE_ERR," accept_rtu: ERR !QP_RTR_RTS > \n"); > + dapli_cm_destroy(cm_ptr); > + dapls_ib_reinit_ep(ep_ptr); /* reset QP state */ > + return DAT_INTERNAL_ERROR; > +} > + > +/* > + * PASSIVE: read RTU from active peer, post CONN event > + */ > +void > +dapli_socket_accept_rtu(ib_cm_handle_t cm_ptr) > +{ > + int len; > + short rtu_data = 0; > + > /* complete handshake after final QP state change */ > - len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data) ); > - if ( len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f ) { > + len = read(cm_ptr->socket, &rtu_data, sizeof(rtu_data)); > + if (len != sizeof(rtu_data) || ntohs(rtu_data) != 0x0e0f) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - " accept_final: ERR %s, rcnt=%d > rdata=%x\n", > - strerror(errno), len, ntohs(rtu_data) ); > + " accept_rtu: ERR %s, rcnt=%d rdata=%x\n", > + strerror(errno), len, ntohs(rtu_data)); > goto bail; > } > > + /* save state and reference to EP, queue for disc event */ > + cm_ptr->state = SCM_CONNECTED; > + > /* final data exchange if remote QP state is good to go */ > dapl_dbg_log( DAPL_DBG_TYPE_EP," PASSIVE: connected!\n" ); > - dapls_cr_callback ( cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp > ); > - return DAT_SUCCESS; > - > + dapls_cr_callback(cm_ptr, IB_CME_CONNECTED, NULL, cm_ptr->sp); > + return; > bail: > - dapl_dbg_log( DAPL_DBG_TYPE_ERR," accept_final: ERR !QP_RTR_RTS > \n"); > - if ( cm_ptr >= 0 ) > - close( cm_ptr->socket ); > - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); > - dapls_ib_reinit_ep( ep_ptr ); /* reset QP state */ > - > - return DAT_INTERNAL_ERROR; > + dapls_ib_reinit_ep(cm_ptr->ep); /* reset QP state */ > + dapli_cm_destroy(cm_ptr); > + dapls_cr_callback(cm_ptr, IB_CME_DESTINATION_REJECT, NULL, > cm_ptr->sp); > } > > > @@ -528,18 +746,13 @@ dapls_ib_connect ( > > dapl_dbg_log ( DAPL_DBG_TYPE_EP, > " connect(ep_handle %p ....)\n", ep_handle); > - /* > - * Sanity check > - */ > - if ( NULL == ep_handle ) > - return DAT_SUCCESS; > > ep_ptr = (DAPL_EP*)ep_handle; > qp_ptr = ep_ptr->qp_handle; > > - return (dapli_socket_connect( ep_ptr, remote_ia_address, > - remote_conn_qual, > - private_data_size, private_data > )); > + return (dapli_socket_connect(ep_ptr, remote_ia_address, > + remote_conn_qual, > + private_data_size, private_data)); > } > > /* > @@ -556,12 +769,11 @@ dapls_ib_connect ( > * > * Returns: > * DAT_SUCCESS > - * > */ > DAT_RETURN > -dapls_ib_disconnect ( > +dapls_ib_disconnect( > IN DAPL_EP *ep_ptr, > - IN DAT_CLOSE_FLAGS close_flags ) > + IN DAT_CLOSE_FLAGS close_flags) > { > ib_cm_handle_t cm_ptr = ep_ptr->cm_handle; > > @@ -569,28 +781,13 @@ dapls_ib_disconnect ( > "dapls_ib_disconnect(ep_handle %p ....)\n", > ep_ptr); > > - if ( cm_ptr->socket >= 0 ) { > - close( cm_ptr->socket ); > - cm_ptr->socket = -1; > - } > - > /* reinit to modify QP state */ > dapls_ib_reinit_ep(ep_ptr); > > - if ( ep_ptr->cr_ptr ) { > - dapls_cr_callback ( ep_ptr->cm_handle, > - IB_CME_DISCONNECTED, > - NULL, > - ((DAPL_CR *)ep_ptr->cr_ptr)->sp_ptr > ); > - } else { > - dapl_evd_connection_callback ( ep_ptr->cm_handle, > - IB_CME_DISCONNECTED, > - NULL, > - ep_ptr ); > - ep_ptr->cm_handle = NULL; > - dapl_os_free( cm_ptr, sizeof( *cm_ptr ) ); > - } > - return DAT_SUCCESS; > + if (cm_ptr == NULL) > + return DAT_SUCCESS; > + else > + return(dapli_socket_disconnect(cm_ptr)); > } > > /* > @@ -679,13 +876,14 @@ dapls_ib_remove_conn_listener ( > ia_ptr, sp_ptr, cm_ptr ); > > /* close accepted socket, free cm_srvc_handle and return */ > - if ( cm_ptr != NULL ) { > - if ( cm_ptr->l_socket >= 0 ) { > - close( cm_ptr->l_socket ); > + if (cm_ptr != NULL) { > + if (cm_ptr->socket >= 0) { > + close(cm_ptr->socket ); > cm_ptr->socket = -1; > } > /* cr_thread will free */ > sp_ptr->cm_srvc_handle = NULL; > + write(g_scm_pipe[1], "w", sizeof "w"); > } > return DAT_SUCCESS; > } > @@ -720,23 +918,22 @@ dapls_ib_accept_connection ( > DAPL_CR *cr_ptr; > DAPL_EP *ep_ptr; > > - dapl_dbg_log (DAPL_DBG_TYPE_EP, > - "dapls_ib_accept_connection(cr %p ep %p prd > %p,%d)\n", > - cr_handle, ep_handle, p_data, p_size ); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + "dapls_ib_accept_connection(cr %p ep %p prd > %p,%d)\n", > + cr_handle, ep_handle, p_data, p_size ); > > - cr_ptr = (DAPL_CR *) cr_handle; > - ep_ptr = (DAPL_EP *) ep_handle; > + cr_ptr = (DAPL_CR *)cr_handle; > + ep_ptr = (DAPL_EP *)ep_handle; > > /* allocate and attach a QP if necessary */ > - if ( ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED ) { > + if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { > DAT_RETURN status; > - status = dapls_ib_qp_alloc( ep_ptr->header.owner_ia, > - ep_ptr, ep_ptr ); > - if ( status != DAT_SUCCESS ) > + status = dapls_ib_qp_alloc(ep_ptr->header.owner_ia, > + ep_ptr, ep_ptr); > + if (status != DAT_SUCCESS) > return status; > } > - > - return ( dapli_socket_accept_final(ep_ptr, cr_ptr, p_size, > p_data) ); > + return(dapli_socket_accept_usr(ep_ptr, cr_ptr, p_size, p_data)); > } > > > @@ -759,19 +956,29 @@ dapls_ib_accept_connection ( > DAT_RETURN > dapls_ib_reject_connection ( > IN ib_cm_handle_t ib_cm_handle, > - IN int reject_reason ) > + IN int reject_reason) > { > ib_cm_srvc_handle_t cm_ptr = ib_cm_handle; > + struct iovec iovec; > > dapl_dbg_log (DAPL_DBG_TYPE_EP, > "dapls_ib_reject_connection(cm_handle %p reason > %x)\n", > - ib_cm_handle, reject_reason ); > - > - /* just close the socket and return */ > - if ( cm_ptr->socket > 0 ) { > - close( cm_ptr->socket ); > + ib_cm_handle, reject_reason); > + > + /* write reject data to indicate reject */ > + if (cm_ptr->socket >= 0) { > + cm_ptr->dst.rej = (uint16_t)reject_reason; > + cm_ptr->dst.rej = htons(cm_ptr->dst.rej); > + iovec.iov_base = &cm_ptr->dst; > + iovec.iov_len = sizeof(ib_qp_cm_t); > + writev(cm_ptr->socket, &iovec, 1); > + close(cm_ptr->socket); > cm_ptr->socket = -1; > } > + > + /* cr_thread will destroy CR */ > + cm_ptr->state = SCM_REJECTED; > + write(g_scm_pipe[1], "w", sizeof "w"); > return DAT_SUCCESS; > } > > @@ -991,24 +1198,25 @@ dapls_ib_get_cm_event ( > return ib_cm_event; > } > > -/* async CR processing thread to avoid blocking applications */ > +/* outbound/inbound CR processing thread to avoid blocking applications > */ > +#define SCM_MAX_CONN 8192 > void cr_thread(void *arg) > { > struct dapl_hca *hca_ptr = arg; > - ib_cm_srvc_handle_t cr, next_cr; > - int max_fd; > - fd_set rfd,rfds; > - struct timeval to; > + ib_cm_handle_t cr, next_cr; > + int ret,idx; > + char rbuf[2]; > + struct pollfd ufds[SCM_MAX_CONN]; > > dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cr_thread: ENTER hca > %p\n",hca_ptr); > > dapl_os_lock( &hca_ptr->ib_trans.lock ); > hca_ptr->ib_trans.cr_state = IB_THREAD_RUN; > while (hca_ptr->ib_trans.cr_state == IB_THREAD_RUN) { > - > - FD_ZERO( &rfds ); > - max_fd = -1; > - > + idx=0; > + ufds[idx].fd = g_scm_pipe[0]; /* wakeup and process work */ > + ufds[idx].events = POLLIN; > + > if (!dapl_llist_is_empty(&hca_ptr->ib_trans.list)) > next_cr = dapl_llist_peek_head (&hca_ptr->ib_trans.list); > else > @@ -1016,51 +1224,70 @@ void cr_thread(void *arg) > > while (next_cr) { > cr = next_cr; > - dapl_dbg_log (DAPL_DBG_TYPE_CM," thread: cm_ptr %p\n", cr ); > - if (cr->l_socket == -1 || > + if ((cr->socket == -1) || > hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) { > > - dapl_dbg_log(DAPL_DBG_TYPE_CM," thread: Freeing %p\n", > cr); > + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: Free %p\n", > cr); > next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list, > > (DAPL_LLIST_ENTRY*)&cr->entry ); > dapl_llist_remove_entry(&hca_ptr->ib_trans.list, > (DAPL_LLIST_ENTRY*)&cr->entry); > - dapl_os_free( cr, sizeof(*cr) ); > + dapl_os_free(cr, sizeof(*cr)); > continue; > } > - > - FD_SET( cr->l_socket, &rfds ); /* add to select set */ > - if ( cr->l_socket > max_fd ) > - max_fd = cr->l_socket; > - > - /* individual select poll to check for work */ > - FD_ZERO(&rfd); > - FD_SET(cr->l_socket, &rfd); > - dapl_os_unlock(&hca_ptr->ib_trans.lock); > - to.tv_sec = 0; > - to.tv_usec = 0; > - if ( select(cr->l_socket + 1,&rfd, NULL, NULL, &to) < 0) { > - dapl_dbg_log (DAPL_DBG_TYPE_CM, > - " thread: ERR %s on cr %p sk %d\n", > - strerror(errno), cr, cr->l_socket); > - close(cr->l_socket); > - cr->l_socket = -1; > - } else if ( FD_ISSET(cr->l_socket, &rfd) && > - dapli_socket_accept(cr)) { > - close(cr->l_socket); > - cr->l_socket = -1; > + > + if (idx==SCM_MAX_CONN-1) { > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + "SCM ERR: cm_thread exceeded FD_SETSIZE > %d\n",idx+1); > + continue; > } > - dapl_os_lock( &hca_ptr->ib_trans.lock ); > + > + /* Add to ufds for poll, check for immediate work */ > + ufds[++idx].fd = cr->socket; /* add listen or cr */ > + ufds[idx].events = POLLIN; > + > + /* check socket for event, accept in or connect out */ > + dapl_dbg_log(DAPL_DBG_TYPE_CM," poll cr=%p, fd=%d,%d\n", > + cr, cr->socket, ufds[idx].fd); > + dapl_os_unlock(&hca_ptr->ib_trans.lock); > + ret = poll(&ufds[idx],1,1); > + dapl_dbg_log(DAPL_DBG_TYPE_CM," poll wakeup ret=%d cr->st=%d > ev=%d fd=%d\n", > + ret,cr->state,ufds[idx].revents,ufds[idx].fd); > + > + /* data on listen, qp exchange, and on disconnect request */ > + if ((ret == 1) && ufds[idx].revents == POLLIN) { > + if (cr->socket > 0) { > + if (cr->state == SCM_LISTEN) > + dapli_socket_accept(cr); > + else if (cr->state == SCM_ACCEPTED) > + dapli_socket_accept_rtu(cr); > + else if (cr->state == SCM_CONN_PENDING) > + dapli_socket_connect_rtu(cr); > + else if (cr->state == SCM_CONNECTED) > + dapli_socket_disconnect(cr); > + } > + } else if (ret != 0) { > + dapl_dbg_log(DAPL_DBG_TYPE_CM, > + " cr_thread(cr=%p) st=%d poll ERR= %s\n", > + cr,cr->state,strerror(errno)); > + /* POLLUP or poll error case, issue event if connected > */ > + if (cr->state == SCM_CONNECTED) > + dapli_socket_disconnect(cr); > + } > + dapl_os_lock(&hca_ptr->ib_trans.lock); > next_cr = dapl_llist_next_entry(&hca_ptr->ib_trans.list, > - > (DAPL_LLIST_ENTRY*)&cr->entry ); > + > (DAPL_LLIST_ENTRY*)&cr->entry); > } > - dapl_os_unlock( &hca_ptr->ib_trans.lock ); > - to.tv_sec = 0; > - to.tv_usec = 100000; /* wakeup and check destroy */ > - select(max_fd + 1, &rfds, NULL, NULL, &to); > - dapl_os_lock( &hca_ptr->ib_trans.lock ); > + dapl_os_unlock(&hca_ptr->ib_trans.lock); > + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: sleep, %d\n", idx+1); > + poll(ufds,idx+1,-1); /* infinite, all sockets and pipe */ > + /* if pipe used to wakeup, consume */ > + if (ufds[0].revents == POLLIN) > + read(g_scm_pipe[0], rbuf, 2); > + dapl_dbg_log(DAPL_DBG_TYPE_CM," cr_thread: wakeup\n"); > + dapl_os_lock(&hca_ptr->ib_trans.lock); > } > - dapl_os_unlock( &hca_ptr->ib_trans.lock ); > + dapl_os_unlock(&hca_ptr->ib_trans.lock); > hca_ptr->ib_trans.cr_state = IB_THREAD_EXIT; > dapl_dbg_log(DAPL_DBG_TYPE_UTIL," cr_thread(hca %p) > exit\n",hca_ptr); > } > diff --git a/dapl/openib_scm/dapl_ib_cq.c b/dapl/openib_scm/dapl_ib_cq.c > index 7ac7037..56b729e 100644 > --- a/dapl/openib_scm/dapl_ib_cq.c > +++ b/dapl/openib_scm/dapl_ib_cq.c > @@ -97,7 +97,7 @@ void dapli_cq_thread_destroy(struct dapl_hca *hca_ptr) > while (hca_ptr->ib_trans.cq_state != IB_THREAD_EXIT) { > struct timespec sleep, remain; > sleep.tv_sec = 0; > - sleep.tv_nsec = 200000000; /* 200 ms */ > + sleep.tv_nsec = 2000000; /* 2 ms */ > dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > " cq_thread_destroy: waiting for > cq_thread\n"); > nanosleep (&sleep, &remain); > @@ -422,12 +422,21 @@ DAT_RETURN dapls_ib_cq_free ( > IN DAPL_IA *ia_ptr, > IN DAPL_EVD *evd_ptr) > { > - if ( evd_ptr->ib_cq_handle != IB_INVALID_HANDLE ) { > - /* copy all entries on CQ to EVD before destroying */ > - dapls_evd_copy_cq(evd_ptr); > + DAT_EVENT event; > + > + if (evd_ptr->ib_cq_handle != IB_INVALID_HANDLE) { > + /* pull off CQ and EVD entries and toss */ > + while (dapls_ib_completion_poll == DAT_SUCCESS); > + while (dapl_evd_dequeue(evd_ptr,&event) != > DAT_QUEUE_EMPTY); > +#if 1 > + ibv_destroy_cq(evd_ptr->ib_cq_handle); > + evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; > + return DAT_SUCCESS; > +#else > if (ibv_destroy_cq(evd_ptr->ib_cq_handle)) > return(dapl_convert_errno(errno,"destroy_cq")); > evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; > +#endif > } > return DAT_SUCCESS; > } > @@ -600,7 +609,7 @@ dapls_ib_wait_object_wait ( > status = ETIMEDOUT; > > dapl_dbg_log (DAPL_DBG_TYPE_CM, > - " cq_object_wait: RET evd %p ibv_cq %p ibv_ctx %p > %s\n", > + " cq_object_wait: RET evd %p ibv_cq %p %s\n", > evd_ptr, ibv_cq,strerror(errno)); > > return(dapl_convert_errno(status,"cq_wait_object_wait")); > diff --git a/dapl/openib_scm/dapl_ib_dto.h > b/dapl/openib_scm/dapl_ib_dto.h > index bea3e4d..b15f347 100644 > --- a/dapl/openib_scm/dapl_ib_dto.h > +++ b/dapl/openib_scm/dapl_ib_dto.h > @@ -35,7 +35,7 @@ > * > * Description: > * > - * The uDAPL openib provider - DTO operations and CQE macros > + * The OpenIB SCM (socket CM) provider - DTO operations and CQE > macros > * > > ************************************************************************ > **** > * Source Control System Information > @@ -61,26 +61,25 @@ STATIC _INLINE_ int > dapls_cqe_opcode(ib_work_completion_t *cqe_p); > */ > STATIC _INLINE_ DAT_RETURN > dapls_ib_post_recv ( > - IN DAPL_EP *ep_ptr, > + IN DAPL_EP *ep_ptr, > IN DAPL_COOKIE *cookie, > - IN DAT_COUNT segments, > + IN DAT_COUNT segments, > IN DAT_LMR_TRIPLET *local_iov ) > { > - ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; > - ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; > - struct ibv_recv_wr wr; > - struct ibv_recv_wr *bad_wr; > - DAT_COUNT i, total_len; > - int ret; > + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; > + ib_data_segment_t *ds_array_p; > + struct ibv_recv_wr wr; > + struct ibv_recv_wr *bad_wr; > + DAT_COUNT i, total_len; > > - dapl_dbg_log (DAPL_DBG_TYPE_EP, > - " post_rcv: ep %p cookie %p segs %d l_iov %p\n", > - ep_ptr, cookie, segments, local_iov); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_rcv: ep %p cookie %p segs %d l_iov %p\n", > + ep_ptr, cookie, segments, local_iov); > > - if ( segments <= DEFAULT_DS_ENTRIES ) > + if (segments <= DEFAULT_DS_ENTRIES) > ds_array_p = ds_array; > else > - ds_array_start_p = ds_array_p = > + ds_array_p = > dapl_os_alloc(segments * > sizeof(ib_data_segment_t)); > > if (NULL == ds_array_p) > @@ -93,18 +92,18 @@ dapls_ib_post_recv ( > wr.wr_id = (uint64_t)(uintptr_t)cookie; > wr.sg_list = ds_array_p; > > - for (i = 0; i < segments; i++ ) { > - if ( !local_iov[i].segment_length ) > + for (i = 0; i < segments; i++) { > + if (!local_iov[i].segment_length) > continue; > > - ds_array_p->addr = (uint64_t) > local_iov[i].virtual_address; > + ds_array_p->addr = (uint64_t) > local_iov[i].virtual_address; > ds_array_p->length = local_iov[i].segment_length; > - ds_array_p->lkey = local_iov[i].lmr_context; > + ds_array_p->lkey = local_iov[i].lmr_context; > > - dapl_dbg_log ( DAPL_DBG_TYPE_EP, > - " post_rcv: l_key 0x%x va %p len %d\n", > - ds_array_p->lkey, ds_array_p->addr, > - ds_array_p->length ); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_rcv: l_key 0x%x va %p len %d\n", > + ds_array_p->lkey, ds_array_p->addr, > + ds_array_p->length ); > > total_len += ds_array_p->length; > wr.num_sge++; > @@ -114,18 +113,12 @@ dapls_ib_post_recv ( > if (cookie != NULL) > cookie->val.dto.size = total_len; > > - ret = ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr); > + if (ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr)) > + return( dapl_convert_errno(errno,"ibv_recv") ); > > - if (ds_array_start_p != NULL) > - dapl_os_free(ds_array_start_p, segments * > sizeof(ib_data_segment_t)); > - > - if (ret) > - return( dapl_convert_errno(EFAULT,"ibv_recv") ); > - > return DAT_SUCCESS; > } > > - > /* > * dapls_ib_post_send > * > @@ -133,35 +126,36 @@ dapls_ib_post_recv ( > */ > STATIC _INLINE_ DAT_RETURN > dapls_ib_post_send ( > - IN DAPL_EP *ep_ptr, > - IN ib_send_op_type_t op_type, > - IN DAPL_COOKIE *cookie, > - IN DAT_COUNT segments, > - IN DAT_LMR_TRIPLET *local_iov, > - IN const DAT_RMR_TRIPLET *remote_iov, > - IN DAT_COMPLETION_FLAGS completion_flags) > + IN DAPL_EP *ep_ptr, > + IN ib_send_op_type_t op_type, > + IN DAPL_COOKIE *cookie, > + IN DAT_COUNT segments, > + IN DAT_LMR_TRIPLET *local_iov, > + IN const DAT_RMR_TRIPLET *remote_iov, > + IN DAT_COMPLETION_FLAGS completion_flags) > { > - dapl_dbg_log (DAPL_DBG_TYPE_EP, > - " post_snd: ep %p op %d ck %p sgs %d l_iov %p > r_iov %p f %d\n", > - ep_ptr, op_type, cookie, segments, local_iov, > - remote_iov, completion_flags); > - > - ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; > - ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; > - struct ibv_send_wr wr; > - struct ibv_send_wr *bad_wr; > - ib_hca_transport_t *ibt_ptr = > &ep_ptr->header.owner_ia->hca_ptr->ib_trans; > - DAT_COUNT i, total_len; > - int ret; > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_snd: ep %p op %d ck %p sgs", > + "%d l_iov %p r_iov %p f %d\n", > + ep_ptr, op_type, cookie, segments, local_iov, > + remote_iov, completion_flags); > + > + ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; > + ib_data_segment_t *ds_array_p; > + struct ibv_send_wr wr; > + struct ibv_send_wr *bad_wr; > + ib_hca_transport_t *ibt_ptr = > + &ep_ptr->header.owner_ia->hca_ptr->ib_trans; > + DAT_COUNT i, total_len; > > - dapl_dbg_log (DAPL_DBG_TYPE_EP, > - " post_snd: ep %p cookie %p segs %d l_iov %p\n", > - ep_ptr, cookie, segments, local_iov); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_snd: ep %p cookie %p segs %d l_iov %p\n", > + ep_ptr, cookie, segments, local_iov); > > - if( segments <= DEFAULT_DS_ENTRIES ) > + if(segments <= DEFAULT_DS_ENTRIES) > ds_array_p = ds_array; > else > - ds_array_start_p = ds_array_p = > + ds_array_p = > dapl_os_alloc(segments * > sizeof(ib_data_segment_t)); > > if (NULL == ds_array_p) > @@ -180,14 +174,14 @@ dapls_ib_post_send ( > if ( !local_iov[i].segment_length ) > continue; > > - ds_array_p->addr = (uint64_t) > local_iov[i].virtual_address; > + ds_array_p->addr = (uint64_t) > local_iov[i].virtual_address; > ds_array_p->length = local_iov[i].segment_length; > - ds_array_p->lkey = local_iov[i].lmr_context; > + ds_array_p->lkey = local_iov[i].lmr_context; > > - dapl_dbg_log ( DAPL_DBG_TYPE_EP, > - " post_snd: lkey 0x%x va %p len %d \n", > - ds_array_p->lkey, ds_array_p->addr, > - ds_array_p->length ); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_snd: lkey 0x%x va %p len %d\n", > + ds_array_p->lkey, ds_array_p->addr, > + ds_array_p->length ); > > total_len += ds_array_p->length; > wr.num_sge++; > @@ -196,20 +190,21 @@ dapls_ib_post_send ( > > if (cookie != NULL) > cookie->val.dto.size = total_len; > - > + > if ((op_type == OP_RDMA_WRITE) || (op_type == OP_RDMA_READ)) { > wr.wr.rdma.remote_addr = remote_iov->target_address; > wr.wr.rdma.rkey = remote_iov->rmr_context; > - dapl_dbg_log ( DAPL_DBG_TYPE_EP, > - " post_snd_rdma: rkey 0x%x va > %#016Lx\n", > - wr.wr.rdma.rkey, wr.wr.rdma.remote_addr > ); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_snd_rdma: rkey 0x%x va %#016Lx\n", > + wr.wr.rdma.rkey, wr.wr.rdma.remote_addr); > } > > + > /* inline data for send or write ops */ > - if ((total_len <= ibt_ptr->max_inline_send ) && > + if ((total_len <= ibt_ptr->max_inline_send) && > ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE))) > wr.send_flags |= IBV_SEND_INLINE; > - > + > /* set completion flags in work request */ > wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG & > completion_flags) ? 0 : > IBV_SEND_SIGNALED; > @@ -218,24 +213,19 @@ dapls_ib_post_send ( > wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG & > completion_flags) ? IBV_SEND_SOLICITED : > 0; > > - dapl_dbg_log (DAPL_DBG_TYPE_EP, > - " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", > - wr.opcode, wr.send_flags, wr.sg_list, > wr.num_sge); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", > + wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); > > - ret = ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr); > + if (ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr)) > + return( dapl_convert_errno(errno,"ibv_recv") ); > > - if (ds_array_start_p != NULL) > - dapl_os_free(ds_array_start_p, segments * > sizeof(ib_data_segment_t)); > - > - if (ret) > - return( dapl_convert_errno(EFAULT,"ibv_send") ); > - > - dapl_dbg_log (DAPL_DBG_TYPE_EP," post_snd: returned\n"); > + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); > return DAT_SUCCESS; > } > > STATIC _INLINE_ DAT_RETURN > -dapls_ib_optional_prv_dat ( > +dapls_ib_optional_prv_dat( > IN DAPL_CR *cr_ptr, > IN const void *event_data, > OUT DAPL_CR **cr_pp) > @@ -243,34 +233,68 @@ dapls_ib_optional_prv_dat ( > return DAT_SUCCESS; > } > > +/* map Work Completions to DAPL WR operations */ > STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) > { > - switch (cqe_p->opcode) { > + switch (cqe_p->opcode) { > case IBV_WC_SEND: > - return (OP_SEND); > + return (OP_SEND); > case IBV_WC_RDMA_WRITE: > - return (OP_RDMA_WRITE); > + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) > + return (OP_RDMA_WRITE_IMM); > + else > + return (OP_RDMA_WRITE); > case IBV_WC_RDMA_READ: > - return (OP_RDMA_READ); > + return (OP_RDMA_READ); > case IBV_WC_COMP_SWAP: > - return (OP_COMP_AND_SWAP); > + return (OP_COMP_AND_SWAP); > case IBV_WC_FETCH_ADD: > - return (OP_FETCH_AND_ADD); > + return (OP_FETCH_AND_ADD); > case IBV_WC_BIND_MW: > - return (OP_BIND_MW); > + return (OP_BIND_MW); > case IBV_WC_RECV: > - return (OP_RECEIVE); > + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) > + return (OP_RECEIVE_IMM); > + else > + return (OP_RECEIVE); > case IBV_WC_RECV_RDMA_WITH_IMM: > - return (OP_RECEIVE_IMM); > + return (OP_RECEIVE_IMM); > default: > - return (OP_INVALID); > - } > + return (OP_INVALID); > + } > +} > + > +#define DAPL_GET_CQE_OPTYPE(cqe_p) dapls_cqe_opcode(cqe_p) > +#define DAPL_GET_CQE_WRID(cqe_p) ((ib_work_completion_t*)cqe_p)->wr_id > +#define DAPL_GET_CQE_STATUS(cqe_p) > ((ib_work_completion_t*)cqe_p)->status > +#define DAPL_GET_CQE_BYTESNUM(cqe_p) > ((ib_work_completion_t*)cqe_p)->byte_len > +#define DAPL_GET_CQE_IMMED_DATA(cqe_p) > ((ib_work_completion_t*)cqe_p)->imm_data > +#define DAPL_GET_CQE_VENDOR_ERR(cqe_p) > ((ib_work_completion_t*)cqe_p)->vendor_err > + > +STATIC _INLINE_ char * dapls_dto_op_str(int op) > +{ > + static char *optable[] = > + { > + "OP_RDMA_WRITE", > + "OP_RDMA_WRITE_IMM", > + "OP_SEND", > + "OP_SEND_IMM", > + "OP_RDMA_READ", > + "OP_COMP_AND_SWAP", > + "OP_FETCH_AND_ADD", > + "OP_RECEIVE", > + "OP_RECEIVE_IMM", > + "OP_BIND_MW", > + 0 > + }; > + return ((op < 0 || op > 9) ? "Invalid CQE OP?" : optable[op]); > } > > -#define DAPL_GET_CQE_OPTYPE(cqe_p) dapls_cqe_opcode(cqe_p) > -#define DAPL_GET_CQE_WRID(cqe_p) > ((ib_work_completion_t*)cqe_p)->wr_id > -#define DAPL_GET_CQE_STATUS(cqe_p) > ((ib_work_completion_t*)cqe_p)->status > -#define DAPL_GET_CQE_BYTESNUM(cqe_p) > ((ib_work_completion_t*)cqe_p)->byte_len > -#define DAPL_GET_CQE_IMMED_DATA(cqe_p) > ((ib_work_completion_t*)cqe_p)->imm_data > +static _INLINE_ char * > +dapls_cqe_op_str(IN ib_work_completion_t *cqe_ptr) > +{ > + return dapls_dto_op_str(DAPL_GET_CQE_OPTYPE(cqe_ptr)); > +} > +#define DAPL_GET_CQE_OP_STR(cqe) dapls_cqe_op_str(cqe) > > #endif /* _DAPL_IB_DTO_H_ */ > diff --git a/dapl/openib_scm/dapl_ib_mem.c > b/dapl/openib_scm/dapl_ib_mem.c > index de36c0f..6a5e4a2 100644 > --- a/dapl/openib_scm/dapl_ib_mem.c > +++ b/dapl/openib_scm/dapl_ib_mem.c > @@ -1,4 +1,6 @@ > /* > + * Copyright (c) 2005-2007 Intel Corporation. All rights reserved. > + * > * This Software is licensed under one of the following licenses: > * > * 1) under the terms of the "Common Public License 1.0" a copy of > which is > @@ -25,12 +27,11 @@ > > /********************************************************************** > * > - * MODULE: dapl_det_mem.c > + * MODULE: dapl_ib_mem.c > * > - * PURPOSE: Intel DET APIs: Memory windows, registration, > - * and protection domain > + * PURPOSE: Memory windows, registration, and protection domain > * > - * $Id: $ > + * $Id:$ > * > > **********************************************************************/ > > @@ -61,8 +62,7 @@ > * > */ > STATIC _INLINE_ int > -dapls_convert_privileges ( > - IN DAT_MEM_PRIV_FLAGS privileges) > +dapls_convert_privileges(IN DAT_MEM_PRIV_FLAGS privileges) > { > int access = 0; > > @@ -101,16 +101,15 @@ dapls_convert_privileges ( > * > */ > DAT_RETURN > -dapls_ib_pd_alloc ( > - IN DAPL_IA *ia_ptr, > - IN DAPL_PZ *pz ) > +dapls_ib_pd_alloc(IN DAPL_IA *ia_ptr, IN DAPL_PZ *pz) > { > /* get a protection domain */ > pz->pd_handle = ibv_alloc_pd(ia_ptr->hca_ptr->ib_hca_handle); > if (!pz->pd_handle) > return(dapl_convert_errno(ENOMEM,"alloc_pd")); > > - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " pd_alloc: pd_handle=%p\n", > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > + " pd_alloc: pd_handle=%p\n", > pz->pd_handle ); > > return DAT_SUCCESS; > @@ -134,13 +133,18 @@ dapls_ib_pd_alloc ( > * > */ > DAT_RETURN > -dapls_ib_pd_free ( > - IN DAPL_PZ *pz ) > +dapls_ib_pd_free(IN DAPL_PZ *pz ) > { > if (pz->pd_handle != IB_INVALID_HANDLE) { > +#if 1 > + ibv_dealloc_pd(pz->pd_handle); > + pz->pd_handle = IB_INVALID_HANDLE; > + return DAT_SUCCESS; > +#else > if (ibv_dealloc_pd(pz->pd_handle)) > return(dapl_convert_errno(errno,"dealloc_pd")); > pz->pd_handle = IB_INVALID_HANDLE; > +#endif > } > return DAT_SUCCESS; > } > @@ -165,25 +169,24 @@ dapls_ib_pd_free ( > * > */ > DAT_RETURN > -dapls_ib_mr_register ( > - IN DAPL_IA *ia_ptr, > - IN DAPL_LMR *lmr, > - IN DAT_PVOID virt_addr, > - IN DAT_VLEN length, > - IN DAT_MEM_PRIV_FLAGS privileges) > +dapls_ib_mr_register(IN DAPL_IA *ia_ptr, > + IN DAPL_LMR *lmr, > + IN DAT_PVOID virt_addr, > + IN DAT_VLEN length, > + IN DAT_MEM_PRIV_FLAGS privileges) > { > ib_pd_handle_t ib_pd_handle; > > ib_pd_handle = ((DAPL_PZ *)lmr->param.pz_handle)->pd_handle; > > - dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, > - " mr_register: ia=%p, lmr=%p va=%p ln=%d > pv=0x%x\n", > - ia_ptr, lmr, virt_addr, length, privileges ); > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > + " mr_register: ia=%p, lmr=%p va=%p ln=%d > pv=0x%x\n", > + ia_ptr, lmr, virt_addr, length, privileges ); > > /* TODO: shared memory */ > if (lmr->param.mem_type == DAT_MEM_TYPE_SHARED_VIRTUAL) { > - dapl_dbg_log( DAPL_DBG_TYPE_ERR, > - " mr_register_shared: NOT IMPLEMENTED\n"); > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + " mr_register_shared: NOT IMPLEMENTED\n"); > > return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > > } > > @@ -200,16 +203,16 @@ dapls_ib_mr_register ( > lmr->param.lmr_context = lmr->mr_handle->lkey; > lmr->param.rmr_context = lmr->mr_handle->rkey; > lmr->param.registered_size = length; > - lmr->param.registered_address = (DAT_VADDR)(uintptr_t) > virt_addr; > + lmr->param.registered_address = (DAT_VADDR)(uintptr_t)virt_addr; > > - dapl_dbg_log ( DAPL_DBG_TYPE_UTIL, > - " mr_register: mr=%p h %x pd %p ctx %p > ,lkey=0x%x, rkey=0x%x priv=%x\n", > - lmr->mr_handle, lmr->mr_handle->handle, > - lmr->mr_handle->pd, > - lmr->mr_handle->context, > - lmr->mr_handle->lkey, > - lmr->mr_handle->rkey, > - length, dapls_convert_privileges(privileges) ); > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > + " mr_register: mr=%p addr=%p h %x pd %p ctx %p " > + "lkey=0x%x rkey=0x%x priv=%x\n", > + lmr->mr_handle, lmr->mr_handle->addr, > + lmr->mr_handle->handle, > + lmr->mr_handle->pd, lmr->mr_handle->context, > + lmr->mr_handle->lkey, lmr->mr_handle->rkey, > + length, dapls_convert_privileges(privileges)); > > return DAT_SUCCESS; > } > @@ -231,8 +234,7 @@ dapls_ib_mr_register ( > * > */ > DAT_RETURN > -dapls_ib_mr_deregister ( > - IN DAPL_LMR *lmr ) > +dapls_ib_mr_deregister(IN DAPL_LMR *lmr) > { > if (lmr->mr_handle != IB_INVALID_HANDLE) { > if (ibv_dereg_mr(lmr->mr_handle)) > @@ -263,13 +265,14 @@ dapls_ib_mr_deregister ( > * > */ > DAT_RETURN > -dapls_ib_mr_register_shared ( > - IN DAPL_IA *ia_ptr, > - IN DAPL_LMR *lmr, > - IN DAT_MEM_PRIV_FLAGS privileges ) > +dapls_ib_mr_register_shared(IN DAPL_IA *ia_ptr, > + IN DAPL_LMR *lmr, > + IN DAT_MEM_PRIV_FLAGS privileges) > { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mr_register_shared: NOT > IMPLEMENTED\n"); > - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + " mr_register_shared: NOT IMPLEMENTED\n"); > + > + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > } > > /* > @@ -289,12 +292,13 @@ dapls_ib_mr_register_shared ( > * > */ > DAT_RETURN > -dapls_ib_mw_alloc ( > - IN DAPL_RMR *rmr ) > +dapls_ib_mw_alloc (IN DAPL_RMR *rmr) > { > > - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_alloc: NOT IMPLEMENTED\n"); > - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + " mw_alloc: NOT IMPLEMENTED\n"); > + > + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > } > > /* > @@ -314,11 +318,12 @@ dapls_ib_mw_alloc ( > * > */ > DAT_RETURN > -dapls_ib_mw_free ( > - IN DAPL_RMR *rmr ) > +dapls_ib_mw_free(IN DAPL_RMR *rmr) > { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_free: NOT IMPLEMENTED\n"); > - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + " mw_free: NOT IMPLEMENTED\n"); > + > + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > } > > /* > @@ -339,17 +344,18 @@ dapls_ib_mw_free ( > * > */ > DAT_RETURN > -dapls_ib_mw_bind ( > - IN DAPL_RMR *rmr, > - IN DAPL_LMR *lmr, > - IN DAPL_EP *ep, > - IN DAPL_COOKIE *cookie, > - IN DAT_VADDR virtual_address, > - IN DAT_VLEN length, > - IN DAT_MEM_PRIV_FLAGS mem_priv, > - IN DAT_BOOLEAN is_signaled) > +dapls_ib_mw_bind(IN DAPL_RMR *rmr, > + IN DAPL_LMR *lmr, > + IN DAPL_EP *ep, > + IN DAPL_COOKIE *cookie, > + IN DAT_VADDR virtual_address, > + IN DAT_VLEN length, > + IN DAT_MEM_PRIV_FLAGS mem_priv, > + IN DAT_BOOLEAN is_signaled) > { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_bind: NOT IMPLEMENTED\n"); > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + " mw_bind: NOT IMPLEMENTED\n"); > + > return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > } > > @@ -372,14 +378,15 @@ dapls_ib_mw_bind ( > * > */ > DAT_RETURN > -dapls_ib_mw_unbind ( > - IN DAPL_RMR *rmr, > - IN DAPL_EP *ep, > - IN DAPL_COOKIE *cookie, > - IN DAT_BOOLEAN is_signaled ) > +dapls_ib_mw_unbind(IN DAPL_RMR *rmr, > + IN DAPL_EP *ep, > + IN DAPL_COOKIE *cookie, > + IN DAT_BOOLEAN is_signaled ) > { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR," mw_unbind: NOT IMPLEMENTED\n"); > - return DAT_ERROR (DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > + dapl_dbg_log(DAPL_DBG_TYPE_ERR, > + " mw_unbind: NOT IMPLEMENTED\n"); > + > + return DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > } > > /* > diff --git a/dapl/openib_scm/dapl_ib_qp.c b/dapl/openib_scm/dapl_ib_qp.c > index 3a1e3c8..1eba2bd 100644 > --- a/dapl/openib_scm/dapl_ib_qp.c > +++ b/dapl/openib_scm/dapl_ib_qp.c > @@ -110,15 +110,23 @@ dapls_ib_qp_alloc ( > /* Setup attributes and create qp */ > dapl_os_memzero((void*)&qp_create, sizeof(qp_create)); > qp_create.send_cq = req_cq; > - qp_create.recv_cq = rcv_cq; > qp_create.cap.max_send_wr = attr->max_request_dtos; > - qp_create.cap.max_recv_wr = attr->max_recv_dtos; > qp_create.cap.max_send_sge = attr->max_request_iov; > - qp_create.cap.max_recv_sge = attr->max_recv_iov; > qp_create.cap.max_inline_data = > ia_ptr->hca_ptr->ib_trans.max_inline_send; > qp_create.qp_type = IBV_QPT_RC; > qp_create.qp_context = (void*)ep_ptr; > > + /* ibv assumes rcv_cq is never NULL, set to req_cq */ > + if (rcv_cq == NULL) { > + qp_create.recv_cq = req_cq; > + qp_create.cap.max_recv_wr = 0; > + qp_create.cap.max_recv_sge = 0; > + } else { > + qp_create.recv_cq = rcv_cq; > + qp_create.cap.max_recv_wr = attr->max_recv_dtos; > + qp_create.cap.max_recv_sge = attr->max_recv_iov; > + } > + > ep_ptr->qp_handle = ibv_create_qp( ib_pd_handle, &qp_create); > if (!ep_ptr->qp_handle) > return(dapl_convert_errno(ENOMEM, "create_qp")); > @@ -298,9 +306,10 @@ dapls_modify_qp_state ( IN ib_qp_handle_t > qp_handle, > IN ib_qp_state_t qp_state, > IN ib_qp_cm_t *qp_cm ) > { > - struct ibv_qp_attr qp_attr; > + struct ibv_qp_attr qp_attr; > enum ibv_qp_attr_mask mask = IBV_QP_STATE; > - DAPL_EP *ep_ptr = (DAPL_EP*)qp_handle->qp_context; > + DAPL_EP *ep_ptr = > (DAPL_EP*)qp_handle->qp_context; > + DAPL_IA *ia_ptr = ep_ptr->header.owner_ia; > > dapl_os_memzero((void*)&qp_attr, sizeof(qp_attr)); > qp_attr.qp_state = qp_state; > @@ -315,14 +324,16 @@ dapls_modify_qp_state ( IN ib_qp_handle_t > qp_handle, > IBV_QP_RQ_PSN | > IBV_QP_MAX_DEST_RD_ATOMIC | > IBV_QP_MIN_RNR_TIMER; > + > qp_attr.qp_state = IBV_QPS_RTR; > - qp_attr.path_mtu = IBV_MTU_1024; > + qp_attr.path_mtu = IBV_MTU_2048; > qp_attr.dest_qp_num = qp_cm->qpn; > qp_attr.rq_psn = 1; > qp_attr.max_dest_rd_atomic = > ep_ptr->param.ep_attr.max_rdma_read_out; > - qp_attr.min_rnr_timer = 12; > + qp_attr.min_rnr_timer = > ia_ptr->hca_ptr->ib_trans.rnr_timer; > qp_attr.ah_attr.is_global = 0; > + qp_attr.ah_attr.grh.dgid = qp_cm->gid; > qp_attr.ah_attr.dlid = qp_cm->lid; > qp_attr.ah_attr.sl = 0; > qp_attr.ah_attr.src_path_bits = 0; > @@ -343,30 +354,25 @@ dapls_modify_qp_state ( IN ib_qp_handle_t > qp_handle, > IBV_QP_RNR_RETRY | > IBV_QP_SQ_PSN | > IBV_QP_MAX_QP_RD_ATOMIC; > + > qp_attr.qp_state = IBV_QPS_RTS; > - qp_attr.timeout = 14; > - qp_attr.retry_cnt = 7; > - qp_attr.rnr_retry = 7; > + qp_attr.timeout = > ia_ptr->hca_ptr->ib_trans.ack_timer; > + qp_attr.retry_cnt = > ia_ptr->hca_ptr->ib_trans.ack_retry; > + qp_attr.rnr_retry = > ia_ptr->hca_ptr->ib_trans.rnr_retry; > qp_attr.sq_psn = 1; > qp_attr.max_rd_atomic = > ep_ptr->param.ep_attr.max_rdma_read_out; > > - dapl_dbg_log (DAPL_DBG_TYPE_EP, > - " modify_qp_rts: psn %x rd_atomic %d\n", > - qp_attr.sq_psn, qp_attr.max_rd_atomic ); > + dapl_dbg_log(DAPL_DBG_TYPE_EP, > + " modify_qp_rts: psn %x rd_atomic %d ack > %d " > + " retry %d rnr_retry %d\n", > + qp_attr.sq_psn, qp_attr.max_rd_atomic, > + qp_attr.timeout, qp_attr.retry_cnt, > + qp_attr.rnr_retry ); > break; > } > case IBV_QPS_INIT: > { > - DAPL_IA *ia_ptr; > - DAPL_EP *ep_ptr; > - /* need to find way back to port num */ > - ep_ptr = (DAPL_EP*)qp_handle->qp_context; > - if (ep_ptr) > - ia_ptr = ep_ptr->header.owner_ia; > - else > - break; > - > mask |= IBV_QP_PKEY_INDEX | > IBV_QP_PORT | > IBV_QP_ACCESS_FLAGS; > @@ -377,7 +383,8 @@ dapls_modify_qp_state ( IN ib_qp_handle_t > qp_handle, > IBV_ACCESS_LOCAL_WRITE | > IBV_ACCESS_REMOTE_WRITE | > IBV_ACCESS_REMOTE_READ | > - IBV_ACCESS_REMOTE_ATOMIC; > + IBV_ACCESS_REMOTE_ATOMIC | > + IBV_ACCESS_MW_BIND; > > dapl_dbg_log (DAPL_DBG_TYPE_EP, > " modify_qp_init: pi %x port %x acc > %x\n", > diff --git a/dapl/openib_scm/dapl_ib_util.c > b/dapl/openib_scm/dapl_ib_util.c > index 5e34b47..a9941f5 100644 > --- a/dapl/openib_scm/dapl_ib_util.c > +++ b/dapl/openib_scm/dapl_ib_util.c > @@ -62,6 +62,7 @@ static const char rcsid[] = "$Id: $"; > #include > > int g_dapl_loopback_connection = 0; > +int g_scm_pipe[2]; > > /* just get IP address for hostname */ > DAT_RETURN getipaddr( char *addr, int addr_len) > @@ -70,14 +71,14 @@ DAT_RETURN getipaddr( char *addr, int addr_len) > struct hostent *h_ptr; > struct utsname ourname; > > - if ( uname( &ourname ) < 0 ) > + if (uname( &ourname ) < 0) > return DAT_INTERNAL_ERROR; > > - h_ptr = gethostbyname( ourname.nodename ); > - if ( h_ptr == NULL ) > + h_ptr = gethostbyname(ourname.nodename); > + if (h_ptr == NULL) > return DAT_INTERNAL_ERROR; > > - if ( h_ptr->h_addrtype == AF_INET ) { > + if (h_ptr->h_addrtype == AF_INET) { > int i; > struct in_addr **alist = > (struct in_addr **)h_ptr->h_addr_list; > @@ -87,18 +88,17 @@ DAT_RETURN getipaddr( char *addr, int addr_len) > > /* Walk the list of addresses for host */ > for (i=0; alist[i] != NULL; i++) { > - > - /* first non-loopback address */ > > - if ( *(uint32_t*)alist[i] != htonl(0x7f000001) ) > { > - dapl_os_memcpy( &ipv4_addr->sin_addr, > - h_ptr->h_addr_list[i], > - 4 ); > + /* first non-loopback address */ > + if (*(uint32_t*)alist[i] != htonl(0x7f000001)) { > + dapl_os_memcpy(&ipv4_addr->sin_addr, > + h_ptr->h_addr_list[i], > + 4); > break; > } > } > /* if no acceptable address found */ > if (*(uint32_t*)&ipv4_addr->sin_addr == 0) > - return DAT_INVALID_ADDRESS; > + return DAT_INVALID_ADDRESS; > } else > return DAT_INVALID_ADDRESS; > > @@ -122,6 +122,10 @@ DAT_RETURN getipaddr( char *addr, int addr_len) > */ > int32_t dapls_ib_init (void) > { > + /* create pipe for waking up thread */ > + if (pipe(g_scm_pipe)) > + return 1; > + > return 0; > } > > @@ -156,7 +160,7 @@ DAT_RETURN dapls_ib_open_hca ( > int i; > DAT_RETURN dat_status = DAT_SUCCESS; > > - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > " open_hca: %s - %p\n", hca_name, hca_ptr ); > > /* Get list of all IB devices, find match, open */ > @@ -170,65 +174,83 @@ DAT_RETURN dapls_ib_open_hca ( > > for (i = 0; dev_list[i]; ++i) { > hca_ptr->ib_trans.ib_dev = dev_list[i]; > - if > (!strcmp(ibv_get_device_name(hca_ptr->ib_trans.ib_dev),hca_name)) > + if > (!strcmp(ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > + hca_name)) > goto found; > } > > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: IB device %s not found\n", > - hca_name); > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: device %s not found\n", > + hca_name); > goto err; > > found: > - dapl_dbg_log (DAPL_DBG_TYPE_UTIL," open_hca: Found dev %s > %016llx\n", > - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > - (unsigned long > long)bswap_64(ibv_get_device_guid(hca_ptr->ib_trans.ib_dev))); > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL," open_hca: Found dev %s > %016llx\n", > + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > + (unsigned long long) > + > bswap_64(ibv_get_device_guid(hca_ptr->ib_trans.ib_dev))); > > hca_ptr->ib_hca_handle = > ibv_open_device(hca_ptr->ib_trans.ib_dev); > if (!hca_ptr->ib_hca_handle) { > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: IB dev open failed for > %s\n", > - > ibv_get_device_name(hca_ptr->ib_trans.ib_dev) ); > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: dev open failed for %s, err=%s\n", > > + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > + strerror(errno)); > goto err; > } > > - /* set inline max with enviroment or default */ > + /* set RC tunables via enviroment or default */ > hca_ptr->ib_trans.max_inline_send = > - dapl_os_get_env_val ( "DAPL_MAX_INLINE", > INLINE_SEND_DEFAULT ); > + dapl_os_get_env_val("DAPL_MAX_INLINE", > INLINE_SEND_DEFAULT); > + hca_ptr->ib_trans.ack_retry = > + dapl_os_get_env_val("DAPL_ACK_RETRY", SCM_ACK_RETRY); > + hca_ptr->ib_trans.ack_timer = > + dapl_os_get_env_val("DAPL_ACK_TIMER", SCM_ACK_TIMER); > + hca_ptr->ib_trans.rnr_retry = > + dapl_os_get_env_val("DAPL_RNR_RETRY", SCM_RNR_RETRY); > + hca_ptr->ib_trans.rnr_timer = > + dapl_os_get_env_val("DAPL_RNR_TIMER", SCM_RNR_TIMER); > > /* initialize cq_lock */ > dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.cq_lock); > - if (dat_status != DAT_SUCCESS) > - { > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: failed to init cq_lock\n"); > + if (dat_status != DAT_SUCCESS) { > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: failed to init cq_lock\n"); > goto bail; > } > > /* EVD events without direct CQ channels, non-blocking */ > hca_ptr->ib_trans.ib_cq = > ibv_create_comp_channel(hca_ptr->ib_hca_handle); > + if (hca_ptr->ib_trans.ib_cq == NULL) { > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: ibv_create_comp_channel ERR %s\n", > + strerror(errno)); > + goto bail; > + } > + > opts = fcntl(hca_ptr->ib_trans.ib_cq->fd, F_GETFL); /* uCQ */ > if (opts < 0 || fcntl(hca_ptr->ib_trans.ib_cq->fd, > F_SETFL, opts | O_NONBLOCK) < 0) { > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: ERR with CQ FD\n" ); > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: fcntl on ib_cq->fd %d ERR %d %s\n", > > + hca_ptr->ib_trans.ib_cq->fd, opts, > + strerror(errno)); > goto bail; > } > > if (dapli_cq_thread_init(hca_ptr)) { > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: cq_thread_init failed for > %s\n", > - > ibv_get_device_name(hca_ptr->ib_trans.ib_dev) ); > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: cq_thread_init failed for %s\n", > + > ibv_get_device_name(hca_ptr->ib_trans.ib_dev)); > goto bail; > } > > /* initialize cr_list lock */ > dat_status = dapl_os_lock_init(&hca_ptr->ib_trans.lock); > - if (dat_status != DAT_SUCCESS) > - { > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: failed to init lock\n"); > + if (dat_status != DAT_SUCCESS) { > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: failed to init cr_list lock\n"); > goto bail; > } > > @@ -240,10 +262,9 @@ found: > dat_status = dapl_os_thread_create(cr_thread, > (void*)hca_ptr, > &hca_ptr->ib_trans.thread ); > - if (dat_status != DAT_SUCCESS) > - { > - dapl_dbg_log (DAPL_DBG_TYPE_ERR, > - " open_hca: failed to create thread\n"); > + if (dat_status != DAT_SUCCESS) { > + dapl_log(DAPL_DBG_TYPE_ERR, > + " open_hca: failed to create thread\n"); > goto bail; > } > > @@ -251,7 +272,7 @@ found: > while (hca_ptr->ib_trans.cr_state != IB_THREAD_RUN) { > struct timespec sleep, remain; > sleep.tv_sec = 0; > - sleep.tv_nsec = 20000000; /* 20 ms */ > + sleep.tv_nsec = 2000000; /* 2 ms */ > dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > " open_hca: waiting for cr_thread\n"); > nanosleep (&sleep, &remain); > @@ -259,16 +280,15 @@ found: > > /* get the IP address of the device */ > dat_status = getipaddr((char*)&hca_ptr->hca_address, > - sizeof(DAT_SOCK_ADDR6) ); > - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, > - " open_hca: %s, port %d, %s %d.%d.%d.%d\n", > - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > hca_ptr->port_num, > - ((struct sockaddr_in > *)&hca_ptr->hca_address)->sin_family == AF_INET ? "AF_INET":"AF_INET6", > - ((struct sockaddr_in > *)&hca_ptr->hca_address)->sin_addr.s_addr >> 0 & 0xff, > - ((struct sockaddr_in > *)&hca_ptr->hca_address)->sin_addr.s_addr >> 8 & 0xff, > - ((struct sockaddr_in > *)&hca_ptr->hca_address)->sin_addr.s_addr >> 16 & 0xff, > - ((struct sockaddr_in > *)&hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff ); > - > + sizeof(DAT_SOCK_ADDR6)); > + > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > + " open_hca: devname %s, port %d, hostname_IP %s\n", > > + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > + hca_ptr->port_num, > + inet_ntoa(((struct sockaddr_in *) > + &hca_ptr->hca_address)->sin_addr)); > + > ibv_free_device_list(dev_list); > return dat_status; > > @@ -308,15 +328,15 @@ DAT_RETURN dapls_ib_close_hca ( IN DAPL_HCA > *hca_ptr ) > > return(dapl_convert_errno(errno,"ib_close_device")); > hca_ptr->ib_hca_handle = IB_INVALID_HANDLE; > } > - > dapl_os_lock_destroy(&hca_ptr->ib_trans.cq_lock); > > /* destroy cr_thread and lock */ > hca_ptr->ib_trans.cr_state = IB_THREAD_CANCEL; > + write(g_scm_pipe[1], "w", sizeof "w"); > while (hca_ptr->ib_trans.cr_state != IB_THREAD_EXIT) { > struct timespec sleep, remain; > sleep.tv_sec = 0; > - sleep.tv_nsec = 20000000; /* 20 ms */ > + sleep.tv_nsec = 2000000; /* 2 ms */ > dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > " close_hca: waiting for cr_thread\n"); > nanosleep (&sleep, &remain); > @@ -378,14 +398,11 @@ DAT_RETURN dapls_ib_query_hca ( > ia_attr->vendor_name[DAT_NAME_MAX_LENGTH - 1] = '\0'; > ia_attr->ia_address_ptr = > (DAT_IA_ADDRESS_PTR)&hca_ptr->hca_address; > > - dapl_dbg_log (DAPL_DBG_TYPE_UTIL, > - " query_hca: %s %s %d.%d.%d.%d\n", > - ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > - ((struct sockaddr_in > *)ia_attr->ia_address_ptr)->sin_family == AF_INET ? > "AF_INET":"AF_INET6", > - ((struct sockaddr_in > *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 0 & 0xff, > - ((struct sockaddr_in > *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 8 & 0xff, > - ((struct sockaddr_in > *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 16 & 0xff, > - ((struct sockaddr_in > *)ia_attr->ia_address_ptr)->sin_addr.s_addr >> 24 & 0xff ); > + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, > + " query_hca: %s %s \n", > + > ibv_get_device_name(hca_ptr->ib_trans.ib_dev), > + inet_ntoa(((struct sockaddr_in *) > + > &hca_ptr->hca_address)->sin_addr)); > > ia_attr->hardware_version_major = dev_attr.hw_ver; > /* ia_attr->hardware_version_minor = dev_attr.fw_ver; > */ > @@ -408,11 +425,14 @@ DAT_RETURN dapls_ib_query_hca ( > ia_attr->max_pzs = dev_attr.max_pd; > ia_attr->max_mtu_size = > port_attr.max_msg_sz; > ia_attr->max_rdma_size = > port_attr.max_msg_sz; > + ia_attr->max_iov_segments_per_rdma_read = > dev_attr.max_sge; > + ia_attr->max_iov_segments_per_rdma_write = > dev_attr.max_sge; > ia_attr->num_transport_attr = 0; > ia_attr->transport_attr = NULL; > ia_attr->num_vendor_attr = 0; > ia_attr->vendor_attr = NULL; > - > + hca_ptr->ib_trans.ack_timer = > DAPL_MAX(dev_attr.local_ca_ack_delay, > + > hca_ptr->ib_trans.ack_timer); > dapl_dbg_log (DAPL_DBG_TYPE_UTIL, > " query_hca: (%x.%x) ep %d ep_q %d evd %d evd_q > %d\n", > ia_attr->hardware_version_major, > @@ -420,11 +440,10 @@ DAT_RETURN dapls_ib_query_hca ( > ia_attr->max_eps, ia_attr->max_dto_per_ep, > ia_attr->max_evds, ia_attr->max_evd_qlen ); > dapl_dbg_log (DAPL_DBG_TYPE_UTIL, > - " query_hca: msg %llu rdma %llu iov %d lmr %d > rmr %d\n", > + " query_hca: msg %llu rdma %llu iov %d lmr %d > rmr %d ack_time %d\n", > ia_attr->max_mtu_size, ia_attr->max_rdma_size, > ia_attr->max_iov_segments_per_dto, > ia_attr->max_lmrs, > - ia_attr->max_rmrs ); > - > + ia_attr->max_rmrs,hca_ptr->ib_trans.ack_timer ); > } > > if (ep_attr != NULL) { > @@ -443,7 +462,6 @@ DAT_RETURN dapls_ib_query_hca ( > ep_attr->max_recv_dtos, ep_attr->max_recv_iov, > ep_attr->max_rdma_read_in, > ep_attr->max_rdma_read_out); > } > - > return DAT_SUCCESS; > } > > diff --git a/dapl/openib_scm/dapl_ib_util.h > b/dapl/openib_scm/dapl_ib_util.h > index 0d928df..81a1752 100644 > --- a/dapl/openib_scm/dapl_ib_util.h > +++ b/dapl/openib_scm/dapl_ib_util.h > @@ -71,15 +71,18 @@ typedef ib_hca_handle_t dapl_ibal_ca_t; > > /* CM mappings, user CM not complete use SOCKETS */ > > -/* destination info to exchange until real IB CM shows up */ > +/* destination info to exchange, define wire protocol version */ > +#define DSCM_VER 2 > typedef struct _ib_qp_cm > { > - uint32_t qpn; > + uint16_t ver; > + uint16_t rej; > uint16_t lid; > uint16_t port; > - int p_size; > + uint32_t qpn; > + uint32_t p_size; > DAT_SOCK_ADDR6 ia_address; > - > + union ibv_gid gid; > } ib_qp_cm_t; > > /* > @@ -94,20 +97,34 @@ struct ib_llist_entry > struct dapl_llist_entry *list_head; > }; > > +typedef enum scm_state > +{ > + SCM_INIT, > + SCM_LISTEN, > + SCM_CONN_PENDING, > + SCM_ACCEPTING, > + SCM_ACCEPTED, > + SCM_REJECTED, > + SCM_CONNECTED, > + SCM_DISCONNECTED, > + SCM_DESTROY > +} SCM_STATE; > + > struct ib_cm_handle > { > struct ib_llist_entry entry; > + DAPL_OS_LOCK lock; > + SCM_STATE state; > int socket; > - int l_socket; > - struct dapl_hca *hca_ptr; > - DAT_HANDLE cr; > + struct dapl_hca *hca; > DAT_HANDLE sp; > + struct dapl_ep *ep; > ib_qp_cm_t dst; > unsigned char p_data[256]; > }; > > typedef struct ib_cm_handle *ib_cm_handle_t; > -typedef ib_cm_handle_t ib_cm_srvc_handle_t; > +typedef ib_cm_handle_t ib_cm_srvc_handle_t; > > DAT_RETURN getipaddr(char *addr, int addr_len); > > @@ -163,6 +180,12 @@ typedef struct ibv_comp_channel > *ib_wait_obj_handle_t; > /* inline send rdma threshold */ > #define INLINE_SEND_DEFAULT 128 > > +/* RC timer - retry count defaults */ > +#define SCM_ACK_TIMER 15 /* 5 bits, 4.096us*2^ack_timer. 15 == > 134ms */ > +#define SCM_ACK_RETRY 7 /* 3 bits, 7 * 134ms = 940ms */ > +#define SCM_RNR_TIMER 28 /* 5 bits, 28 == 163ms, 31 == 491ms */ > +#define SCM_RNR_RETRY 7 /* 3 bits, 7 == infinite */ > + > /* CM private data areas */ > #define IB_MAX_REQ_PDATA_SIZE 92 > #define IB_MAX_REP_PDATA_SIZE 196 > @@ -268,7 +291,10 @@ typedef struct _ib_hca_transport > ib_async_cq_handler_t async_cq_error; > ib_async_dto_handler_t async_cq; > ib_async_qp_handler_t async_qp_error; > - > + uint8_t ack_timer; > + uint8_t ack_retry; > + uint8_t rnr_timer; > + uint8_t rnr_retry; > } ib_hca_transport_t; > > /* provider specfic fields for shared memory support */ > -- > 1.5.2.5 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Mon Jun 23 11:37:53 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 11:37:53 -0700 Subject: [ofa-general] [PATCH 5/11] core: XRC base implementation In-Reply-To: <200806231601.35420.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:01:35 +0300") References: <200806231601.35420.jackm@dev.mellanox.co.il> Message-ID: > +struct ib_uverbs_close_xrc_domain { > + __u64 response; > + __u32 xrcd_handle; > + __u64 driver_data[0]; > +}; This creates 32/64 compatibility problems: on a 32-bit architecture, driver_data may only be aligned to 32 bits. We need to insert a __u32 reserved; field after xrcd_handle for alignment. From rdreier at cisco.com Mon Jun 23 11:41:53 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 11:41:53 -0700 Subject: [ofa-general] Re: [PATCH 8/11] core: XRC receive-only QPs In-Reply-To: <200806231601.43706.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:01:43 +0300") References: <200806231601.43706.jackm@dev.mellanox.co.il> Message-ID: > +struct ib_uverbs_create_xrc_rcv_qp { > + __u64 response; > + __u64 user_handle; > + __u32 xrc_domain_handle; > + __u32 max_send_wr; > + __u32 max_recv_wr; > + __u32 max_send_sge; > + __u32 max_recv_sge; > + __u32 max_inline_data; > + __u8 sq_sig_all; > + __u8 qp_type; > + __u8 reserved[2]; > + __u64 driver_data[0]; > +}; same alignment problem -- reserved needs to be [6] not [2] to keep the alignment of driver_data constant. > +struct ib_uverbs_modify_xrc_rcv_qp { > + __u32 xrc_domain_handle; > + __u32 qp_num; > + struct ib_uverbs_qp_dest dest; > + struct ib_uverbs_qp_dest alt_dest; > + __u32 attr_mask; > + __u32 qkey; > + __u32 rq_psn; > + __u32 sq_psn; > + __u32 dest_qp_num; > + __u32 qp_access_flags; > + __u16 pkey_index; > + __u16 alt_pkey_index; > + __u8 qp_state; > + __u8 cur_qp_state; > + __u8 path_mtu; > + __u8 path_mig_state; > + __u8 en_sqd_async_notify; > + __u8 max_rd_atomic; > + __u8 max_dest_rd_atomic; > + __u8 min_rnr_timer; > + __u8 port_num; > + __u8 timeout; > + __u8 retry_cnt; > + __u8 rnr_retry; > + __u8 alt_port_num; > + __u8 alt_timeout; > + __u8 reserved[2]; > + __u64 driver_data[0]; > +}; ditto > +struct ib_uverbs_query_xrc_rcv_qp { > + __u64 response; > + __u32 xrc_domain_handle; > + __u32 qp_num; > + __u32 attr_mask; > + __u64 driver_data[0]; > +}; needs __u32 reserved; after attr_mask. - R. From swise at opengridcomputing.com Mon Jun 23 11:45:47 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 23 Jun 2008 13:45:47 -0500 Subject: [ofa-general] [PATCH for-2.6.27] RDMA/cxgb3: Fix up some of the cxgb3 ib_device_attr fields. Message-ID: <20080623184546.7002.4380.stgit@dell3.ogc.int> Roland, I think this one is ready to merge too. Steve. --- RDMA/cxgb3: Fix up some of the cxgb3 ib_device_attr fields. - set fw_ver - set hw_ver - set max_qp_wr to something reasonable - set max_cqe to something reasonable Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.h | 3 ++- drivers/infiniband/hw/cxgb3/iwch.c | 4 ++-- drivers/infiniband/hw/cxgb3/iwch_provider.c | 21 +++++++++++++++++++++ 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 4bb5926..cdc6ac9 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -45,13 +45,14 @@ #define T3_CTRL_QP_SIZE_LOG2 8 #define T3_CTRL_CQ_ID 0 -/* TBD */ #define T3_MAX_NUM_RI (1<<15) #define T3_MAX_NUM_QP (1<<15) #define T3_MAX_NUM_CQ (1<<15) #define T3_MAX_NUM_PD (1<<15) #define T3_MAX_PBL_SIZE 256 #define T3_MAX_RQ_SIZE 1024 +#define T3_MAX_QP_DEPTH (T3_MAX_RQ_SIZE-1) +#define T3_MAX_CQ_DEPTH 8192 #define T3_MAX_NUM_STAG (1<<15) #define T3_MAX_MR_SIZE 0x100000000ULL diff --git a/drivers/infiniband/hw/cxgb3/iwch.c b/drivers/infiniband/hw/cxgb3/iwch.c index 71554ea..e09cc1a 100644 --- a/drivers/infiniband/hw/cxgb3/iwch.c +++ b/drivers/infiniband/hw/cxgb3/iwch.c @@ -74,11 +74,11 @@ static void rnic_init(struct iwch_dev *rnicp) rnicp->attr.vendor_id = 0x168; rnicp->attr.vendor_part_id = 7; rnicp->attr.max_qps = T3_MAX_NUM_QP - 32; - rnicp->attr.max_wrs = (1UL << 24) - 1; + rnicp->attr.max_wrs = T3_MAX_QP_DEPTH; rnicp->attr.max_sge_per_wr = T3_MAX_SGE; rnicp->attr.max_sge_per_rdma_write_wr = T3_MAX_SGE; rnicp->attr.max_cqs = T3_MAX_NUM_CQ - 1; - rnicp->attr.max_cqes_per_cq = (1UL << 24) - 1; + rnicp->attr.max_cqes_per_cq = T3_MAX_CQ_DEPTH; rnicp->attr.max_mem_regs = cxio_num_stags(&rnicp->rdev); rnicp->attr.max_phys_buf_entries = T3_MAX_PBL_SIZE; rnicp->attr.max_pds = T3_MAX_NUM_PD - 1; diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index d33d4ed..29738e1 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1093,6 +1093,25 @@ static int iwch_query_gid(struct ib_device *ibdev, u8 port, return 0; } +static u64 fw_vers_string_to_u64(struct iwch_dev *iwch_dev) +{ + struct ethtool_drvinfo info; + struct net_device *lldev = iwch_dev->rdev.t3cdev_p->lldev; + char *cp, *next; + unsigned fw_maj, fw_min, fw_mic; + + lldev->ethtool_ops->get_drvinfo(lldev, &info); + next = info.fw_version+1; + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_maj); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_min); + cp = strsep(&next, "."); + sscanf(cp, "%i", &fw_mic); + return (((u64)fw_maj & 0xffff) << 32) | ((fw_min & 0xffff) << 16) | + (fw_mic & 0xffff); +} + static int iwch_query_device(struct ib_device *ibdev, struct ib_device_attr *props) { @@ -1103,6 +1122,8 @@ static int iwch_query_device(struct ib_device *ibdev, dev = to_iwch_dev(ibdev); memset(props, 0, sizeof *props); memcpy(&props->sys_image_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); + props->hw_ver = dev->rdev.t3cdev_p->type; + props->fw_ver = fw_vers_string_to_u64(dev); props->device_cap_flags = dev->device_cap_flags; props->vendor_id = (u32)dev->rdev.rnic_info.pdev->vendor; props->vendor_part_id = (u32)dev->rdev.rnic_info.pdev->device; From rdreier at cisco.com Mon Jun 23 12:17:01 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:17:01 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr Message-ID: It seems there is some confusion about struct ib_device_attr.vendor_id: mthca and mlx4 are using an OUI for vendor_id, while ipath and cxgb3 use the PCI vendor ID (and I don't know what ehca does). Since these namespaces are not coordinated, there is the (remote) chance of a collision. As far as I can tell, the ipath usage at least is a bug, since the IB spec says that NodeInfo:VendorID should be "device vendor, per IEEE" suggesting that OUI is the right thing to use. If that is the case, then it makes sense for iWARP devices to use OUI as well. However changing this means that any apps that look at the vendor_id field need to be updated to handle both the old and new value. This comes up because iw_nes currently doesn't set vendor_id at all, and I wonder what we should stick in there. What do other people think the way forward is? - R. From rdreier at cisco.com Mon Jun 23 12:23:47 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:23:47 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This will get a fix for one rare bug seen by SGI on large clusters: Eli Cohen (1): IB/mthca: Clear ICM pages before handing to FW drivers/infiniband/hw/mthca/mthca_memfree.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/mthca/mthca_memfree.c b/drivers/infiniband/hw/mthca/mthca_memfree.c index b224079..d5862e5 100644 --- a/drivers/infiniband/hw/mthca/mthca_memfree.c +++ b/drivers/infiniband/hw/mthca/mthca_memfree.c @@ -109,7 +109,11 @@ static int mthca_alloc_icm_pages(struct scatterlist *mem, int order, gfp_t gfp_m { struct page *page; - page = alloc_pages(gfp_mask, order); + /* + * Use __GFP_ZERO because buggy firmware assumes ICM pages are + * cleared, and subtle failures are seen if they aren't. + */ + page = alloc_pages(gfp_mask | __GFP_ZERO, order); if (!page) return -ENOMEM; From jsquyres at cisco.com Mon Jun 23 12:23:53 2008 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 23 Jun 2008 15:23:53 -0400 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: Message-ID: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Are there many apps out there yet that are depending on iWarp vendor ID/part ID? OMPI uses these to set device-specific values at startup, but we have not released this code yet (it will be in our upcoming v1.3 series). Does MVAPICH* use them? What about non-MPI ULPs / applications? My $0.02 is to make it all one namespace since there's precedent for it from InfiniBand: OUI's. Modify the Chelsio and NetEffect drivers to return the OUI. Hopefully, there's not enough iWarp apps out there reading this info that it will matter. On Jun 23, 2008, at 3:17 PM, Roland Dreier wrote: > It seems there is some confusion about struct > ib_device_attr.vendor_id: > mthca and mlx4 are using an OUI for vendor_id, while ipath and cxgb3 > use > the PCI vendor ID (and I don't know what ehca does). Since these > namespaces are not coordinated, there is the (remote) chance of a > collision. As far as I can tell, the ipath usage at least is a bug, > since the IB spec says that NodeInfo:VendorID should be "device > vendor, > per IEEE" suggesting that OUI is the right thing to use. If that is > the > case, then it makes sense for iWARP devices to use OUI as well. > > However changing this means that any apps that look at the vendor_id > field need to be updated to handle both the old and new value. > > This comes up because iw_nes currently doesn't set vendor_id at all, > and > I wonder what we should stick in there. > > What do other people think the way forward is? > > - R. > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Jeff Squyres Cisco Systems From rdreier at cisco.com Mon Jun 23 12:27:31 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:27:31 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> (Jeff Squyres's message of "Mon, 23 Jun 2008 15:23:53 -0400") References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: > Are there many apps out there yet that are depending on iWarp vendor > ID/part ID? OMPI is the only one I know of for sure. - R. From rdreier at cisco.com Mon Jun 23 12:28:40 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:28:40 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> (Jeff Squyres's message of "Mon, 23 Jun 2008 15:23:53 -0400") References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: > OMPI uses these to set device-specific values at startup, but we have > not released this code yet (it will be in our upcoming v1.3 series). > Does MVAPICH* use them? What about non-MPI ULPs / applications? Don't you already look at device IDs? Or do you mean that the iwarp stuff isn't released? One question is whether we want to change (fix?) the ipath driver to use OUI. Strict IB spec compliances says we should. - R. From jsquyres at cisco.com Mon Jun 23 12:31:40 2008 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 23 Jun 2008 15:31:40 -0400 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: On Jun 23, 2008, at 3:28 PM, Roland Dreier wrote: >> OMPI uses these to set device-specific values at startup, but we have >> not released this code yet (it will be in our upcoming v1.3 series). >> Does MVAPICH* use them? What about non-MPI ULPs / applications? > > Don't you already look at device IDs? Or do you mean that the iwarp > stuff isn't released? I mean that the OMPI doesn't support iWARP until our next release series (v1.3). So yes, we've been looking at IB vendor/part ID's for a long time. But the issue is no big deal for iWARP devices because we haven't released iWARP-capable OMPI yet. > One question is whether we want to change (fix?) the ipath driver to > use > OUI. Strict IB spec compliances says we should. FWIW, OMPI checks these values for QL HCAs: ----- [QLogic InfiniPath] vendor_id = 0x1fc1 vendor_part_id = 13 use_eager_rdma = 1 mtu = 2048 [QLogic InfiniPath] vendor_id = 0x1fc1,0x1077 vendor_part_id = 16,29216 use_eager_rdma = 1 mtu = 4096 ----- Translation: vendor_id values 0x1fc1 and 0x1077. -- Jeff Squyres Cisco Systems From rdreier at cisco.com Mon Jun 23 12:35:30 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:35:30 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: (Jeff Squyres's message of "Mon, 23 Jun 2008 15:31:40 -0400") References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: > Translation: vendor_id values 0x1fc1 and 0x1077. Which are PCI IDs, not IEEE OUIs, unfortunately. From jsquyres at cisco.com Mon Jun 23 12:38:50 2008 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 23 Jun 2008 15:38:50 -0400 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: <89E146BA-124D-4250-963A-3B900B3E171E@cisco.com> On Jun 23, 2008, at 3:35 PM, Roland Dreier wrote: >> Translation: vendor_id values 0x1fc1 and 0x1077. > > Which are PCI IDs, not IEEE OUIs, unfortunately. If we decide to move to OUI's, it's easy enough for me to change in OMPI. It would be good to know this soon so that we can get the changes in OMPI v1.2.7 (coming out RSN) and v1.3 (still have a little time). Heck, I could just hedge our bets and put in QL's OUI in the file and let that *also* be checked if the debate / implementation-release cycle on this change is going to take a while... What's QL's OUI? -- Jeff Squyres Cisco Systems From rdreier at cisco.com Mon Jun 23 12:41:06 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:41:06 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <89E146BA-124D-4250-963A-3B900B3E171E@cisco.com> (Jeff Squyres's message of "Mon, 23 Jun 2008 15:38:50 -0400") References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <89E146BA-124D-4250-963A-3B900B3E171E@cisco.com> Message-ID: > What's QL's OUI? QLogic has lots of OUIs allocated so it's hard to say. The HCAs I have (which are not particularly new and I think predate the acquisition of Pathscale by QLogic) use 0x1175 but newer HCAs may use another OUI. - R. From jgunthorpe at obsidianresearch.com Mon Jun 23 12:44:21 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Mon, 23 Jun 2008 13:44:21 -0600 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: <20080623194421.GI29198@obsidianresearch.com> On Mon, Jun 23, 2008 at 03:31:40PM -0400, Jeff Squyres wrote: >> One question is whether we want to change (fix?) the ipath driver to use >> OUI. Strict IB spec compliances says we should. > > > FWIW, OMPI checks these values for QL HCAs: > > [QLogic InfiniPath] > vendor_id = 0x1fc1 > vendor_part_id = 13 > use_eager_rdma = 1 > mtu = 2048 > > [QLogic InfiniPath] > vendor_id = 0x1fc1,0x1077 > vendor_part_id = 16,29216 > use_eager_rdma = 1 > mtu = 4096 > > Translation: vendor_id values 0x1fc1 and 0x1077. If you are using the vendor_id to key sillicon specific optimizations then I'd think it is better to look at the PCI ID rather than the OUI. The OUI belongs to whoever sticks the chip on a card, and has no relation to the sillicon capabilities.. Jason From rdreier at cisco.com Mon Jun 23 12:47:14 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 12:47:14 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080623194421.GI29198@obsidianresearch.com> (Jason Gunthorpe's message of "Mon, 23 Jun 2008 13:44:21 -0600") References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> Message-ID: > If you are using the vendor_id to key sillicon specific optimizations > then I'd think it is better to look at the PCI ID rather than the > OUI. The OUI belongs to whoever sticks the chip on a card, and has no > relation to the sillicon capabilities.. PCI vendor ID is pretty much the same boat though. OUI at least has the benefit that it's sensible for non-PCI devices such as IBM ehca. - R. From hrosenstock at xsigo.com Mon Jun 23 12:50:13 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 23 Jun 2008 12:50:13 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: Message-ID: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-23 at 12:17 -0700, Roland Dreier wrote: > As far as I can tell, the ipath usage at least is a bug, > since the IB spec says that NodeInfo:VendorID should be "device > vendor, per IEEE" suggesting that OUI is the right thing to use. That was the IBA intention AFAIK. -- Hal From swise at opengridcomputing.com Mon Jun 23 12:53:04 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 23 Jun 2008 14:53:04 -0500 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> References: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> Message-ID: <485FFF20.9090703@opengridcomputing.com> Hal Rosenstock wrote: > On Mon, 2008-06-23 at 12:17 -0700, Roland Dreier wrote: > >> As far as I can tell, the ipath usage at least is a bug, >> since the IB spec says that NodeInfo:VendorID should be "device >> vendor, per IEEE" suggesting that OUI is the right thing to use. >> > > That was the IBA intention AFAIK. > > -- Hal > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > OUI is ok by me... They are globally unique, yes? :) From ctung at NetEffect.com Mon Jun 23 12:55:17 2008 From: ctung at NetEffect.com (Chien Tung) Date: Mon, 23 Jun 2008 14:55:17 -0500 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC08235FA4@venom2> > This comes up because iw_nes currently doesn't set vendor_id > at all, and I wonder what we should stick in there. > > What do other people think the way forward is? I was in the middle of fixing it. I can make it OUI (0x001255). Chien From hrosenstock at xsigo.com Mon Jun 23 12:55:32 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 23 Jun 2008 12:55:32 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> References: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> Message-ID: <1214250932.13056.244.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-23 at 12:50 -0700, Hal Rosenstock wrote: > On Mon, 2008-06-23 at 12:17 -0700, Roland Dreier wrote: > > As far as I can tell, the ipath usage at least is a bug, > > since the IB spec says that NodeInfo:VendorID should be "device > > vendor, per IEEE" suggesting that OUI is the right thing to use. > > That was the IBA intention AFAIK. Also, further device identification is available from device_id. -- Hal > > -- Hal > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ctung at NetEffect.com Mon Jun 23 12:58:37 2008 From: ctung at NetEffect.com (Chien Tung) Date: Mon, 23 Jun 2008 14:58:37 -0500 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <1214250932.13056.244.camel@hrosenstock-ws.xsigo.com> References: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> <1214250932.13056.244.camel@hrosenstock-ws.xsigo.com> Message-ID: <5E701717F2B2ED4EA60F87C8AA57B7CC08235FA9@venom2> > Also, further device identification is available from device_id. Do you mean vendor_part_id or PCI config device_id? I was going to set vendor_part_id the same as PCI config device id. Chien From jgunthorpe at obsidianresearch.com Mon Jun 23 13:02:13 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Mon, 23 Jun 2008 14:02:13 -0600 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> Message-ID: <20080623200213.GJ29198@obsidianresearch.com> On Mon, Jun 23, 2008 at 12:47:14PM -0700, Roland Dreier wrote: > > If you are using the vendor_id to key sillicon specific optimizations > > then I'd think it is better to look at the PCI ID rather than the > > OUI. The OUI belongs to whoever sticks the chip on a card, and has no > > relation to the sillicon capabilities.. > > PCI vendor ID is pretty much the same boat though. Well, the way PCI ID usually works is that the sillicon vendor bakes their ID into the chip and then allows downstream vendors to alter the subsytem id, ie: 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express (rev 02) Subsystem: Dell Unknown device 01de 2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:13:72:29:e6:5d brd ff:ff:ff:ff:ff:ff 001372 (base 16) Dell Inc. So unless people are abusing this, a PCI ID is tied to a piece of sillicon, while OUI+Subsystem ID+GUID/MAC is tied to the board vendor.. So, yes, there tend to be many PCI IDs for sillicon that is very similar, but the OUI tells you nothing about the underlying sillicon. Using an OUI to trigger sillicon specific tuning only works while we have a small set of suppliers. I'm just commenting on Jeff's use of the vendor_id in OMPI, I have no real opinion on what should go in the structure member. > OUI at least has the benefit that it's sensible for non-PCI devices such > as IBM ehca. IBM has a vendor ID, they could assign a device ID in their space to the ehca.. Jason From jsquyres at cisco.com Mon Jun 23 13:07:52 2008 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 23 Jun 2008 16:07:52 -0400 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080623200213.GJ29198@obsidianresearch.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> Message-ID: <1DA0A788-CA79-4B3A-983C-0342981A9A77@cisco.com> On Jun 23, 2008, at 4:02 PM, Jason Gunthorpe wrote: > So unless people are abusing this, a PCI ID is tied to a piece of > sillicon, while OUI+Subsystem ID+GUID/MAC is tied to the board > vendor.. So, yes, there tend to be many PCI IDs for sillicon that is > very similar, but the OUI tells you nothing about the underlying > sillicon. > > Using an OUI to trigger sillicon specific tuning only works while we > have a small set of suppliers. The same could be said for having a small set of PCI ID's, no? There are both a small number of suppliers and PCI ID's today, so I guess it hasn't mattered to this point. > I'm just commenting on Jeff's use of the vendor_id in OMPI, I have no > real opinion on what should go in the structure member. We use the (vendor_id, vendor_part_id) tuple to identify an HCA and the options that we should use with it because these were the two values available to us via the verbs API. Is there something better that we should do? I'm fairly ambivalent -- on what the actual value should be in vendor ID, as long as it's a consistent value/namespace. -- Jeff Squyres Cisco Systems From hrosenstock at xsigo.com Mon Jun 23 13:10:04 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 23 Jun 2008 13:10:04 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080623194421.GI29198@obsidianresearch.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> Message-ID: <1214251804.13056.247.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-23 at 13:44 -0600, Jason Gunthorpe wrote: > If you are using the vendor_id to key sillicon specific optimizations > then I'd think it is better to look at the PCI ID rather than the > OUI. The OUI belongs to whoever sticks the chip on a card, and has no > relation to the sillicon capabilities.. >From an IBA perspective, it would be [vendor ID, device ID, revision] if one needs to look that "deep". -- Hal From koop at cse.ohio-state.edu Mon Jun 23 13:10:18 2008 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Mon, 23 Jun 2008 16:10:18 -0400 (EDT) Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> Message-ID: > Are there many apps out there yet that are depending on iWarp vendor > ID/part ID? > > OMPI uses these to set device-specific values at startup, but we have > not released this code yet (it will be in our upcoming v1.3 series). > Does MVAPICH* use them? What about non-MPI ULPs / applications? In MVAPICH we do not use the vendor IDs at this time, but instead the ca_type returned from umad_get_ca. Matt > > My $0.02 is to make it all one namespace since there's precedent for > it from InfiniBand: OUI's. Modify the Chelsio and NetEffect drivers > to return the OUI. Hopefully, there's not enough iWarp apps out there > reading this info that it will matter. > > > > On Jun 23, 2008, at 3:17 PM, Roland Dreier wrote: > > > It seems there is some confusion about struct > > ib_device_attr.vendor_id: > > mthca and mlx4 are using an OUI for vendor_id, while ipath and cxgb3 > > use > > the PCI vendor ID (and I don't know what ehca does). Since these > > namespaces are not coordinated, there is the (remote) chance of a > > collision. As far as I can tell, the ipath usage at least is a bug, > > since the IB spec says that NodeInfo:VendorID should be "device > > vendor, > > per IEEE" suggesting that OUI is the right thing to use. If that is > > the > > case, then it makes sense for iWARP devices to use OUI as well. > > > > However changing this means that any apps that look at the vendor_id > > field need to be updated to handle both the old and new value. > > > > This comes up because iw_nes currently doesn't set vendor_id at all, > > and > > I wonder what we should stick in there. > > > > What do other people think the way forward is? > > > > - R. > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From rdreier at cisco.com Mon Jun 23 13:10:09 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 13:10:09 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080623200213.GJ29198@obsidianresearch.com> (Jason Gunthorpe's message of "Mon, 23 Jun 2008 14:02:13 -0600") References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> Message-ID: > Well, the way PCI ID usually works is that the sillicon vendor bakes > their ID into the chip and then allows downstream vendors to alter > the subsytem id, ie: > 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express (rev 02) That's one model of how things work, but this is actually a good example: look in the tg3.c driver for this chip -- not all the vendor IDs are Broadcom. > > OUI at least has the benefit that it's sensible for non-PCI devices such > > as IBM ehca. > IBM has a vendor ID, they could assign a device ID in their space to the > ehca.. Not sure what you mean: use an IBM PCI ID for ehca? - R. From hrosenstock at xsigo.com Mon Jun 23 13:13:56 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 23 Jun 2008 13:13:56 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <5E701717F2B2ED4EA60F87C8AA57B7CC08235FA9@venom2> References: <1214250613.13056.241.camel@hrosenstock-ws.xsigo.com> <1214250932.13056.244.camel@hrosenstock-ws.xsigo.com> <5E701717F2B2ED4EA60F87C8AA57B7CC08235FA9@venom2> Message-ID: <1214252036.13056.251.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-23 at 14:58 -0500, Chien Tung wrote: > > Also, further device identification is available from device_id. > > Do you mean vendor_part_id or PCI config device_id? I was going to set > vendor_part_id the same as PCI config device id. I meant DeviceID from the NodeInfo attribute (which is named vendor_part_id in the structure). -- Hal > Chien > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Mon Jun 23 13:23:23 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 13:23:23 -0700 Subject: [ofa-general] Re: [PATCH REPOST #2] IB/ehca: In case of lost interrupts, trigger EOI to reenable interrupts In-Reply-To: <200806131655.43410.ossrosch@linux.vnet.ibm.com> (Stefan Roscher's message of "Fri, 13 Jun 2008 16:55:41 +0200") References: <200806131655.43410.ossrosch@linux.vnet.ibm.com> Message-ID: ok, I queued this for 2.6.27. thanks everyone From jgunthorpe at obsidianresearch.com Mon Jun 23 13:23:21 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Mon, 23 Jun 2008 14:23:21 -0600 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <1214251804.13056.247.camel@hrosenstock-ws.xsigo.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <1214251804.13056.247.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080623202321.GK29198@obsidianresearch.com> On Mon, Jun 23, 2008 at 01:10:04PM -0700, Hal Rosenstock wrote: > On Mon, 2008-06-23 at 13:44 -0600, Jason Gunthorpe wrote: > > If you are using the vendor_id to key sillicon specific optimizations > > then I'd think it is better to look at the PCI ID rather than the > > OUI. The OUI belongs to whoever sticks the chip on a card, and has no > > relation to the sillicon capabilities.. > > >From an IBA perspective, it would be [vendor ID, device ID, revision] if > one needs to look that "deep". Well, who's OUI is the vendor ID? The chip makers or the board makers? That has never really seemed clear to me, and the spec doesn't say. It is useless to identify the sillicon if it is the board makers, but there is no equivilent to 'subsystem ID' like PCI has for the board.. Jason From chu11 at llnl.gov Mon Jun 23 13:24:58 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 23 Jun 2008 13:24:58 -0700 Subject: [ofa-general] [OPENSM PATCH 0/3]: "guid-routing-order" option for minhop/updn routing Message-ID: <1214252698.5369.537.camel@cardanus.llnl.gov> Hey Sasha, Here's the adjusted patches, as discussed in the previous thread. It now supports both minhop and updn. I wasn't sure of what the cleanest way to make __osm_ucast_mgr_process_tbl() usable by both the original and ordered- routing options. PLMK what you think in patch 2/3. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From chu11 at llnl.gov Mon Jun 23 13:25:02 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 23 Jun 2008 13:25:02 -0700 Subject: [ofa-general] [OPENSM PATCH 1/3]: add guid_routing_order_file option Message-ID: <1214252702.5369.538.camel@cardanus.llnl.gov> Nothing fancy. manpage, cmdline options, config options, etc. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-add-guid_routing_order_file-option.patch Type: text/x-patch Size: 5877 bytes Desc: not available URL: From chu11 at llnl.gov Mon Jun 23 13:25:06 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 23 Jun 2008 13:25:06 -0700 Subject: [ofa-general] [OPENSM PATCH 2/3]: rearch __osm_ucast_mgr_process_tbl() usage Message-ID: <1214252706.5369.539.camel@cardanus.llnl.gov> Re-work __osm_ucast_mgr_process_tbl() API so either a table or a list can be passed for routing. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-rearch-__osm_ucast_mgr_process_tbl-usage.patch Type: text/x-patch Size: 3254 bytes Desc: not available URL: From chu11 at llnl.gov Mon Jun 23 13:25:08 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 23 Jun 2008 13:25:08 -0700 Subject: [ofa-general] [OPENSM PATCH 3/3]: implement guid_routing_order_file Message-ID: <1214252708.5369.540.camel@cardanus.llnl.gov> the actual implementation. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0003-support-guid_routing_order_file.patch Type: text/x-patch Size: 9171 bytes Desc: not available URL: From jsquyres at cisco.com Mon Jun 23 13:30:46 2008 From: jsquyres at cisco.com (Jeff Squyres) Date: Mon, 23 Jun 2008 16:30:46 -0400 Subject: [ofa-general] maximum IB/iWARP max_inline_data size In-Reply-To: <485D016B.4080009@gmail.com> References: <485D016B.4080009@gmail.com> Message-ID: On Jun 21, 2008, at 9:26 AM, Dotan Barak wrote: >> Our bias in OMPI is to figure such values dynamically at run-time. >> With T3/NE020, it seems to be easy. Note that if you choose a >> max_inline_data value too large, ibv_create_qp will simply fail. >> So how can we figure out what the max_inline_data value is for >> Mellanox adapters at run-time? (I'd prefer not to repeatedly >> trying to create a QP with increasing values of max_inline_data >> until it fails). > Is it possible to add such a capability to the HCA capabilities > structure (the high limit which is supported by the device)? That would be nice, but it doesn't solve my immediate problem. :-) Paul Hargrove told me (off list) that he ended up looping over max_inline_data values testing to see if ibv_create_qp() fails with EINVAL. Yuck! -- Jeff Squyres Cisco Systems From rdreier at cisco.com Mon Jun 23 13:34:46 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 13:34:46 -0700 Subject: [ofa-general] [PATCH] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: <484F8A20.9070101@Voltaire.COM> (Moni Shoua's message of "Wed, 11 Jun 2008 11:17:36 +0300") References: <484F8A20.9070101@Voltaire.COM> Message-ID: Please run your patches through scripts/checkpatch.pl and consider the output before sending them... it's easy to do and saves everyone time fixing trivial style mistakes. More substantive comments: > +static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int level) Instead of "int level" here, I would prefer to see something like enum ipoib_flush_level { IPOIB_FLUSH_LIGHT, IPOIB_FLUSH_NORMAL, IPOIB_FLUSH_HEAVY, }; so that code like > - if (pkey_event) { > + if (level == 2) { remains at somewhat self-documenting. > +static void path_refresh(struct net_device *dev, struct ipoib_path *path) > +{ > + struct ipoib_dev_priv *priv = netdev_priv(dev); > + ipoib_dbg(priv, "mark path LID 0x%04x GID " IPOIB_GID_FMT " invalid\n", > + be16_to_cpu(path->pathrec.dlid), IPOIB_GID_ARG(path->pathrec.dgid)); > + path->valid = 0; > +} I'm not sure this wrapper makes sense, because it is so trivial and it doesn't even do what the name suggests -- it doesn't refresh the path, it just marks it as invalid. > +void ipoib_refresh_paths(struct net_device *dev) > +{ > + struct ipoib_dev_priv *priv = netdev_priv(dev); > + struct ipoib_path *path, *tp; > + spin_lock_irq(&priv->tx_lock); > + spin_lock(&priv->lock); > + > + list_for_each_entry_safe(path, tp, &priv->path_list, list) { > + if (path->query) > + ib_sa_cancel_query(path->query_id, path->query); > + spin_unlock(&priv->lock); > + spin_unlock_irq(&priv->tx_lock); > + wait_for_completion(&path->done); > + path_refresh(dev, path); > + spin_lock_irq(&priv->tx_lock); > + spin_lock(&priv->lock); > + } > + spin_unlock(&priv->lock); > + spin_unlock_irq(&priv->tx_lock); > +} This looks broken for a couple of reasons: - What protects against paths being added to the path list while this loop is running? ipoib_flush_paths() moves the list to a new place to avoid this problem. - What happens if you mark a path as invalid and do wait_for_completion(), and then ipoib_flush_paths() runs before another path record query is done? It seems that the second wait_for_completion() will wait forever. - R. From jgunthorpe at obsidianresearch.com Mon Jun 23 13:41:03 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Mon, 23 Jun 2008 14:41:03 -0600 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> Message-ID: <20080623204103.GL29198@obsidianresearch.com> On Mon, Jun 23, 2008 at 01:10:09PM -0700, Roland Dreier wrote: > > Well, the way PCI ID usually works is that the sillicon vendor bakes > > their ID into the chip and then allows downstream vendors to alter > > the subsytem id, ie: > > > 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express (rev 02) > > That's one model of how things work, but this is actually a good > example: look in the tg3.c driver for this chip -- not all the vendor > IDs are Broadcom. Right, but we don't know what caused this. At least two of the non-broadcom IDs have later special casing in the driver, which suggests they may actually be different sillicon - produced under contract, perhaps... Considering the wide use of tg3's by 3rd parties it seems that having such a small number of non-broadcom ids does suggest the model works and is accepted by the vendors. If IBA vendor_id is thought to be the same as PCI vendor ID - where it is the silicon manufactures OUI, then that is great, the iwarp folks should follow the same model, and derive their vendor OUI/device_id from the PCI ID. But if it is closer to the PCI subsystem ID then it is pretty useless... I don't have a wide enough collection of cards to tell what people have been doing to date.. Personally, I feel the intention of the spec was to follow the PCI model, it does introduce subsystem ID's for the IO controller stuff, for instance. But that is moot if people have already produced alot of equipment that does differently :) Jason From rdreier at cisco.com Mon Jun 23 13:43:30 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 13:43:30 -0700 Subject: [ofa-general] Re: [PATCH 2/3 v2] mlx4: multicast cast loopback block In-Reply-To: (Ron Livne's message of "Sun, 22 Jun 2008 16:36:40 +0000 (UTC)") References: Message-ID: Please run your patches through scripts/checkpatch.pl and consider the output before sending them... it's easy to do and saves everyone time fixing trivial style mistakes. More substantive comments: > + if (to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK) > + return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, > + &to_mqp(ibqp)->mqp, gid->raw, 1); > + else > + return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, > + &to_mqp(ibqp)->mqp, gid->raw, 0); Am I missing something? Is there a reason not to write this as: return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, &to_mqp(ibqp)->mqp, gid->raw, !!(to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK)); - R. From arlin.r.davis at intel.com Mon Jun 23 14:06:57 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Mon, 23 Jun 2008 14:06:57 -0700 Subject: [ofa-general] [PATCH 4/7][v1.2] dapl scm: Adding socket cmprovider for better scalability on large homogeneous clusters. In-Reply-To: <20080623183646.GG29683@ammonite.mv.qlogic.com> References: <20080623183646.GG29683@ammonite.mv.qlogic.com> Message-ID: > >It seems like you have some inconsistency in how you handle the SGEs in >dapls_ib_post_recv and dapls_ib_post_send between uDAPL v1 and >uDAPL v2. > >For example in v1, you now leak the SGEs if the SGEs aren't allocated >on the stack whereas in v2, you free them. > Good catch. Looks like the fix was lost in the scm merge. reapplied your patch to v1 scm provider: -- Fix memory leak in provider specific post send/recv if there's more than DEFAULT_DS_ENTRIES iovecs. Signed-off-by: Patrick Marchand Latifi diff --git a/dapl/openib_scm/dapl_ib_dto.h b/dapl/openib_scm/dapl_ib_dto.h index b15f347..7dd3075 100644 --- a/dapl/openib_scm/dapl_ib_dto.h +++ b/dapl/openib_scm/dapl_ib_dto.h @@ -67,10 +67,11 @@ dapls_ib_post_recv ( IN DAT_LMR_TRIPLET *local_iov ) { ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; - ib_data_segment_t *ds_array_p; + ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; struct ibv_recv_wr wr; struct ibv_recv_wr *bad_wr; DAT_COUNT i, total_len; + int ret; dapl_dbg_log(DAPL_DBG_TYPE_EP, " post_rcv: ep %p cookie %p segs %d l_iov %p\n", @@ -79,7 +80,7 @@ dapls_ib_post_recv ( if (segments <= DEFAULT_DS_ENTRIES) ds_array_p = ds_array; else - ds_array_p = + ds_array_start_p = ds_array_p = dapl_os_alloc(segments * sizeof(ib_data_segment_t)); if (NULL == ds_array_p) @@ -113,8 +114,13 @@ dapls_ib_post_recv ( if (cookie != NULL) cookie->val.dto.size = total_len; - if (ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr)) - return( dapl_convert_errno(errno,"ibv_recv") ); + ret = ibv_post_recv(ep_ptr->qp_handle, &wr, &bad_wr); + + if (ds_array_start_p != NULL) + dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); + + if (ret) + return(dapl_convert_errno(errno,"ibv_recv")); return DAT_SUCCESS; } @@ -141,12 +147,13 @@ dapls_ib_post_send ( remote_iov, completion_flags); ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES]; - ib_data_segment_t *ds_array_p; + ib_data_segment_t *ds_array_p, *ds_array_start_p = NULL; struct ibv_send_wr wr; struct ibv_send_wr *bad_wr; ib_hca_transport_t *ibt_ptr = &ep_ptr->header.owner_ia->hca_ptr->ib_trans; DAT_COUNT i, total_len; + int ret; dapl_dbg_log(DAPL_DBG_TYPE_EP, " post_snd: ep %p cookie %p segs %d l_iov %p\n", @@ -155,7 +162,7 @@ dapls_ib_post_send ( if(segments <= DEFAULT_DS_ENTRIES) ds_array_p = ds_array; else - ds_array_p = + ds_array_start_p = ds_array_p = dapl_os_alloc(segments * sizeof(ib_data_segment_t)); if (NULL == ds_array_p) @@ -217,9 +224,14 @@ dapls_ib_post_send ( " post_snd: op 0x%x flags 0x%x sglist %p, %d\n", wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge); - if (ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr)) - return( dapl_convert_errno(errno,"ibv_recv") ); + ret = ibv_post_send(ep_ptr->qp_handle, &wr, &bad_wr); + if (ds_array_start_p != NULL) + dapl_os_free(ds_array_start_p, segments * sizeof(ib_data_segment_t)); + + if (ret) + return(dapl_convert_errno(errno,"ibv_send")); + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); return DAT_SUCCESS; } From rdreier at cisco.com Mon Jun 23 14:08:41 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 14:08:41 -0700 Subject: [ofa-general] Re: [PATCH 6/11] core: XRC - file descriptors associated with XRC domains In-Reply-To: <200806231601.38023.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:01:37 +0300") References: <200806231601.38023.jackm@dev.mellanox.co.il> Message-ID: > +static struct inode * xrc_fd2inode(unsigned int fd) > +{ > + struct file * f = fget(fd); > + > + if (!f) > + return NULL; > + > + return f->f_dentry->d_inode; > +} Am I missing something, or is there no fput() that matches this fget()? Is there some reason why we don't need an igrab() of the inode we're using here? (If we do need it then a corresponding iput() when we're done with the inode is required of course) - R. From patrick.latifi at qlogic.com Mon Jun 23 14:15:32 2008 From: patrick.latifi at qlogic.com (Patrick Marchand Latifi) Date: Mon, 23 Jun 2008 14:15:32 -0700 Subject: [ofa-general] [PATCH 4/7][v1.2] dapl scm: Adding socket cmprovider for better scalability on large homogeneous clusters. In-Reply-To: References: <20080623183646.GG29683@ammonite.mv.qlogic.com> Message-ID: <20080623211531.GH29683@ammonite.mv.qlogic.com> On Mon, Jun 23, 2008 at 02:06:57PM -0700, Davis, Arlin R wrote: > > Good catch. Looks like the fix was lost in the scm merge. > > reapplied your patch to v1 scm provider: > looks good to me, -pat From rdreier at cisco.com Mon Jun 23 14:43:18 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 14:43:18 -0700 Subject: [ofa-general] Re: [PATCH] IPOIB: add LRO support. In-Reply-To: <485F9D00.6090406@dev.mellanox.co.il> (Vladimir Sokolovsky's message of "Mon, 23 Jun 2008 15:54:24 +0300") References: <485F9D00.6090406@dev.mellanox.co.il> Message-ID: Overall, looks mostly fine (although as I've said a lot of times today, please run scripts/checkpatch.pl and look at the output before sending patches -- it saves lots of time for everyone). One question -- given this: > + if (skb->protocol != htons(ETH_P_IP)) > + return -1; is there any way we could have anything other than IPv4 here: > + if (iph->protocol != IPPROTO_TCP || iph->version != 4) > + return -1; - R. From rdreier at cisco.com Mon Jun 23 14:53:22 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 14:53:22 -0700 Subject: [ofa-general] Re: [PATCH 8/11] core: XRC receive-only QPs In-Reply-To: <200806231601.43706.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:01:43 +0300") References: <200806231601.43706.jackm@dev.mellanox.co.il> Message-ID: > @@ -769,6 +775,7 @@ struct ib_ucontext { > struct list_head srq_list; > struct list_head ah_list; > struct list_head xrc_domain_list; > + struct list_head xrc_reg_qp_list; > int closing; > }; Wouldn't it be cleaner to keep the like of recv QPs per xrcd? Then you wouldn't have to do stuff like: > + list_for_each_entry(tmp, &file->ucontext->xrc_reg_qp_list, list) > + if (cmd.xrcd_handle == tmp->domain_handle) { instead you could just do a list_empty() test. - R. From LeighpegLarkin at itcompany.com Mon Jun 23 10:11:48 2008 From: LeighpegLarkin at itcompany.com (Betsy Dupree) Date: Mon, 23 Jun 2008 20:11:48 +0300 Subject: [ofa-general] Obtain an accredited BA, MA or PhD in 30 days! Message-ID: <8IX056EJXVWDA443@itcompany.com> Want a better job? We can help you in under 20 days Add the title "College graduate" to your resume Call 24/7 on.... For US: 1-419-735-9250 Outside US: +1-419-735-9250 From patrick.latifi at qlogic.com Mon Jun 23 16:17:31 2008 From: patrick.latifi at qlogic.com (Patrick Marchand Latifi) Date: Mon, 23 Jun 2008 16:17:31 -0700 Subject: [ofa-general] [PATCH 3/7][v1.2] dapl: add opcode to string function to report opcode during failures. In-Reply-To: References: Message-ID: <20080623231731.GI29683@ammonite.mv.qlogic.com> minor nit below: On Fri, Jun 20, 2008 at 11:48:18AM -0700, Davis, Arlin R wrote: > @@ -485,6 +489,19 @@ bail: > return dat_status; > } > > +STATIC _INLINE_ char * DAPL_GET_DTO_OP_STR(int op) > +{ > + static char *dto_ops[] = > + { > + "OP_SEND", > + "OP_RECEIVE", > + "OP_RDMA_WRITE", > + "OP_RDMA_READ", > + 0 > + }; > + return ((op < 0 || op > 3) ? "Invalid DTO OP?" : dto_ops[op]); > +} > + the terminating 0 is superfluous here -- if you are out of bounds, you will return the string "Invalid DTO OP?". Affects both uDAPL v1 and v2. -pat From rdreier at cisco.com Mon Jun 23 16:29:10 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 16:29:10 -0700 Subject: [ofa-general] [PATCH] RDMA/cxgb3: Remove write-only iwch_rnic_attributes fields Message-ID: The members struct iwch_rnic_attributes.vendor_id and .vendor_part_id are write-only, so we might as well get rid of them. Signed-off-by: Roland Dreier --- Unless I'm missing something, this should be OK to merge for 2.6.27, right? diff --git a/drivers/infiniband/hw/cxgb3/iwch.c b/drivers/infiniband/hw/cxgb3/iwch.c index 71554ea..4ea8bc0 100644 --- a/drivers/infiniband/hw/cxgb3/iwch.c +++ b/drivers/infiniband/hw/cxgb3/iwch.c @@ -71,8 +71,6 @@ static void rnic_init(struct iwch_dev *rnicp) idr_init(&rnicp->mmidr); spin_lock_init(&rnicp->lock); - rnicp->attr.vendor_id = 0x168; - rnicp->attr.vendor_part_id = 7; rnicp->attr.max_qps = T3_MAX_NUM_QP - 32; rnicp->attr.max_wrs = (1UL << 24) - 1; rnicp->attr.max_sge_per_wr = T3_MAX_SGE; diff --git a/drivers/infiniband/hw/cxgb3/iwch.h b/drivers/infiniband/hw/cxgb3/iwch.h index d2409a5..3773453 100644 --- a/drivers/infiniband/hw/cxgb3/iwch.h +++ b/drivers/infiniband/hw/cxgb3/iwch.h @@ -48,8 +48,6 @@ struct iwch_qp; struct iwch_mr; struct iwch_rnic_attributes { - u32 vendor_id; - u32 vendor_part_id; u32 max_qps; u32 max_wrs; /* Max for any SQ/RQ */ u32 max_sge_per_wr; From sashak at voltaire.com Mon Jun 23 17:33:21 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 03:33:21 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624003321.GC7341@sashak.voltaire.com> On 07:01 Mon 23 Jun , Hal Rosenstock wrote: > > Inlining is only a hint to the compiler and given the over(ab)use of > inline in OpenSM (by my count almost 500 instances) I doubt this has the > intended effect. I asked in order to understand about are there any specific reasons for this patch or just it is "nice to have" stuff (and not to mark my disagreement). > Are the inlines really needed in these two cases ? Inlining is potential optimization and as you said it is optional, so word "needed" is not the best which describes this :) I likely fine about the first case (especially about osm_port_prof_set_ignored_port()), but the second function really looks as "one-line stuff" for me. > In general, I think OpenSM needs a more careful look as to what really > needs inlining. Sure. Agreed here. > > Would be nice to not mix in one patch. > > Already done on list. Thanks. Sasha From rdreier at cisco.com Mon Jun 23 21:38:37 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 21:38:37 -0700 Subject: [ofa-general] [PATCH for-2.6.27] RDMA/cxgb3: Fix up some of the cxgb3 ib_device_attr fields. In-Reply-To: <20080623184546.7002.4380.stgit@dell3.ogc.int> (Steve Wise's message of "Mon, 23 Jun 2008 13:45:47 -0500") References: <20080623184546.7002.4380.stgit@dell3.ogc.int> Message-ID: thanks, applied From rdreier at cisco.com Mon Jun 23 21:44:16 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 23 Jun 2008 21:44:16 -0700 Subject: [ofa-general] [PATCH 11/11] mlx4: XRC receive-only QPs In-Reply-To: <200806231602.02721.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 23 Jun 2008 16:02:02 +0300") References: <200806231602.02721.jackm@dev.mellanox.co.il> Message-ID: > +int mlx4_ib_modify_xrc_rcv_qp(struct ib_xrcd *ibxrcd, u32 qp_num, > + struct ib_qp_attr *attr, int attr_mask) > +{ > + struct mlx4_ib_dev *dev = to_mdev(ibxrcd->device); > + struct mlx4_ib_xrcd *xrcd = to_mxrcd(ibxrcd); > + struct mlx4_qp *mqp; > + int err; > + > + if (!(dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_XRC)) > + return -ENOSYS; > + > + mqp = __mlx4_qp_lookup(dev->dev, qp_num); > + if (unlikely(!mqp)) { > + printk(KERN_WARNING "mlx4_ib_reg_xrc_rcv_qp: unknown QPN %06x\n", > + qp_num); > + return -EINVAL; > + } I don't see any locking here... what protects against the QP from being destroyed out from under you in another thread? - R. From vlad at dev.mellanox.co.il Tue Jun 24 00:43:58 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Tue, 24 Jun 2008 10:43:58 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support Message-ID: <4860A5BE.7090608@dev.mellanox.co.il> From 0e3f742e46cec3370582a967283b47953ccd7573 Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Mon, 23 Jun 2008 15:38:27 +0300 Subject: [PATCH] IPOIB: add LRO support. add "ipoib_use_lro" module parameter to enable LRO. add "ipoib_lro_max_aggr" module parameter to set the Max number of packets to be aggregated. LRO statistics accessible through ethtool. Signed-off-by: Eli Cohen Signed-off-by: Vladimir Sokolovsky --- drivers/infiniband/ulp/ipoib/Kconfig | 1 + drivers/infiniband/ulp/ipoib/ipoib.h | 13 ++++++ drivers/infiniband/ulp/ipoib/ipoib_ethtool.c | 50 ++++++++++++++++++++++ drivers/infiniband/ulp/ipoib/ipoib_ib.c | 8 +++- drivers/infiniband/ulp/ipoib/ipoib_main.c | 57 ++++++++++++++++++++++++++ 5 files changed, 128 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/Kconfig b/drivers/infiniband/ulp/ipoib/Kconfig index 1f76bad..691525c 100644 --- a/drivers/infiniband/ulp/ipoib/Kconfig +++ b/drivers/infiniband/ulp/ipoib/Kconfig @@ -1,6 +1,7 @@ config INFINIBAND_IPOIB tristate "IP-over-InfiniBand" depends on NETDEVICES && INET && (IPV6 || IPV6=n) + select INET_LRO ---help--- Support for the IP-over-InfiniBand protocol (IPoIB). This transports IP packets over InfiniBand so you can use your IB diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 8754b36..985ee87 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -50,6 +50,7 @@ #include #include #include +#include /* constants */ @@ -94,6 +95,9 @@ enum { IPOIB_MCAST_FLAG_BUSY = 2, /* joining or already joined */ IPOIB_MCAST_FLAG_ATTACHED = 3, + IPOIB_MAX_LRO_DESCRIPTORS = 8, + IPOIB_LRO_MAX_AGGR = 64, + MAX_SEND_CQE = 16, IPOIB_CM_COPYBREAK = 256, }; @@ -248,6 +252,11 @@ struct ipoib_ethtool_st { u16 max_coalesced_frames; }; +struct ipoib_lro { + struct net_lro_mgr lro_mgr; + struct net_lro_desc lro_desc[IPOIB_MAX_LRO_DESCRIPTORS]; +}; + /* * Device private locking: tx_lock protects members used in TX fast * path (and we use LLTX so upper layers don't do extra locking). @@ -334,6 +343,8 @@ struct ipoib_dev_priv { int hca_caps; struct ipoib_ethtool_st ethtool; struct timer_list poll_timer; + + struct ipoib_lro lro; }; struct ipoib_ah { @@ -679,6 +690,8 @@ extern struct ib_sa_client ipoib_sa_client; #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG extern int ipoib_debug_level; +extern int ipoib_use_lro; +extern int ipoib_lro_max_aggr; #define ipoib_dbg(priv, format, arg...) \ do { \ diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c index 10279b7..79709f0 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c @@ -86,11 +86,61 @@ static int ipoib_set_coalesce(struct net_device *dev, return 0; } +#define IPOIB_STATS_LEN ARRAY_SIZE(ipoib_gstrings_stats) + +static const char ipoib_gstrings_stats[][ETH_GSTRING_LEN] = { + "LRO aggregated", "LRO flushed", + "LRO avg aggr", "LRO no_desc" +}; + +static void +ipoib_get_strings(struct net_device *netdev, u32 stringset, u8 *data) +{ + switch (stringset) { + case ETH_SS_STATS: + memcpy(data, *ipoib_gstrings_stats, + sizeof(ipoib_gstrings_stats)); + data += sizeof(ipoib_gstrings_stats); + break; + } +} + +static int ipoib_get_sset_count(struct net_device *dev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return IPOIB_STATS_LEN; + default: + return -EOPNOTSUPP; + } +} + +static void ipoib_get_ethtool_stats(struct net_device *dev, + struct ethtool_stats *stats, uint64_t *data) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + int index = 0; + + /* Get LRO statistics */ + data[index++] = priv->lro.lro_mgr.stats.aggregated; + data[index++] = priv->lro.lro_mgr.stats.flushed; + if (priv->lro.lro_mgr.stats.flushed) + data[index++] = priv->lro.lro_mgr.stats.aggregated / + priv->lro.lro_mgr.stats.flushed; + else + data[index++] = 0; + data[index++] = priv->lro.lro_mgr.stats.no_desc; + +} + static const struct ethtool_ops ipoib_ethtool_ops = { .get_drvinfo = ipoib_get_drvinfo, .get_tso = ethtool_op_get_tso, .get_coalesce = ipoib_get_coalesce, .set_coalesce = ipoib_set_coalesce, + .get_strings = ipoib_get_strings, + .get_sset_count = ipoib_get_sset_count, + .get_ethtool_stats = ipoib_get_ethtool_stats, }; void ipoib_set_ethtool_ops(struct net_device *dev) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index eca8518..9126975 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -288,7 +288,10 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) if (test_bit(IPOIB_FLAG_CSUM, &priv->flags) && likely(wc->csum_ok)) skb->ip_summed = CHECKSUM_UNNECESSARY; - netif_receive_skb(skb); + if (ipoib_use_lro) + lro_receive_skb(&priv->lro.lro_mgr, skb, 0); + else + netif_receive_skb(skb); repost: if (unlikely(ipoib_ib_post_receive(dev, wr_id))) @@ -448,6 +451,9 @@ poll_more: goto poll_more; } + if (ipoib_use_lro) + lro_flush_all(&priv->lro.lro_mgr); + return done; } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index bfe1dbf..ab6ebd9 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -60,6 +60,17 @@ MODULE_PARM_DESC(send_queue_size, "Number of descriptors in send queue"); module_param_named(recv_queue_size, ipoib_recvq_size, int, 0444); MODULE_PARM_DESC(recv_queue_size, "Number of descriptors in receive queue"); +int ipoib_use_lro __read_mostly; +module_param_named(ipoib_use_lro, ipoib_use_lro, int, 0644); +MODULE_PARM_DESC(ipoib_use_lro, " Large Receive Offload, 1: enable, " + "0: disable, Default = 0"); + +int ipoib_lro_max_aggr __read_mostly = IPOIB_LRO_MAX_AGGR; +module_param_named(ipoib_lro_max_aggr, ipoib_lro_max_aggr, int, 0644); +MODULE_PARM_DESC(ipoib_lro_max_aggr, " LRO: Max packets to be aggregated. " + "Default = 64"); + + #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG int ipoib_debug_level; @@ -936,6 +947,50 @@ static const struct header_ops ipoib_header_ops = { .create = ipoib_hard_header, }; +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, + void **tcph, u64 *hdr_flags, void *priv) +{ + unsigned int ip_len; + struct iphdr *iph; + + if (unlikely(skb->protocol != htons(ETH_P_IP))) + return -1; + + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) + return -1; + + /* non tcp packet */ + skb_reset_network_header(skb); + iph = ip_hdr(skb); + if (iph->protocol != IPPROTO_TCP) + return -1; + + ip_len = ip_hdrlen(skb); + skb_set_transport_header(skb, ip_len); + *tcph = tcp_hdr(skb); + + /* check if ip header and tcp header are complete */ + if (iph->tot_len < ip_len + tcp_hdrlen(skb)) + return -1; + + *hdr_flags = LRO_IPV4 | LRO_TCP; + *iphdr = iph; + + return 0; +} + +static void ipoib_lro_setup(struct ipoib_dev_priv *priv) +{ + priv->lro.lro_mgr.max_aggr = ipoib_lro_max_aggr; + priv->lro.lro_mgr.max_desc = IPOIB_MAX_LRO_DESCRIPTORS; + priv->lro.lro_mgr.lro_arr = priv->lro.lro_desc; + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; + priv->lro.lro_mgr.features = LRO_F_NAPI; + priv->lro.lro_mgr.dev = priv->dev; + priv->lro.lro_mgr.ip_summed = CHECKSUM_UNNECESSARY; + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; +} + static void ipoib_setup(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -975,6 +1030,8 @@ static void ipoib_setup(struct net_device *dev) priv->dev = dev; + ipoib_lro_setup(priv); + spin_lock_init(&priv->lock); spin_lock_init(&priv->tx_lock); -- 1.5.5.1 From cracknk67 at dimambro.com Tue Jun 24 00:59:58 2008 From: cracknk67 at dimambro.com (Jane Ziegler) Date: Tue, 24 Jun 2008 08:59:58 +0100 Subject: [ofa-general] If you can't beat CEOs join them Message-ID: <01c8d5d8$adb21300$21b0a5c1@cracknk67> Bacheelor, MasteerMBA, and Doctoraate diplomas available in the field of your choice that's right, you can even become a Doctor and receive all the benefits that comes with it! Our Diplomas/Certificates are recognised in most countries No required examination, tests, classes, books, or interviews. ** No one is turned down ** Confidentiality assured CALL US 24 HOURS A DAY, 7 DAYS A WEEK For US: 1-501-647-0781 Outside US: +1-501-647-0781 "Just leave your NAME & PHONE NO. (with CountryCode)" in the voicemail our staff will get back to you in next few days -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sumit.Gaur at Sun.COM Tue Jun 24 01:51:40 2008 From: Sumit.Gaur at Sun.COM (Sumit Gaur - Sun Microsystem) Date: Tue, 24 Jun 2008 14:21:40 +0530 Subject: [ofa-general] Is response time of GMP is more than SMP Message-ID: <4860B59C.5010306@Sun.COM> Hi, I am using OFED 2.5.* and observing that my SMI requests are serving very fast and response time is very less on the contrary my GSI requests were served in more time and response time sometime goes more than 2 sec. Any light on this different behavior. Is it expected behavior if yes then why ? Thanks and Regards sumit From vlad at lists.openfabrics.org Tue Jun 24 02:35:04 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 24 Jun 2008 02:35:04 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080624-0200 daily build status Message-ID: <20080624093504.1DC4CE60A24@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-67.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.c:116: error: '__GFP_ZERO' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.c:116: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.c:116: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband/hw/mthca] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-67.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-67.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.c:116: error: '__GFP_ZERO' undeclared (first use in this function) /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.c:116: error: (Each undeclared identifier is reported only once /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.c:116: error: for each function it appears in.) make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/mthca/mthca_memfree.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/hw/mthca] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080624-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From ogerlitz at voltaire.com Tue Jun 24 02:41:36 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 24 Jun 2008 12:41:36 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860A5BE.7090608@dev.mellanox.co.il> References: <4860A5BE.7090608@dev.mellanox.co.il> Message-ID: <4860C150.8070006@voltaire.com> Vladimir Sokolovsky wrote: > add "ipoib_use_lro" module parameter to enable LRO. How about that for this one we'll go on a finer granularity of being able to set it per device through sysfs? > + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; > + priv->lro.lro_mgr.ip_summed = CHECKSUM_UNNECESSARY; > + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; Looking in _lro_proc_skb() which is the code that would be processing ipoib skbs, it seems that the values of lro_mgr.ip_summed and lro_mgr.ip_summed_aggr should correlate to whether the HCA is capable of doing checksum offload or not. Or. From ogerlitz at voltaire.com Tue Jun 24 02:50:10 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 24 Jun 2008 12:50:10 +0300 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: <485A7903.4050407@voltaire.com> References: <485A7903.4050407@voltaire.com> Message-ID: <4860C352.5070603@voltaire.com> Or Gerlitz wrote: >> It becomes a mess if I put patches from Jeff's tree into my tree >> too. What are the dependencies of your patches on pending changes >> for other trees? One simple possibility is just to wait on putting >> these patches into any tree until the prerequisites are upstream. > My patches are only dependent on the three commits to Jeff's tree > which I have stated in the 0/4 email message. They are not dependent > on other patches to the infiniband tree or any other tree. I do want > them to be queued for push into 2.6.27 at some tree. > So one option is that you pull Jeff's davem-next into a branch at your > tree and once Jeff does his push you ask to pull this branch whose > only diff is would be those three patches, I think it possible, and > you did it with libiscsi on the time we pushed iser. If this becomes a > mess, lets just push the four patches to Jeff. > Roland, Please let me know your preference, and if you want us to go through Jeff, whether you want me to push this to him or you want me to do so. Or. From tej at qcindustries.com Tue Jun 24 03:10:33 2008 From: tej at qcindustries.com (Caitlin Mclaughlin) Date: Tue, 24 Jun 2008 11:10:33 +0100 Subject: [ofa-general] Luxury Message-ID: <01c8d5ea$eb7d9a80$61a8e552@tej> We are now bringing forward the replicas of the watches,Accessories, boxsets, Cufflinks, Keychains, Lighters, Pens, Tiffany & CoJewelry. A replica presupposes the exact copy of the original, that is thequality and all the other features the original possesses. For instance,youmight get a $18000 Rolex replica for a $229.00 price only! Our replica is ofsuch an amazing quality that it can be destinguished from the original only inthe jewellery workshop. You're anxiously wanting to be glamorous enough fordemanding high soiciety standarts, but your earnings are not that sky high? Youare willing to reward your lover with an outrageous gift but you don't wantthat to empty your wallet? Come, visit our shop! http://brilkean[dot]com/ Note! This link is unclickable! Copy this link and paste into your browser.When pasting don’t forget to change [dot] for ‘.’ P.S When applying for a well-paid job, when flirting with agirl, when asking for a credit in the bank you get your chances doubled with ARolex President replica on your hand! PSS: The only missing thing here is a luxury suit! menireoplis poBqsyfpm7D7BqBmzlE -------------- next part -------------- An HTML attachment was scrubbed... URL: From monis at Voltaire.COM Tue Jun 24 04:20:00 2008 From: monis at Voltaire.COM (Moni Shoua) Date: Tue, 24 Jun 2008 14:20:00 +0300 Subject: [ofa-general] [PATCH] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: References: <484F8A20.9070101@Voltaire.COM> Message-ID: <4860D860.1080805@Voltaire.COM> Roland Dreier wrote: > Please run your patches through scripts/checkpatch.pl and consider the > output before sending them... it's easy to do and saves everyone time > fixing trivial style mistakes. > I did ran the script on the patch and besides one WARNING for a line that exceeds 80 columns which I believe is allowed sometimes. > More substantive comments: > > > +static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int level) > > Instead of "int level" here, I would prefer to see something like > > enum ipoib_flush_level { > IPOIB_FLUSH_LIGHT, > IPOIB_FLUSH_NORMAL, > IPOIB_FLUSH_HEAVY, > }; > > so that code like > > > - if (pkey_event) { > > + if (level == 2) { > > remains at somewhat self-documenting. OK > > > +static void path_refresh(struct net_device *dev, struct ipoib_path *path) > > +{ > > + struct ipoib_dev_priv *priv = netdev_priv(dev); > > + ipoib_dbg(priv, "mark path LID 0x%04x GID " IPOIB_GID_FMT " invalid\n", > > + be16_to_cpu(path->pathrec.dlid), IPOIB_GID_ARG(path->pathrec.dgid)); > > + path->valid = 0; > > +} > > I'm not sure this wrapper makes sense, because it is so trivial and it > doesn't even do what the name suggests -- it doesn't refresh the path, > it just marks it as invalid. > I'll get rid of the wrapper. Please see below > This looks broken for a couple of reasons: > > - What protects against paths being added to the path list while this > loop is running? ipoib_flush_paths() moves the list to a new place > to avoid this problem. Actually if I take out the spin_unlock() from inside of the loop I am protected because adding a patch can only be started from the xmit function that takes the tx_lock. I let myself to give away spin_unlock because it was redundant from the start, we are only marking the path as invalid here. > - What happens if you mark a path as invalid and do > wait_for_completion(), and then ipoib_flush_paths() runs before > another path record query is done? It seems that the second > wait_for_completion() will wait forever. Again, you are right but wait_for_completion() is also not needed here so I let it go. Finally,the function looks like this now (I also changed it's name to make it more accurate) +void ipoib_mark_paths_invalid(struct net_device *dev) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + struct ipoib_path *path, *tp; + spin_lock_irq(&priv->tx_lock); + spin_lock(&priv->lock); + + list_for_each_entry_safe(path, tp, &priv->path_list, list) { + ipoib_dbg(priv, "mark path LID 0x%04x GID " IPOIB_GID_FMT " invalid\n", + be16_to_cpu(path->pathrec.dlid), + IPOIB_GID_ARG(path->pathrec.dgid)); + path->valid = 0; + } + spin_unlock(&priv->lock); + spin_unlock_irq(&priv->tx_lock); +} + > > - R. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hrosenstock at xsigo.com Tue Jun 24 04:48:27 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 04:48:27 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080623202321.GK29198@obsidianresearch.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <1214251804.13056.247.camel@hrosenstock-ws.xsigo.com> <20080623202321.GK29198@obsidianresearch.com> Message-ID: <1214308107.13056.261.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-23 at 14:23 -0600, Jason Gunthorpe wrote: > Well, who's OUI is the vendor ID? The chip makers or the board makers? > That has never really seemed clear to me, and the spec doesn't > say. As you point out, that isn't stated in the spec. In practice, the OUI used for vendor ID relates to the chip. I suppose that might not be the case if there were some uniquification needed to be detected by the driver. The board vendor's OUI is used in the GUIDs. -- Hal From hrosenstock at xsigo.com Tue Jun 24 04:54:26 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 04:54:26 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080623204103.GL29198@obsidianresearch.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> <20080623204103.GL29198@obsidianresearch.com> Message-ID: <1214308466.13056.263.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-23 at 14:41 -0600, Jason Gunthorpe wrote: > Personally, I feel the intention of the spec was to follow the PCI > model, it does introduce subsystem ID's for the IO controller > stuff, for instance. Where would the subsystem IDs go ? -- Hal From vlad at dev.mellanox.co.il Tue Jun 24 05:07:24 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Tue, 24 Jun 2008 15:07:24 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860C150.8070006@voltaire.com> References: <4860A5BE.7090608@dev.mellanox.co.il> <4860C150.8070006@voltaire.com> Message-ID: <4860E37C.20502@dev.mellanox.co.il> Or Gerlitz wrote: > Vladimir Sokolovsky wrote: >> add "ipoib_use_lro" module parameter to enable LRO. > How about that for this one we'll go on a finer granularity of being > able to set it per device through sysfs? Yes, we thought about this and I am planning to add this granularity later, in a different patch. > >> + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; >> + priv->lro.lro_mgr.ip_summed = CHECKSUM_UNNECESSARY; >> + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; > Looking in _lro_proc_skb() which is the code that would be processing > ipoib skbs, it seems that the values of lro_mgr.ip_summed and > lro_mgr.ip_summed_aggr should correlate to whether the HCA is capable of > doing checksum offload or not. > > Or. > +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, + void **tcph, u64 *hdr_flags, void *priv) +{ ... + + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) + return -1; This will return -1 if HCA is not capable of doing checksum offload. Regards, Vladimir From makc at sgi.com Tue Jun 24 05:16:08 2008 From: makc at sgi.com (Max Matveev) Date: Tue, 24 Jun 2008 22:16:08 +1000 Subject: [ofa-general] [PATCH] allow explicit HCA selection by functions from libibmad In-Reply-To: <20080620145722.GH3158@sashak.voltaire.com> References: <18522.29300.545966.523800@kuku.melbourne.sgi.com> <20080620134622.GG3158@sashak.voltaire.com> <18523.47597.85812.745257@kuku.melbourne.sgi.com> <20080620145722.GH3158@sashak.voltaire.com> Message-ID: <18528.58760.868810.615990@kuku.melbourne.sgi.com> I'm writing a data collection agent for PCP (http://oss.sgi.com/projects/pcp) which needs to pull performance counters and port information from the mulitple HCAs and/or switches on a subnet. What's more, a single host could be connected to multiple disjoined fabrics so the choice of HCA port to send MAD rpcs on becomes important. I could write my own library for sending MADs but there is one already which does 90% of what I need, I just need to be add a few extra entry points. The attached patch against ~sashak/management.git does that for me. max -------------- next part -------------- A non-text attachment was scrubbed... Name: patch.ofed Type: text/patch Size: 18127 bytes Desc: not available URL: From ogerlitz at voltaire.com Tue Jun 24 02:44:54 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 24 Jun 2008 12:44:54 +0300 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH 2/3 v2] mlx4: multicast cast loopback block In-Reply-To: References: Message-ID: <4860C216.3010400@voltaire.com> Roland Dreier wrote: > Please run your patches through scripts/checkpatch.pl and consider the output before sending them... it's easy to do and saves everyone time fixing trivial style mistakes. Roland, Ron will be here later this week and then he can make the check and resend. > Am I missing something? Is there a reason not to write this as: > > return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, > &to_mqp(ibqp)->mqp, gid->raw, > !!(to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK)); personally, I find !! code somehow hard to read, but if this is the common practice in such cases, we can fix that... Or. From ogerlitz at voltaire.com Tue Jun 24 05:37:55 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 24 Jun 2008 15:37:55 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860E37C.20502@dev.mellanox.co.il> References: <4860A5BE.7090608@dev.mellanox.co.il> <4860C150.8070006@voltaire.com> <4860E37C.20502@dev.mellanox.co.il> Message-ID: <4860EAA3.6010108@voltaire.com> Vladimir Sokolovsky wrote: > +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, > + void **tcph, u64 *hdr_flags, void *priv) > +{ > > ... > > + > + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) > + return -1; > > This will return -1 if HCA is not capable of doing checksum offload. > Yes, but in that case __lro_proc_skb() would use the value of lro_mgr->ip_summed which you set independently of the HCA capabilities, see below the code at the "out" label. Or > static int __lro_proc_skb(struct net_lro_mgr *lro_mgr, struct sk_buff *skb, > struct vlan_group *vgrp, u16 vlan_tag, void *priv) > { > ... > if (!lro_mgr->get_skb_header > || lro_mgr->get_skb_header(skb, (void *)&iph, (void *)&tcph, > &flags, priv)) > goto out; > ... > > > out2: /* send aggregated SKBs to stack */ > lro_flush(lro_mgr, lro_desc); > > out: /* Original SKB has to be posted to stack */ > skb->ip_summed = lro_mgr->ip_summed; > return 1; > } > > > From hrosenstock at xsigo.com Tue Jun 24 05:54:26 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 05:54:26 -0700 Subject: [ofa-general] [PATCH] opensm/osm_multicast.h: Remove no longer needed element in osm_mgrp structure Message-ID: <1214312066.13056.268.camel@hrosenstock-ws.xsigo.com> opensm/osm_multicast.h: Remove no longer needed element in osm_mgrp structure Dependent on "Convert mgrp_mlid_tbl into array" patch Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_multicast.h b/opensm/include/opensm/osm_multicast.h index c0bd16e..d1e0990 100644 --- a/opensm/include/opensm/osm_multicast.h +++ b/opensm/include/opensm/osm_multicast.h @@ -153,7 +153,6 @@ typedef struct osm_mcast_mgr_ctxt { * SYNOPSIS */ typedef struct osm_mgrp { - cl_map_item_t map_item; ib_net16_t mlid; osm_mtree_node_t *p_root; cl_qmap_t mcm_port_tbl; @@ -165,9 +164,6 @@ typedef struct osm_mgrp { } osm_mgrp_t; /* * FIELDS -* map_item -* Map Item for qmap linkage. Must be first element!! -* * mlid * The network ordered LID of this Multicast Group (must be * >= 0xC000). From sashak at voltaire.com Tue Jun 24 06:09:50 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 16:09:50 +0300 Subject: [ofa-general] Re: [OPENSM PATCH 0/3]: "guid-routing-order" option for minhop/updn routing In-Reply-To: <1214252698.5369.537.camel@cardanus.llnl.gov> References: <1214252698.5369.537.camel@cardanus.llnl.gov> Message-ID: <20080624130950.GL7341@sashak.voltaire.com> Hi Al, On 13:24 Mon 23 Jun , Al Chu wrote: > > Here's the adjusted patches, as discussed in the previous thread. It > now supports both minhop and updn. > > I wasn't sure of what the cleanest way to make > __osm_ucast_mgr_process_tbl() usable by both the original and ordered- > routing options. PLMK what you think in patch 2/3. That is fine as it is now. Soon I will also need to add sorting by leaf switches (as we discussed), likely will need to rework some parts for this. All three patches applied. Thanks. Sasha From eli at mellanox.co.il Tue Jun 24 06:28:47 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Tue, 24 Jun 2008 16:28:47 +0300 Subject: [ofa-general] Re: [RFC] Receive Core Afinnity (aka RSS) support In-Reply-To: References: <1213628310.5447.91.camel@mtls03> Message-ID: <1214314127.23583.26.camel@mtls03> Roland, how about the following approach. We will have a single function at the IB core that will create a list of QPs for use with RSS. These QPs will be used as the receive queues while the hw driver will take care of all the underlying requirements to implement RSS. The function will meet the needs of any RSS implementation that is needed. For example, the follwing function can be used: /** * ib_create_rca_qps - Creates a range of QPs associated with the * specified protection domain. * @pd: The protection domain associated with the QP. * @qp_init_attr: A list of initial attributes required to create the * QPs. If QP creation succeeds, then the attributes are updated to * the actual capabilities of the created QP. * @nqps: The number of required QPs * @list: pointer to an array to hold allocated QPs * @default_qp: the QP that will handle all received packets that are * not mapped to any of the other QPs */ int ib_create_rca_qps(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr, int nqps, struct ib_qp *list[], struct ib_qp *default_qp); For IPoIB I suggest to use the UD qp for default_qp; -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From swise at opengridcomputing.com Tue Jun 24 06:32:22 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 24 Jun 2008 08:32:22 -0500 Subject: [ofa-general] Re: [PATCH] RDMA/cxgb3: Remove write-only iwch_rnic_attributes fields In-Reply-To: References: Message-ID: <4860F766.7070301@opengridcomputing.com> Roland Dreier wrote: > The members struct iwch_rnic_attributes.vendor_id and .vendor_part_id > are write-only, so we might as well get rid of them. > > Signed-off-by: Roland Dreier > Acked-by: Steve Wise From eli at mellanox.co.il Tue Jun 24 06:42:38 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Tue, 24 Jun 2008 16:42:38 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860EAA3.6010108@voltaire.com> References: <4860A5BE.7090608@dev.mellanox.co.il> <4860C150.8070006@voltaire.com> <4860E37C.20502@dev.mellanox.co.il> <4860EAA3.6010108@voltaire.com> Message-ID: <1214314958.23583.32.camel@mtls03> On Tue, 2008-06-24 at 15:37 +0300, Or Gerlitz wrote: > Vladimir Sokolovsky wrote: > > +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, > > + void **tcph, u64 *hdr_flags, void *priv) > > +{ > > > > ... > > > > + > > + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) > > + return -1; > > > > This will return -1 if HCA is not capable of doing checksum offload. > > > Yes, but in that case __lro_proc_skb() would use the value of > lro_mgr->ip_summed which you set independently of the HCA capabilities, > see below the code at the "out" label. > You're right about your observation. But I can't find a reason why we have this statement at all: out: /* Original SKB has to be posted to stack */ skb->ip_summed = lro_mgr->ip_summed; return 1; It seems that the skb's ip_summed field is not changed in the path that leads to this label so why changing this filed to something that was initialized globally? Maybe one of the authors can comment on that? -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From ogerlitz at voltaire.com Tue Jun 24 06:51:50 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 24 Jun 2008 16:51:50 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860A5BE.7090608@dev.mellanox.co.il> References: <4860A5BE.7090608@dev.mellanox.co.il> Message-ID: <4860FBF6.5030200@voltaire.com> Hi Eli, Vlad Do you have any performance data that you can share? specifically, assuming I use connectx such that I have already checksum and LSO offloads working for the datagram mode and 2K MTU. What would be the advantage added by LRO? would it be only to the CPU utilization or there's also better BW? Or. From THEMANN at de.ibm.com Tue Jun 24 07:04:42 2008 From: THEMANN at de.ibm.com (Jan-Bernd Themann) Date: Tue, 24 Jun 2008 16:04:42 +0200 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <1214314958.23583.32.camel@mtls03> Message-ID: Hi, you are right, this can be removed. Feel free to post a patch. Regards, Jan-Bernd "Eli Cohen" wrote on 24.06.2008 15:42:38: > On Tue, 2008-06-24 at 15:37 +0300, Or Gerlitz wrote: > > Vladimir Sokolovsky wrote: > > > +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, > > > + void **tcph, u64 *hdr_flags, void *priv) > > > +{ > > > > > > ... > > > > > > + > > > + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) > > > + return -1; > > > > > > This will return -1 if HCA is not capable of doing checksum offload. > > > > > Yes, but in that case __lro_proc_skb() would use the value of > > lro_mgr->ip_summed which you set independently of the HCA capabilities, > > see below the code at the "out" label. > > > You're right about your observation. But I can't find a reason why we > have this statement at all: > > out: /* Original SKB has to be posted to stack */ > skb->ip_summed = lro_mgr->ip_summed; > return 1; > > It seems that the skb's ip_summed field is not changed in the path that > leads to this label so why changing this filed to something that was > initialized globally? > > Maybe one of the authors can comment on that? > > > > > > -- > Eli Cohen > Mellanox Technologies LTD. > SW- Linux > Phone: +972 (4) 909 7200 (ext 276) > Mobile: +972 (54) 650 1206 > E-mail: eli at mellanox.co.il > > ---------------------------------------------------------------------- > Emails belong on computers, trees belong in forests; if you must print > this, do it on recycled paper. > http://www.greenpeace.org/international/ > ---------------------------------------------------------------------- > > > Disclaimer added by CodeTwo Exchange Rules > http://www.codetwo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Tue Jun 24 07:27:13 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 07:27:13 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/complib/cl_fleximap.h: Improve cl_pfn_fmap_cmp_t PARAMETERS comments Message-ID: <1214317633.13056.274.camel@hrosenstock-ws.xsigo.com> opensm/include/complib/cl_fleximap.h: Improve cl_pfn_fmap_cmp_t PARAMETERS comments Signed-off-by: Hal Rosenstock diff --git a/opensm/include/complib/cl_fleximap.h b/opensm/include/complib/cl_fleximap.h index 125a0ea..0af8766 100644 --- a/opensm/include/complib/cl_fleximap.h +++ b/opensm/include/complib/cl_fleximap.h @@ -194,8 +194,8 @@ typedef intn_t * * RETURN VALUE * Returns 0 if the keys match. -* Returns less than 0 if p_key1 is less than p_key2. -* Returns greater than 0 if p_key1 is greater than p_key2. +* Returns less than 0 if *p_key1 is less than *p_key2. +* Returns greater than 0 if *p_key1 is greater than *p_key2. * * NOTES * This function type is provided as function prototype reference for the From eli at dev.mellanox.co.il Tue Jun 24 07:39:46 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 24 Jun 2008 17:39:46 +0300 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able Message-ID: <1214318386.23583.37.camel@mtls03> >From 59c28d377ddfb7c13e2e1ecbf184211704d0e61d Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Tue, 24 Jun 2008 17:17:14 +0300 Subject: [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able When an SKB cannot be chained to a session, the current code attempts to "restore" its ip_summed field from lro_mgr->ip_summed. However, lro_mgr->ip_summed does not hold the original value; in fact, we'd better not touch skb->ip_summed since it is not modified by the code in the path leading to a failure to chain it. Issue raised by Or Gerlitz Signed-off-by: Eli Cohen --- net/ipv4/inet_lro.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/net/ipv4/inet_lro.c b/net/ipv4/inet_lro.c index 4a4d49f..cfd034a 100644 --- a/net/ipv4/inet_lro.c +++ b/net/ipv4/inet_lro.c @@ -383,8 +383,7 @@ static int __lro_proc_skb(struct net_lro_mgr *lro_mgr, struct sk_buff *skb, out2: /* send aggregated SKBs to stack */ lro_flush(lro_mgr, lro_desc); -out: /* Original SKB has to be posted to stack */ - skb->ip_summed = lro_mgr->ip_summed; +out: return 1; } -- 1.5.6 From THEMANN at de.ibm.com Tue Jun 24 07:45:18 2008 From: THEMANN at de.ibm.com (Jan-Bernd Themann) Date: Tue, 24 Jun 2008 16:45:18 +0200 Subject: [ofa-general] Re: [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <1214318386.23583.37.camel@mtls03> Message-ID: Eli Cohen wrote on 24.06.2008 16:39:46: > >From 59c28d377ddfb7c13e2e1ecbf184211704d0e61d Mon Sep 17 00:00:00 2001 > From: Eli Cohen > Date: Tue, 24 Jun 2008 17:17:14 +0300 > Subject: [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able > > When an SKB cannot be chained to a session, the current code attempts > to "restore" its ip_summed field from lro_mgr->ip_summed. However, > lro_mgr->ip_summed does not hold the original value; in fact, we'd > better not touch skb->ip_summed since it is not modified by the code > in the path leading to a failure to chain it. > > Issue raised by Or Gerlitz > > Signed-off-by: Eli Cohen > --- > net/ipv4/inet_lro.c | 3 +-- > 1 files changed, 1 insertions(+), 2 deletions(-) > > diff --git a/net/ipv4/inet_lro.c b/net/ipv4/inet_lro.c > index 4a4d49f..cfd034a 100644 > --- a/net/ipv4/inet_lro.c > +++ b/net/ipv4/inet_lro.c > @@ -383,8 +383,7 @@ static int __lro_proc_skb(struct net_lro_mgr > *lro_mgr, struct sk_buff *skb, > out2: /* send aggregated SKBs to stack */ > lro_flush(lro_mgr, lro_desc); > > -out: /* Original SKB has to be posted to stack */ > - skb->ip_summed = lro_mgr->ip_summed; > +out: > return 1; > } > > -- > 1.5.6 > > > looks good, thanks! Acked-by: Jan-Bernd Themann -------------- next part -------------- An HTML attachment was scrubbed... URL: From vtuqmvlss at bobhatton.com Tue Jun 24 08:05:55 2008 From: vtuqmvlss at bobhatton.com (Elvia Luna) Date: Tue, 24 Jun 2008 17:05:55 +0200 Subject: [ofa-general] Re: 176186 Message-ID: <01c8d61c$90649b80$aa4e515c@vtuqmvlss> Take more from this world! Ultimate effect! http://www.poieyr.cn/v/ From hrosenstock at xsigo.com Tue Jun 24 08:47:09 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 08:47:09 -0700 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <4860B59C.5010306@Sun.COM> References: <4860B59C.5010306@Sun.COM> Message-ID: <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> Hi Sumit, On Tue, 2008-06-24 at 14:21 +0530, Sumit Gaur - Sun Microsystem wrote: > Hi, > I am using OFED 2.5.* ^^^^^ 1.2.5.* ? > and observing that my SMI requests are serving very fast > and response time is very less on the contrary my GSI requests were served in > more time and response time sometime goes more than 2 sec. Any light on this > different behavior. What are the specific GS requests which are slow in response ? Are they compute intensive ? In general, there are a few possibilities (which can cause this). SM traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most subnets). Some possibilities are: 1. Timeout/retry being hit for some GS traffic (GS request or response lost/corrupted) 2. Data VL busy (is there anything else utilizing VL0 ?) 3. CPU busy (what other things are competing for CPU on the responder ?) -- Hal > Is it expected behavior if yes then why ? > > Thanks and Regards > sumit > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From suri at baymicrosystems.com Tue Jun 24 09:01:28 2008 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Tue, 24 Jun 2008 12:01:28 -0400 Subject: [ofa-general] performance drop for datagram mode with the newconnectx FW In-Reply-To: <20080623114236.GA11852@mtls03> References: <20080623114236.GA11852@mtls03> Message-ID: I also get about 650MB for datagram mode with 2.3 FW. > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf > Of Eli Cohen > Sent: Monday, June 23, 2008 7:43 AM > To: Or Gerlitz > Cc: Eli Cohen; Roland Dreier; general at lists.openfabrics.org > Subject: Re: [ofa-general] performance drop for datagram mode with the newconnectx FW > > On Mon, Jun 23, 2008 at 02:36:39PM +0300, Or Gerlitz wrote: > > Eli, > > > > Using the new connectx FW (2.5), I see performance drop to almost > > zero with ipoib datagram mode. The code that runs on these systems > > is ofed 1.3 and not mainline kernel, details below. > > > > Running netperf With connected mode (64k MTU) I get about 950MB/s > > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to > > see about 650MB/s and above with FW 2.3 and datagram mode. Not that > > it could explain the drop, but the NIC reports to the OS stateless > > offload support - /sys/class/net/ib1/features is 0x11423 > > > > I have opened the ipoib and mlx4 debug prints, and I don't see anything > > special other then the dmesg get quite filled with > > > > ib1: TX ring full, stopping kernel net queue > > > > any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which > > is about five times what I see with ipoib. > > > > > > Or. > > > > git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel > > commit 564e9e9383272f4311fd87ff4e5447cfcebad73a > > > > # uname -a > > Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > > > # cat /etc/redhat-release > > Red Hat Enterprise Linux Server release 5.1 (Tikanga) > > Can you tell if changing the FW to 2.3 gives more reasonable results? > I don't believe such a drop in performance would have passed the QA > tests but I'll check that. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From Abdel.Sadek at lsi.com Tue Jun 24 09:18:10 2008 From: Abdel.Sadek at lsi.com (Sadek, Abdel) Date: Tue, 24 Jun 2008 10:18:10 -0600 Subject: [ofa-general] OFED 1.3 with Lustre 1.6 Message-ID: Hello; Are there any instructions on how to get Lustre 1.6.4.3 to work with OFED 1.3? Thanks. Abdel.. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at dev.mellanox.co.il Tue Jun 24 09:25:15 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Tue, 24 Jun 2008 19:25:15 +0300 Subject: [ofa-general] [PATCH v2 for-2.6.27] IPOIB: add LRO support. Message-ID: <48611FEB.1030709@dev.mellanox.co.il> From 89aa9ca5a7866a542807af2c66fbbaa96c1e0672 Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Tue, 24 Jun 2008 19:20:41 +0300 Subject: [PATCH] IPOIB: add LRO support. add "ipoib_use_lro" module parameter to enable LRO. add "ipoib_lro_max_aggr" module parameter to set the Max number of packets to be aggregated. LRO statistics accessible through ethtool. Signed-off-by: Vladimir Sokolovsky Signed-off-by: Eli Cohen --- Changes from v1: added use_lro parameter per device through sysfs. drivers/infiniband/ulp/ipoib/Kconfig | 1 + drivers/infiniband/ulp/ipoib/ipoib.h | 15 ++++ drivers/infiniband/ulp/ipoib/ipoib_ethtool.c | 50 ++++++++++++++ drivers/infiniband/ulp/ipoib/ipoib_ib.c | 8 ++- drivers/infiniband/ulp/ipoib/ipoib_main.c | 94 ++++++++++++++++++++++++++ 5 files changed, 167 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/Kconfig b/drivers/infiniband/ulp/ipoib/Kconfig index 1f76bad..691525c 100644 --- a/drivers/infiniband/ulp/ipoib/Kconfig +++ b/drivers/infiniband/ulp/ipoib/Kconfig @@ -1,6 +1,7 @@ config INFINIBAND_IPOIB tristate "IP-over-InfiniBand" depends on NETDEVICES && INET && (IPV6 || IPV6=n) + select INET_LRO ---help--- Support for the IP-over-InfiniBand protocol (IPoIB). This transports IP packets over InfiniBand so you can use your IB diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 8754b36..3aae34d 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -50,6 +50,7 @@ #include #include #include +#include /* constants */ @@ -94,6 +95,9 @@ enum { IPOIB_MCAST_FLAG_BUSY = 2, /* joining or already joined */ IPOIB_MCAST_FLAG_ATTACHED = 3, + IPOIB_MAX_LRO_DESCRIPTORS = 8, + IPOIB_LRO_MAX_AGGR = 64, + MAX_SEND_CQE = 16, IPOIB_CM_COPYBREAK = 256, }; @@ -248,6 +252,12 @@ struct ipoib_ethtool_st { u16 max_coalesced_frames; }; +struct ipoib_lro { + int use_lro; + struct net_lro_mgr lro_mgr; + struct net_lro_desc lro_desc[IPOIB_MAX_LRO_DESCRIPTORS]; +}; + /* * Device private locking: tx_lock protects members used in TX fast * path (and we use LLTX so upper layers don't do extra locking). @@ -334,6 +344,8 @@ struct ipoib_dev_priv { int hca_caps; struct ipoib_ethtool_st ethtool; struct timer_list poll_timer; + + struct ipoib_lro lro; }; struct ipoib_ah { @@ -417,6 +429,7 @@ static inline void ipoib_put_ah(struct ipoib_ah *ah) int ipoib_open(struct net_device *dev); int ipoib_add_pkey_attr(struct net_device *dev); int ipoib_add_umcast_attr(struct net_device *dev); +int ipoib_add_use_lro_attr(struct net_device *dev); void ipoib_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_ah *address, u32 qpn); @@ -679,6 +692,8 @@ extern struct ib_sa_client ipoib_sa_client; #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG extern int ipoib_debug_level; +extern int ipoib_use_lro; +extern int ipoib_lro_max_aggr; #define ipoib_dbg(priv, format, arg...) \ do { \ diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c index 10279b7..79709f0 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c @@ -86,11 +86,61 @@ static int ipoib_set_coalesce(struct net_device *dev, return 0; } +#define IPOIB_STATS_LEN ARRAY_SIZE(ipoib_gstrings_stats) + +static const char ipoib_gstrings_stats[][ETH_GSTRING_LEN] = { + "LRO aggregated", "LRO flushed", + "LRO avg aggr", "LRO no_desc" +}; + +static void +ipoib_get_strings(struct net_device *netdev, u32 stringset, u8 *data) +{ + switch (stringset) { + case ETH_SS_STATS: + memcpy(data, *ipoib_gstrings_stats, + sizeof(ipoib_gstrings_stats)); + data += sizeof(ipoib_gstrings_stats); + break; + } +} + +static int ipoib_get_sset_count(struct net_device *dev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return IPOIB_STATS_LEN; + default: + return -EOPNOTSUPP; + } +} + +static void ipoib_get_ethtool_stats(struct net_device *dev, + struct ethtool_stats *stats, uint64_t *data) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + int index = 0; + + /* Get LRO statistics */ + data[index++] = priv->lro.lro_mgr.stats.aggregated; + data[index++] = priv->lro.lro_mgr.stats.flushed; + if (priv->lro.lro_mgr.stats.flushed) + data[index++] = priv->lro.lro_mgr.stats.aggregated / + priv->lro.lro_mgr.stats.flushed; + else + data[index++] = 0; + data[index++] = priv->lro.lro_mgr.stats.no_desc; + +} + static const struct ethtool_ops ipoib_ethtool_ops = { .get_drvinfo = ipoib_get_drvinfo, .get_tso = ethtool_op_get_tso, .get_coalesce = ipoib_get_coalesce, .set_coalesce = ipoib_set_coalesce, + .get_strings = ipoib_get_strings, + .get_sset_count = ipoib_get_sset_count, + .get_ethtool_stats = ipoib_get_ethtool_stats, }; void ipoib_set_ethtool_ops(struct net_device *dev) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index eca8518..009b862 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -288,7 +288,10 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) if (test_bit(IPOIB_FLAG_CSUM, &priv->flags) && likely(wc->csum_ok)) skb->ip_summed = CHECKSUM_UNNECESSARY; - netif_receive_skb(skb); + if (priv->lro.use_lro) + lro_receive_skb(&priv->lro.lro_mgr, skb, 0); + else + netif_receive_skb(skb); repost: if (unlikely(ipoib_ib_post_receive(dev, wr_id))) @@ -448,6 +451,9 @@ poll_more: goto poll_more; } + if (priv->lro.use_lro) + lro_flush_all(&priv->lro.lro_mgr); + return done; } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index bfe1dbf..f8cce51 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -60,6 +60,17 @@ MODULE_PARM_DESC(send_queue_size, "Number of descriptors in send queue"); module_param_named(recv_queue_size, ipoib_recvq_size, int, 0444); MODULE_PARM_DESC(recv_queue_size, "Number of descriptors in receive queue"); +int ipoib_use_lro __read_mostly; +module_param_named(ipoib_use_lro, ipoib_use_lro, int, 0644); +MODULE_PARM_DESC(ipoib_use_lro, " Large Receive Offload, 1: enable, " + "0: disable, Default = 0"); + +int ipoib_lro_max_aggr __read_mostly = IPOIB_LRO_MAX_AGGR; +module_param_named(ipoib_lro_max_aggr, ipoib_lro_max_aggr, int, 0644); +MODULE_PARM_DESC(ipoib_lro_max_aggr, " LRO: Max packets to be aggregated. " + "Default = 64"); + + #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG int ipoib_debug_level; @@ -936,6 +947,85 @@ static const struct header_ops ipoib_header_ops = { .create = ipoib_hard_header, }; +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, + void **tcph, u64 *hdr_flags, void *priv) +{ + unsigned int ip_len; + struct iphdr *iph; + + if (unlikely(skb->protocol != htons(ETH_P_IP))) + return -1; + + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) + return -1; + + /* non tcp packet */ + skb_reset_network_header(skb); + iph = ip_hdr(skb); + if (iph->protocol != IPPROTO_TCP) + return -1; + + ip_len = ip_hdrlen(skb); + skb_set_transport_header(skb, ip_len); + *tcph = tcp_hdr(skb); + + /* check if ip header and tcp header are complete */ + if (iph->tot_len < ip_len + tcp_hdrlen(skb)) + return -1; + + *hdr_flags = LRO_IPV4 | LRO_TCP; + *iphdr = iph; + + return 0; +} + +static ssize_t show_use_lro(struct device *d, struct device_attribute *attr, + char *buf) +{ + struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(d)); + + if (priv->lro.use_lro) + return sprintf(buf, "yes\n"); + else + return sprintf(buf, "no\n"); +} + +static ssize_t set_use_lro(struct device *d, struct device_attribute *attr, + const char *buf, size_t count) +{ + struct net_device *dev = to_net_dev(d); + struct ipoib_dev_priv *priv = netdev_priv(dev); + + if (!strcmp(buf, "yes\n")) + priv->lro.use_lro = 1; + else if (!strcmp(buf, "no\n")) + priv->lro.use_lro = 0; + else + return -EINVAL; + + return count; +} + +static DEVICE_ATTR(use_lro, S_IWUSR | S_IRUGO, show_use_lro, set_use_lro); + +int ipoib_add_use_lro_attr(struct net_device *dev) +{ + return device_create_file(&dev->dev, &dev_attr_use_lro); +} + +static void ipoib_lro_setup(struct ipoib_dev_priv *priv) +{ + priv->lro.use_lro = ipoib_use_lro; + priv->lro.lro_mgr.max_aggr = ipoib_lro_max_aggr; + priv->lro.lro_mgr.max_desc = IPOIB_MAX_LRO_DESCRIPTORS; + priv->lro.lro_mgr.lro_arr = priv->lro.lro_desc; + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; + priv->lro.lro_mgr.features = LRO_F_NAPI; + priv->lro.lro_mgr.dev = priv->dev; + priv->lro.lro_mgr.ip_summed = CHECKSUM_UNNECESSARY; + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; +} + static void ipoib_setup(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -975,6 +1065,8 @@ static void ipoib_setup(struct net_device *dev) priv->dev = dev; + ipoib_lro_setup(priv); + spin_lock_init(&priv->lock); spin_lock_init(&priv->tx_lock); @@ -1204,6 +1296,8 @@ static struct net_device *ipoib_add_port(const char *format, goto sysfs_failed; if (ipoib_add_umcast_attr(priv->dev)) goto sysfs_failed; + if (ipoib_add_use_lro_attr(priv->dev)) + goto sysfs_failed; if (device_create_file(&priv->dev->dev, &dev_attr_create_child)) goto sysfs_failed; if (device_create_file(&priv->dev->dev, &dev_attr_delete_child)) -- 1.5.5.1 From sashak at voltaire.com Tue Jun 24 10:40:45 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 20:40:45 +0300 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624174045.GO7341@sashak.voltaire.com> On 08:47 Tue 24 Jun , Hal Rosenstock wrote: > > Some possibilities are: > 1. Timeout/retry being hit for some GS traffic (GS request or response > lost/corrupted) > 2. Data VL busy (is there anything else utilizing VL0 ?) > 3. CPU busy (what other things are competing for CPU on the responder ?) And also routing itself if comparing direct routed SM against lid routed GS. Sasha From jgunthorpe at obsidianresearch.com Tue Jun 24 10:45:42 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 24 Jun 2008 11:45:42 -0600 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <1214308466.13056.263.camel@hrosenstock-ws.xsigo.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> <20080623204103.GL29198@obsidianresearch.com> <1214308466.13056.263.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624174542.GO29198@obsidianresearch.com> On Tue, Jun 24, 2008 at 04:54:26AM -0700, Hal Rosenstock wrote: > On Mon, 2008-06-23 at 14:41 -0600, Jason Gunthorpe wrote: > > Personally, I feel the intention of the spec was to follow the PCI > > model, it does introduce subsystem ID's for the IO controller > > stuff, for instance. > > Where would the subsystem IDs go ? This is the stuff in the IOControllerProfile (pg 1013) Jason From hrosenstock at xsigo.com Tue Jun 24 10:51:03 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 10:51:03 -0700 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <20080624174542.GO29198@obsidianresearch.com> References: <9B053F82-9DFF-4DB0-BA46-EB6EFE1B5BDE@cisco.com> <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> <20080623204103.GL29198@obsidianresearch.com> <1214308466.13056.263.camel@hrosenstock-ws.xsigo.com> <20080624174542.GO29198@obsidianresearch.com> Message-ID: <1214329863.13056.301.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-24 at 11:45 -0600, Jason Gunthorpe wrote: > On Tue, Jun 24, 2008 at 04:54:26AM -0700, Hal Rosenstock wrote: > > On Mon, 2008-06-23 at 14:41 -0600, Jason Gunthorpe wrote: > > > Personally, I feel the intention of the spec was to follow the PCI > > > model, it does introduce subsystem ID's for the IO controller > > > stuff, for instance. > > > > Where would the subsystem IDs go ? > > This is the stuff in the IOControllerProfile (pg 1013) OK but that's in an optional agent; wouldn't it need to be in something required to be useful for this purpose ? -- Hal From jgunthorpe at obsidianresearch.com Tue Jun 24 10:53:41 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 24 Jun 2008 11:53:41 -0600 Subject: [ofa-general] vendor_id in struct ib_device_attr In-Reply-To: <1214329863.13056.301.camel@hrosenstock-ws.xsigo.com> References: <20080623194421.GI29198@obsidianresearch.com> <20080623200213.GJ29198@obsidianresearch.com> <20080623204103.GL29198@obsidianresearch.com> <1214308466.13056.263.camel@hrosenstock-ws.xsigo.com> <20080624174542.GO29198@obsidianresearch.com> <1214329863.13056.301.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624175341.GQ29198@obsidianresearch.com> On Tue, Jun 24, 2008 at 10:51:03AM -0700, Hal Rosenstock wrote: > On Tue, 2008-06-24 at 11:45 -0600, Jason Gunthorpe wrote: > > On Tue, Jun 24, 2008 at 04:54:26AM -0700, Hal Rosenstock wrote: > > > On Mon, 2008-06-23 at 14:41 -0600, Jason Gunthorpe wrote: > > > > Personally, I feel the intention of the spec was to follow the PCI > > > > model, it does introduce subsystem ID's for the IO controller > > > > stuff, for instance. > > > > > > Where would the subsystem IDs go ? > > > > This is the stuff in the IOControllerProfile (pg 1013) > > OK but that's in an optional agent; wouldn't it need to be in something > required to be useful for this purpose ? Oh, sorry, I don't think the subsystem ID should be something we worry about, I was just pointing out the original authors seemed to hold a view like the PCI SIG and had the vendor/subsystem split, at least for some parts of the spec. Jason From hrosenstock at xsigo.com Tue Jun 24 11:12:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 11:12:38 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes Message-ID: <1214331158.13056.306.camel@hrosenstock-ws.xsigo.com> opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index 9c80c03..0376686 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -217,8 +217,8 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, * p_subn * [in] Pointer to the OSM Subnet object. * -* port_guid -* [in] The port guid +* port_guid +* [in] The port guid * * RETURN VALUE * None. @@ -262,8 +262,8 @@ osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, * p_subn * [in] Pointer to the OSM Subnet object. * -* port_guid -* [in] The port guid +* port_guid +* [in] The port guid * * RETURN VALUE * None. From arlin.r.davis at intel.com Tue Jun 24 11:19:32 2008 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Tue, 24 Jun 2008 11:19:32 -0700 Subject: [ofa-general] [ANNOUNCE] compat-dapl-1.2.8 and dapl-2.0.10 release - major scalability improvements with new socket cm provider Message-ID: New release for dapl 1.2 and 2.0 is available on the OFA download page and in my git tree. Latest package, using socket cm provider, was successfully tested on the TATA CRL cluster (#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on 1798 nodes, 14384 cores at ~76.9% of peak. md5sum: 39a825669913ff4622ee2b120a9a7409 compat-dapl-1.2.8.tar.gz md5sum: b18652954f02023d5b1172bbd36bfc1e dapl-2.0.10.tar.gz For 1.2 and 2.0 support on same system, including development, install RPM packages as follow: compat-dapl-1.2.8-1 compat-dapl-devel-1.2.8-1 dapl-2.0.10-1 dapl-utils-2.0.10-1 dapl-devel-2.0.10-1 dapl-debuginfo-2.0.10-1 Summary of changes since last release: v1,v2 - socket cm provider added for better scalability v1 - package v1.2 as compat-dapl See http://www.openfabrics.org/downloads/dapl/ more details. -arlin From sashak at voltaire.com Tue Jun 24 13:43:40 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:43:40 +0300 Subject: [ofa-general] [PATCH] opensm/complib: cl_item_obj macro In-Reply-To: <20080624130950.GL7341@sashak.voltaire.com> References: <1214252698.5369.537.camel@cardanus.llnl.gov> <20080624130950.GL7341@sashak.voltaire.com> Message-ID: <20080624204340.GR7341@sashak.voltaire.com> Add cl_item_obj() macro which resolves the object reference by its item field. This can be useful with lists and maps when list/map item is not a first member of the object structure. Signed-off-by: Sasha Khapyorsky --- opensm/include/complib/cl_qlist.h | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/opensm/include/complib/cl_qlist.h b/opensm/include/complib/cl_qlist.h index a02da99..c515f7e 100644 --- a/opensm/include/complib/cl_qlist.h +++ b/opensm/include/complib/cl_qlist.h @@ -132,6 +132,10 @@ typedef struct _cl_list_item { * Quick List *********/ +#define cl_item_obj(item_ptr, obj_ptr, item_field) (typeof(obj_ptr)) \ + ((void *)item_ptr - (unsigned long)&((typeof(obj_ptr))0)->item_field) + + /****s* Component Library: Quick List/cl_list_obj_t * NAME * cl_list_obj_t -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Tue Jun 24 13:45:09 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:45:09 +0300 Subject: [ofa-general] [PATCH] opensm/osm_ucast_mgr: code consolidation and cleanup In-Reply-To: <20080624204340.GR7341@sashak.voltaire.com> References: <1214252698.5369.537.camel@cardanus.llnl.gov> <20080624130950.GL7341@sashak.voltaire.com> <20080624204340.GR7341@sashak.voltaire.com> Message-ID: <20080624204509.GS7341@sashak.voltaire.com> There are some code consolidation, functionality merging and removing intermediate lists. Also osm_port_t structure has list_item field now which is used to maintain port order list for routing. This is a preparation for routing port ordering based on leaf switches reverse sorted by number of active links. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_port.h | 4 + opensm/include/opensm/osm_ucast_mgr.h | 5 + opensm/opensm/osm_ucast_mgr.c | 254 ++++++--------------------------- 3 files changed, 55 insertions(+), 208 deletions(-) diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index ed23554..a07c38e 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -1144,6 +1144,7 @@ static inline osm_dr_path_t *osm_physp_get_dr_path_ptr(IN const osm_physp_t * */ typedef struct osm_port { cl_map_item_t map_item; + cl_list_item_t list_item; struct osm_node *p_node; ib_net64_t guid; uint32_t discovery_count; @@ -1158,6 +1159,9 @@ typedef struct osm_port { * map_item * Linkage structure for cl_qmap. MUST BE FIRST MEMBER! * +* list_item +* Linkage structure for cl_qlist. Used by ucast mgr during LFT calculation. +* * p_node * Points to the Node object that owns this port. * diff --git a/opensm/include/opensm/osm_ucast_mgr.h b/opensm/include/opensm/osm_ucast_mgr.h index 9be059c..1dc9a37 100644 --- a/opensm/include/opensm/osm_ucast_mgr.h +++ b/opensm/include/opensm/osm_ucast_mgr.h @@ -44,6 +44,7 @@ #define _OSM_UCAST_MGR_H_ #include +#include #include #include #include @@ -95,6 +96,7 @@ typedef struct osm_ucast_mgr { osm_subn_t *p_subn; osm_log_t *p_log; cl_plock_t *p_lock; + cl_qlist_t port_order_list; boolean_t is_dor; boolean_t any_change; boolean_t some_hop_count_set; @@ -117,6 +119,9 @@ typedef struct osm_ucast_mgr { * is_dor * Dimension Order Routing (DOR) will be done * +* port_order_list +* List of ports ordered for routing. +* * any_change * Initialized to FALSE at the beginning of the algorithm, * set to TRUE by osm_ucast_mgr_set_fwd_table() if any mad diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index 19df8de..b9e484e 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -59,28 +59,6 @@ #include #include -struct osm_ucast_port_context { - osm_ucast_mgr_t *p_mgr; - cl_qmap_t *p_port_tbl; - cl_qlist_t *p_port_list; -}; - -struct guid_routing_order_list_ctx { - osm_ucast_mgr_t *p_mgr; - cl_qlist_t *p_list; -}; - -struct guid_routing_order_guid_list_item { - cl_list_item_t list; - uint64_t ho_guid; - uint64_t no_guid; -}; - -struct guid_routing_order_port_list_item { - cl_list_item_t list; - osm_port_t *p_port; -}; - /********************************************************************** **********************************************************************/ void osm_ucast_mgr_construct(IN osm_ucast_mgr_t * const p_mgr) @@ -515,33 +493,18 @@ static void __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, IN void *context) { - struct osm_ucast_port_context *p_ctx; + osm_ucast_mgr_t *p_mgr = context; osm_switch_t *const p_sw = (osm_switch_t *) p_map_item; - osm_ucast_mgr_t *p_mgr; - osm_node_t *p_node; - osm_port_t *p_port; - struct guid_routing_order_port_list_item *p_item; - const cl_qmap_t *p_port_tbl = NULL; - const cl_qlist_t *p_port_list = NULL; unsigned i, lids_per_port; - p_ctx = (struct osm_ucast_port_context *)context; - p_mgr = p_ctx->p_mgr; - OSM_LOG_ENTER(p_mgr->p_log); - p_port_tbl = p_ctx->p_port_tbl; - p_port_list = p_ctx->p_port_list; - p_node = p_sw->p_node; - - CL_ASSERT(p_node); - CL_ASSERT(osm_node_get_type(p_node) == IB_NODE_TYPE_SWITCH); - CL_ASSERT(p_port_tbl || p_port_list); + CL_ASSERT(p_sw && p_sw->p_node); if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Processing switch 0x%" PRIx64 "\n", - cl_ntoh64(osm_node_get_node_guid(p_node))); + cl_ntoh64(osm_node_get_node_guid(p_sw->p_node))); } /* Initialize LIDs in buffer to invalid port number. */ @@ -556,17 +519,12 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, */ lids_per_port = 1 << p_mgr->p_subn->opt.lmc; for (i = 0; i < lids_per_port; i++) { - if (p_port_tbl) { - for (p_port = (osm_port_t *) cl_qmap_head(p_port_tbl); - p_port != (osm_port_t *) cl_qmap_end(p_port_tbl); - p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_port, i); - } - else { - for (p_item = (void *)cl_qlist_head(p_port_list); - p_item != (void *)cl_qlist_end(p_port_list); - p_item = (void *)cl_qlist_next(&p_item->list)) - __osm_ucast_mgr_process_port(p_mgr, p_sw, p_item->p_port, i); + cl_qlist_t *list = &p_mgr->port_order_list; + cl_list_item_t *item; + for (item = cl_qlist_head(list); item != cl_qlist_end(list); + item = cl_qlist_next(item)) { + osm_port_t *port = cl_item_obj(item, port, list_item); + __osm_ucast_mgr_process_port(p_mgr, p_sw, port, i); } } @@ -729,182 +687,62 @@ static int ucast_mgr_setup_all_switches(osm_subn_t * p_subn) /********************************************************************** **********************************************************************/ -static int -__guid_qlist_insert(void *ctx, uint64_t guid, char *p) -{ - struct guid_routing_order_list_ctx *list_ctx = ctx; - struct guid_routing_order_guid_list_item *item = NULL; - - item = malloc(sizeof(struct guid_routing_order_guid_list_item)); - if (!item) { - OSM_LOG(list_ctx->p_mgr->p_log, OSM_LOG_DEBUG, - "__guid_qlist_insert: insufficient memory\n"); - return -1; - } - - item->ho_guid = guid; - item->no_guid = cl_ntoh64(guid); - cl_qlist_insert_tail(list_ctx->p_list, &item->list); - - return 0; -} - -static void -__clear_port_flag(IN cl_map_item_t * const p_map_item, void *context) -{ - osm_port_t *p_port = (osm_port_t *)p_map_item; - - p_port->flag = 0; -} - -static void -__find_non_specified_ports(IN cl_map_item_t * const p_map_item, void *context) +static int add_guid_to_order_list(void *ctx, uint64_t guid, char *p) { - osm_port_t *p_port = (osm_port_t *)p_map_item; - struct guid_routing_order_list_ctx *list_ctx = context; - osm_ucast_mgr_t *p_mgr = list_ctx->p_mgr; - cl_qlist_t *port_routing_order_list = list_ctx->p_list; - - if (!p_port->flag) { - struct guid_routing_order_port_list_item *p_item = NULL; + osm_ucast_mgr_t *m = ctx; + osm_port_t *port = osm_get_port_by_guid(m->p_subn, cl_hton64(guid)); - p_item = malloc(sizeof(struct guid_routing_order_port_list_item)); - if (!p_item) { - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "__find_non_specified_ports: insufficient memory\n"); - return; - } - - p_item->p_port = p_port; - cl_qlist_insert_tail(port_routing_order_list, &p_item->list); + if (!port) { + OSM_LOG(m->p_log, OSM_LOG_DEBUG, + "port guid not found: 0x%016" PRIx64 "\n", guid); + return 0; } -} -static int -__port_qlist_create(osm_ucast_mgr_t *p_mgr, - cl_qlist_t *guid_routing_order_list, - cl_qlist_t *port_routing_order_list) -{ - cl_qmap_t *p_port_tbl = &p_mgr->p_subn->port_guid_tbl; - struct guid_routing_order_guid_list_item *g_item; - struct guid_routing_order_list_ctx list_ctx; - osm_port_t *p_port; - - /* Set each osm_port_t flag to 0 */ - cl_qmap_apply_func(p_port_tbl, __clear_port_flag, NULL); - - /* Find the port of the guid input */ - for (g_item = (void *)cl_qlist_head(guid_routing_order_list); - g_item != (void *)cl_qlist_end(guid_routing_order_list); - g_item = (void *)cl_qlist_next(&g_item->list)) { - struct guid_routing_order_port_list_item *p_item = NULL; - - p_port = (void *)cl_qmap_get(p_port_tbl, g_item->no_guid); - if (p_port == (void *)cl_qmap_end(p_port_tbl)) { - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "port guid not found: 0x%016" PRIx64 "\n", - g_item->ho_guid); - continue; - } - - if (p_port->flag) { - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "port guid specified multiple times" - "0x%016" PRIx64 "\n", - g_item->ho_guid); - continue; - } - - p_item = malloc(sizeof(struct guid_routing_order_port_list_item)); - if (!p_item) { - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "__port_qlist_create: insufficient memory\n"); - return -1; - } - - p_item->p_port = p_port; - cl_qlist_insert_tail(port_routing_order_list, &p_item->list); - p_port->flag++; + if (port->flag) { + OSM_LOG(m->p_log, OSM_LOG_DEBUG, + "port guid specified multiple times 0x%016" PRIx64 "\n", + guid); + return 0; } - /* Any ports not specified, add them to the end of the list */ - list_ctx.p_mgr = p_mgr; - list_ctx.p_list = port_routing_order_list; - - cl_qmap_apply_func(p_port_tbl, __find_non_specified_ports, - &list_ctx); + cl_qlist_insert_tail(&m->port_order_list, &port->list_item); + port->flag = 1; return 0; } -static void -__osm_ucast_mgr_build_fwd_tables(IN osm_ucast_mgr_t * const p_mgr) +static void add_port_to_order_list(cl_map_item_t * const p_map_item, void *ctx) { - struct osm_ucast_port_context ctx; - osm_opensm_t *p_osm; - cl_qmap_t *p_sw_guid_tbl; - cl_qlist_t guid_routing_order_list; - cl_qlist_t port_routing_order_list; - unsigned int lists_created = 0; - unsigned int use_routing_order = 0; - int ret = 0; + osm_port_t *port = (osm_port_t *)p_map_item; + osm_ucast_mgr_t *m = ctx; - p_osm = p_mgr->p_subn->p_osm; - p_sw_guid_tbl = &p_mgr->p_subn->sw_guid_tbl; + if (!port->flag) + cl_qlist_insert_tail(&m->port_order_list, &port->list_item); + else + port->flag = 0; +} - if (p_osm->subn.opt.guid_routing_order_file) { - struct guid_routing_order_list_ctx list_ctx; +static void ucast_mgr_build_lfts(osm_ucast_mgr_t *p_mgr) +{ + cl_qlist_init(&p_mgr->port_order_list); - OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, + if (p_mgr->p_subn->opt.guid_routing_order_file) { + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Fetching guid routing order file %s\n", - p_osm->subn.opt.guid_routing_order_file); - - cl_qlist_init(&guid_routing_order_list); - cl_qlist_init(&port_routing_order_list); - - lists_created++; + p_mgr->p_subn->opt.guid_routing_order_file); - list_ctx.p_mgr = p_mgr; - list_ctx.p_list = &guid_routing_order_list; - - ret = parse_node_map(p_osm->subn.opt.guid_routing_order_file, - __guid_qlist_insert, - &list_ctx); - if (!ret && cl_qlist_count(&guid_routing_order_list)) { - if (__port_qlist_create(p_mgr, - &guid_routing_order_list, - &port_routing_order_list) < 0) - goto cleanup; - use_routing_order++; - } - else - OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR : " + if (parse_node_map(p_mgr->p_subn->opt.guid_routing_order_file, + add_guid_to_order_list, p_mgr)) + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR : " "cannot parse guid routing order file %s\n", - p_osm->subn.opt.guid_routing_order_file); + p_mgr->p_subn->opt.guid_routing_order_file); } - if (use_routing_order) { - ctx.p_mgr = p_mgr; - ctx.p_port_tbl = NULL; - ctx.p_port_list = &port_routing_order_list; - } - else { - ctx.p_mgr = p_mgr; - ctx.p_port_tbl = &p_mgr->p_subn->port_guid_tbl; - ctx.p_port_list = NULL; - } + cl_qmap_apply_func(&p_mgr->p_subn->port_guid_tbl, + add_port_to_order_list, p_mgr); - cl_qmap_apply_func(p_sw_guid_tbl, __osm_ucast_mgr_process_tbl, - &ctx); - -cleanup: - if (lists_created) { - while (!cl_is_qlist_empty(&guid_routing_order_list)) - free(cl_qlist_remove_head(&guid_routing_order_list)); - - while (!cl_is_qlist_empty(&port_routing_order_list)) - free(cl_qlist_remove_head(&port_routing_order_list)); - } + cl_qmap_apply_func(&p_mgr->p_subn->sw_guid_tbl, + __osm_ucast_mgr_process_tbl, p_mgr); } /********************************************************************** @@ -949,7 +787,7 @@ osm_signal_t osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) if (!p_routing_eng->ucast_build_fwd_tables || (ubft = p_routing_eng->ucast_build_fwd_tables(p_routing_eng->context))) - __osm_ucast_mgr_build_fwd_tables(p_mgr); + ucast_mgr_build_lfts(p_mgr); /* 'file' routing engine has one unique logic corner case */ if (p_routing_eng->name && (strcmp(p_routing_eng->name, "file") == 0) -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Tue Jun 24 13:46:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:46:39 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/include/complib/cl_fleximap.h: Improve cl_pfn_fmap_cmp_t PARAMETERS comments In-Reply-To: <1214317633.13056.274.camel@hrosenstock-ws.xsigo.com> References: <1214317633.13056.274.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624204639.GT7341@sashak.voltaire.com> On 07:27 Tue 24 Jun , Hal Rosenstock wrote: > opensm/include/complib/cl_fleximap.h: Improve cl_pfn_fmap_cmp_t > PARAMETERS comments > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Jun 24 13:47:20 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:47:20 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes In-Reply-To: <1214331158.13056.306.camel@hrosenstock-ws.xsigo.com> References: <1214331158.13056.306.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624204720.GU7341@sashak.voltaire.com> On 11:12 Tue 24 Jun , Hal Rosenstock wrote: > opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From rdreier at cisco.com Tue Jun 24 13:48:56 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 13:48:56 -0700 Subject: [ofa-general] [PATCH] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: <4860D860.1080805@Voltaire.COM> (Moni Shoua's message of "Tue, 24 Jun 2008 14:20:00 +0300") References: <484F8A20.9070101@Voltaire.COM> <4860D860.1080805@Voltaire.COM> Message-ID: > I did ran the script on the patch and besides one WARNING for > a line that exceeds 80 columns which I believe is allowed sometimes. Too-long lines are OK with me, but I get the following with the patch I was replying to: WARNING: line over 80 characters #181: FILE: drivers/infiniband/ulp/ipoib/ipoib_main.c:266: + be16_to_cpu(path->pathrec.dlid), IPOIB_GID_ARG(path->pathrec.dgid)); ERROR: space required after that ',' (ctx:VxV) #275: FILE: drivers/infiniband/ulp/ipoib/ipoib_main.c:665: + ipoib_dbg(priv,"path does not exist\n"); ^ ERROR: space required after that ',' (ctx:VxV) #278: FILE: drivers/infiniband/ulp/ipoib/ipoib_main.c:668: + ipoib_dbg(priv,"path exists but invalid\n"); ^ ERROR: space required after that ',' (ctx:VxV) #310: FILE: drivers/infiniband/ulp/ipoib/ipoib_verbs.c:292: + ipoib_dbg(priv, "Event %d on device %s port %d\n",record->event, ^ ERROR: space prohibited after that open parenthesis '(' #312: FILE: drivers/infiniband/ulp/ipoib/ipoib_verbs.c:294: + if ( record->event == IB_EVENT_SM_CHANGE || total: 4 errors, 1 warnings, 258 lines checked From rdreier at cisco.com Tue Jun 24 13:49:47 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 13:49:47 -0700 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860E37C.20502@dev.mellanox.co.il> (Vladimir Sokolovsky's message of "Tue, 24 Jun 2008 15:07:24 +0300") References: <4860A5BE.7090608@dev.mellanox.co.il> <4860C150.8070006@voltaire.com> <4860E37C.20502@dev.mellanox.co.il> Message-ID: > Yes, we thought about this and I am planning to add this granularity > later, in a different patch. If we're going to change the interface let's do it now so we don't break setups later. - R. From sashak at voltaire.com Tue Jun 24 13:49:23 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:49:23 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_sw_info_rcv.c: Cosmetic formatting change In-Reply-To: <1214135406.13056.219.camel@hrosenstock-ws.xsigo.com> References: <1214135406.13056.219.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624204923.GV7341@sashak.voltaire.com> On 04:50 Sun 22 Jun , Hal Rosenstock wrote: > opensm/osm_sw_info_rcv.c: Cosmetic formatting change > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Jun 24 13:52:59 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:52:59 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_multicast.h: Remove no longer needed element in osm_mgrp structure In-Reply-To: <1214312066.13056.268.camel@hrosenstock-ws.xsigo.com> References: <1214312066.13056.268.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624205259.GW7341@sashak.voltaire.com> Hi Hal, On 05:54 Tue 24 Jun , Hal Rosenstock wrote: > opensm/osm_multicast.h: Remove no longer needed element in osm_mgrp > structure But AFAIR you wanted to map mgrp by mgid? No? Sasha From rdreier at cisco.com Tue Jun 24 13:56:10 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 13:56:10 -0700 Subject: [ofa-general] Re: [PATCH v2 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <48611FEB.1030709@dev.mellanox.co.il> (Vladimir Sokolovsky's message of "Tue, 24 Jun 2008 19:25:15 +0300") References: <48611FEB.1030709@dev.mellanox.co.il> Message-ID: To be honest this looks like a step backwards... > +static ssize_t show_use_lro(struct device *d, struct device_attribute *attr, > + char *buf) > +{ > + struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(d)); > + > + if (priv->lro.use_lro) > + return sprintf(buf, "yes\n"); > + else > + return sprintf(buf, "no\n"); > +} This looks like a crazy interface ("yes"/"no" vs 1/0??) and in any case I think using the existing ethtool support to control LRO would be much simpler and better for users. - R. From sashak at voltaire.com Tue Jun 24 13:56:04 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Jun 2008 23:56:04 +0300 Subject: [ofa-general] Re: [PATCHv2] ibsim: Support short RMPP packets In-Reply-To: <1214229226.13056.230.camel@hrosenstock-ws.xsigo.com> References: <1213886129.13056.168.camel@hrosenstock-ws.xsigo.com> <20080620175833.GK3158@sashak.voltaire.com> <1214229226.13056.230.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624205604.GX7341@sashak.voltaire.com> On 06:53 Mon 23 Jun , Hal Rosenstock wrote: > > An additional change is needed to umad2sim/umad2sim.c: > > @@ -425,7 +425,7 @@ static ssize_t umad2sim_read(struct umad2sim_dev *dev, void > cnt = count - umad_size(); > memcpy(umad_get_mad(umad), req.mad, cnt); > > - return cnt + umad_size(); > + return umad->length; > } Yes, correct. I will apply the final patch. Sasha From hrosenstock at xsigo.com Tue Jun 24 14:18:56 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 14:18:56 -0700 Subject: [ofa-general] Re: [PATCH] opensm/osm_multicast.h: Remove no longer needed element in osm_mgrp structure In-Reply-To: <20080624205259.GW7341@sashak.voltaire.com> References: <1214312066.13056.268.camel@hrosenstock-ws.xsigo.com> <20080624205259.GW7341@sashak.voltaire.com> Message-ID: <1214342336.13056.324.camel@hrosenstock-ws.xsigo.com> Hi Sasha, On Tue, 2008-06-24 at 23:52 +0300, Sasha Khapyorsky wrote: > Hi Hal, > > On 05:54 Tue 24 Jun , Hal Rosenstock wrote: > > opensm/osm_multicast.h: Remove no longer needed element in osm_mgrp > > structure > > But AFAIR you wanted to map mgrp by mgid? No? It's not needed if the MLID array patch is accepted. I'm not sure it's needed once multiple MGIDs can be supported on the same MLID so maybe it'll come back but that remains to be seen AFAIK. -- Hal > Sasha From rdreier at cisco.com Tue Jun 24 14:47:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 14:47:21 -0700 Subject: [ofa-general] Re: [PATCH 2/3 v2] mlx4: multicast cast loopback block In-Reply-To: <4860C216.3010400@voltaire.com> (Or Gerlitz's message of "Tue, 24 Jun 2008 12:44:54 +0300") References: <4860C216.3010400@voltaire.com> Message-ID: > Ron will be here later this week and then he can make the check and resend. sigh... I can fix it up this time but it's such a pointless waste to send patches with problems when a trivial, easy-to-run script can tell you how to fix them. > personally, I find !! code somehow hard to read, but if this is the > common practice in such cases, we can fix that... not a big deal either way. I prefer my way because if I have if (something) func(lots, of, parameters, all, the, same, except, 1); else func(lots, of, parameters, all, the, same, except, 0); then it's not always so clear what the difference between the two function calls really is without reading carefully. By putting !!something into the call: func(lots, of, parameters, all, the, same, except, !!something); it becomes obvious how the value of something is used. - R. From weiny2 at llnl.gov Tue Jun 24 14:48:36 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 24 Jun 2008 14:48:36 -0700 Subject: [ofa-general] [PATCH 0/2] print ibwarn messages when _do_madrpc fails Message-ID: <20080624144836.79014087.weiny2@llnl.gov> These 2 patches print a warning message when _do_madrpc fails. The message includes the destination port. Ira From weiny2 at llnl.gov Tue Jun 24 14:48:38 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 24 Jun 2008 14:48:38 -0700 Subject: [ofa-general] [PATCH 1/2] libibmad: add a drpath2str function for easier printing of DR Paths. Message-ID: <20080624144838.55ec7e3d.weiny2@llnl.gov> >From e5d47f962f7ce9b71f12d31306ae1e0400e46836 Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Tue, 24 Jun 2008 14:42:25 -0700 Subject: [PATCH] libibmad: add a drpath2str function for easier printing of DR Paths. Signed-off-by: Ira K. Weiny --- libibmad/include/infiniband/mad.h | 1 + libibmad/src/portid.c | 12 ++++++++++++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index 4f19a31..ae3c644 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -622,6 +622,7 @@ enum { char * portid2str(ib_portid_t *portid); int portid2portnum(ib_portid_t *portid); int str2drpath(ib_dr_path_t *path, char *routepath, int drslid, int drdlid); +char * drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size); static inline int ib_portid_set(ib_portid_t *portid, int lid, int qp, int qkey) diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c index 056b03d..7279e14 100644 --- a/libibmad/src/portid.c +++ b/libibmad/src/portid.c @@ -114,3 +114,15 @@ str2drpath(ib_dr_path_t *path, char *routepath, int drslid, int drdlid) return path->cnt; } + +char * +drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size) +{ + int i = 0; + int rc = snprintf(dstr, dstr_size, "slid %d; dlid %d; %d", + path->drslid, path->drdlid, path->p[0]); + for (i = 1; i <= path->cnt; i++) { + rc += snprintf(dstr+rc, dstr_size-rc, ",%d", path->p[i]); + } + return (dstr); +} -- 1.5.1 From weiny2 at llnl.gov Tue Jun 24 14:48:39 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 24 Jun 2008 14:48:39 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information Message-ID: <20080624144839.35b6d38f.weiny2@llnl.gov> >From e428df95fb65becd8bff82dc13ae40cfd3563d46 Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Tue, 24 Jun 2008 14:43:24 -0700 Subject: [PATCH] libibmad: print warning when _do_madrpc fails with destination port information Signed-off-by: Ira K. Weiny --- libibmad/src/rpc.c | 21 +++++++++++++++++++-- 1 files changed, 19 insertions(+), 2 deletions(-) diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index cb1940b..7ff0aa5 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include "mad.h" @@ -185,8 +186,16 @@ mad_rpc(void *port_id, ib_rpc_t *rpc, ib_portid_t *dport, void *payload, if ((len = _do_madrpc(p->port_id, sndbuf, rcvbuf, p->class_agents[rpc->mgtclass], - len, rpc->timeout)) < 0) + len, rpc->timeout)) < 0) { + static char gid[256]; + static char drpath[256]; + inet_ntop(AF_INET6, dport->gid, gid, 256); + drpath2str(&(dport->drpath), drpath, 256); + IBWARN("_do_madrpc failed; dport (lid %d; grh \"%s\"; gid %s; drpath %s)", + dport->lid, dport->grh_present ? "yes":"no", + gid, drpath); return 0; + } mad = umad_get_mad(rcvbuf); @@ -223,8 +232,16 @@ mad_rpc_rmpp(void *port_id, ib_rpc_t *rpc, ib_portid_t *dport, if ((len = _do_madrpc(p->port_id, sndbuf, rcvbuf, p->class_agents[rpc->mgtclass], - len, rpc->timeout)) < 0) + len, rpc->timeout)) < 0) { + static char gid[256]; + static char drpath[256]; + inet_ntop(AF_INET6, dport->gid, gid, 256); + drpath2str(&(dport->drpath), drpath, 256); + IBWARN("_do_madrpc failed; dport (lid %d; grh \"%s\"; gid %s; drpath %s)", + dport->lid, dport->grh_present ? "yes":"no", + gid, drpath); return 0; + } mad = umad_get_mad(rcvbuf); -- 1.5.1 From rdreier at cisco.com Tue Jun 24 14:54:14 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 14:54:14 -0700 Subject: [ofa-general] Re: [PATCH v2] Add enum strings and *_str functions for enums In-Reply-To: <20080616102430.505a2000.weiny2@llnl.gov> (Ira Weiny's message of "Mon, 16 Jun 2008 10:24:30 -0700") References: <20080415133548.414aeaea.weiny2@llnl.gov> <200806161705.41183.jackm@dev.mellanox.co.il> <20080616102430.505a2000.weiny2@llnl.gov> Message-ID: How about if we just revert the devinfo.c change that used the new conversion function? I think "down" is a lot nicer in general than "PORT_DOWN" so we might as well let new code use that. But I do agree we don't want to break any apps that are parsing ibv_devinfo output. So the following seems like i twould work? diff --git a/examples/devinfo.c b/examples/devinfo.c index 86ad7da..caa5d5f 100644 --- a/examples/devinfo.c +++ b/examples/devinfo.c @@ -67,6 +67,17 @@ static const char *guid_str(uint64_t node_guid, char *str) return str; } +static const char *port_state_str(enum ibv_port_state pstate) +{ + switch (pstate) { + case IBV_PORT_DOWN: return "PORT_DOWN"; + case IBV_PORT_INIT: return "PORT_INIT"; + case IBV_PORT_ARMED: return "PORT_ARMED"; + case IBV_PORT_ACTIVE: return "PORT_ACTIVE"; + default: return "invalid state"; + } +} + static const char *port_phy_state_str(uint8_t phys_state) { switch (phys_state) { @@ -254,7 +265,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port) } printf("\t\tport:\t%d\n", port); printf("\t\t\tstate:\t\t\t%s (%d)\n", - ibv_port_state_str(port_attr.state), port_attr.state); + port_state_str(port_attr.state), port_attr.state); printf("\t\t\tmax_mtu:\t\t%s (%d)\n", mtu_str(port_attr.max_mtu), port_attr.max_mtu); printf("\t\t\tactive_mtu:\t\t%s (%d)\n", From sashak at voltaire.com Tue Jun 24 14:58:52 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 00:58:52 +0300 Subject: [ofa-general] Re: [PATCH] opensm: Convert mgrp_mlid_tbl into array In-Reply-To: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> References: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624215852.GY7341@sashak.voltaire.com> Hi Hal, On 04:46 Sun 22 Jun , Hal Rosenstock wrote: > opensm: Convert mgrp_mlid_tbl into array Comments are below. > > Signed-off-by: Hal Rosenstock > > diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h > index ef9996c..75be84a 100644 > --- a/opensm/include/opensm/osm_subnet.h > +++ b/opensm/include/opensm/osm_subnet.h > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -446,6 +447,10 @@ typedef struct osm_subn_opt { > * Subnet object > *********/ > > +struct _osm_mgrp_mlid_tbl { > + void *mgroup[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; > +}; > + Why this wrapper struct is needed? > /****s* OpenSM: Subnet/osm_subn_t > * NAME > * osm_subn_t > @@ -467,7 +472,6 @@ typedef struct osm_subn { > cl_qmap_t rtr_guid_tbl; > cl_qlist_t prefix_routes_list; > cl_qmap_t prtn_pkey_tbl; > - cl_qmap_t mgrp_mlid_tbl; > cl_qmap_t sm_guid_tbl; > cl_qlist_t sa_sr_list; > cl_qlist_t sa_infr_list; > @@ -490,6 +494,7 @@ typedef struct osm_subn { > boolean_t first_time_master_sweep; > boolean_t coming_out_of_standby; > unsigned need_update; > + struct _osm_mgrp_mlid_tbl mgrp_mlid_tbl; > } osm_subn_t; > /* > * FIELDS > @@ -513,10 +518,6 @@ typedef struct osm_subn { > * Container of pointers to all Partition objects in the subnet. > * Indexed by P_KEY. > * > -* mgrp_mlid_tbl > -* Container of pointers to all Multicast Group objects in the subnet. > -* Indexed by MLID. > -* > * sm_guid_tbl > * Container of pointers to SM objects representing other SMs > * on the subnet. > @@ -615,6 +616,10 @@ typedef struct osm_subn { > * This flag should be on during first non-master heavy (including > * pre-master discovery stage) > * > +* mgrp_mlid_tbl > +* Array of pointers to all Multicast Group objects in the subnet. > +* Indexed by MLID offset from base MLID. > +* > * SEE ALSO > * Subnet object > *********/ > diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c > index e2f5828..44ee8eb 100644 > --- a/opensm/opensm/osm_drop_mgr.c > +++ b/opensm/opensm/osm_drop_mgr.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -163,6 +164,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > ib_gid_t port_gid; > ib_mad_notice_attr_t notice; > ib_api_status_t status; > + int index; Here and below, 'int i' looks much more friendly as loop index variable. > OSM_LOG_ENTER(sm->p_log); > > @@ -206,14 +208,15 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > > p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); > while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { > - p_mgrp = > - (osm_mgrp_t *) cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, > - p_mcm->mlid); > - if (p_mgrp != > - (osm_mgrp_t *) cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { > - osm_mgrp_remove_port(sm->p_subn, sm->p_log, > - p_mgrp, p_port->guid); > - osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > + for (index = 0; > + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > + p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; > + if (p_mgrp) { > + osm_mgrp_remove_port(sm->p_subn, sm->p_log, > + p_mgrp, p_port->guid); > + osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > + } Here you have p_mcm->mlid which is index of multicast group where this port is member + IB_LID_MCAST_START_HO. So you don't need to loop other all groups in the array. Actually there only one line require changing: p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[p_mcm->mlid - IB_LID_MCAST_START_HO] if (p_mgrp) { osm_mgrp_remove_port(sm->p_subn, sm->p_log, p_mgrp, p_port->guid); osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); } > } > p_mcm = > (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); > diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c > index d7a0f56..15a585e 100644 > --- a/opensm/opensm/osm_mcast_mgr.c > +++ b/opensm/opensm/osm_mcast_mgr.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -1132,6 +1133,8 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, > IN ib_net64_t port_guid) > { > ib_api_status_t status; > + osm_mgrp_t *p_mgroup; > + int index; > > OSM_LOG_ENTER(sm->p_log); > > @@ -1154,8 +1157,15 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, > /* Send a Report to any InformInfo registered for > Trap 67 : MCGroup delete */ > osm_mgrp_send_delete_notice(sm->p_subn, sm->p_log, p_mgrp); > - cl_qmap_remove_item(&sm->p_subn->mgrp_mlid_tbl, > - (cl_map_item_t *) p_mgrp); > + for (index = 0; > + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > + p_mgroup = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; > + if (p_mgroup == p_mgrp) { > + sm->p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; > + break; > + } > + } The same is here. > osm_mgrp_delete(p_mgrp); > } > > @@ -1171,16 +1181,14 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > osm_signal_t signal; > osm_switch_t *p_sw; > cl_qmap_t *p_sw_tbl; > - cl_qmap_t *p_mcast_tbl; > cl_qlist_t *p_list = &sm->mgrp_list; > osm_mgrp_t *p_mgrp; > - osm_mgrp_t *p_next_mgrp; > boolean_t pending_transactions = FALSE; > + int index; > > OSM_LOG_ENTER(sm->p_log); > > p_sw_tbl = &sm->p_subn->sw_guid_tbl; > - p_mcast_tbl = &sm->p_subn->mgrp_mlid_tbl; > /* > While holding the lock, iterate over all the established > multicast groups, servicing each in turn. > @@ -1189,17 +1197,19 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > */ > CL_PLOCK_EXCL_ACQUIRE(sm->p_lock); > > - p_mgrp = (osm_mgrp_t *) cl_qmap_head(p_mcast_tbl); > - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(p_mcast_tbl)) { > + for (index = 0; > + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > /* > We reached here due to some change that caused a heavy sweep > of the subnet. Not due to a specific multicast request. > So the request type is subnet_change and the port guid is 0. > */ > - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > - mcast_mgr_process_mgrp(sm, p_mgrp, > - OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, 0); > - p_mgrp = p_next_mgrp; > + p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; > + if (p_mgrp) > + mcast_mgr_process_mgrp(sm, p_mgrp, > + OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, > + 0); > } > > /* > @@ -1233,13 +1243,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > static > osm_mgrp_t *__get_mgrp_by_mlid(osm_sm_t * sm, IN ib_net16_t const mlid) > { > - cl_map_item_t *map_item; > - > - map_item = cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, mlid); > - if (map_item == cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { > - return NULL; > - } > - return (osm_mgrp_t *) map_item; > + return(sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); > } Same function exists in osm_sa_mcmember_record.c. It is probably good time to consolidate it in something like osm_get_mgrp_by_mlid() (it could be placed in osm_subnet.c where another osm_get_*_by() hosted). > > /********************************************************************** > diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c > index f1d4e54..5a8dd3b 100644 > --- a/opensm/opensm/osm_qos_policy.c > +++ b/opensm/opensm/osm_qos_policy.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2006 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -792,11 +793,8 @@ static void __qos_policy_validate_pkey( > if (!p_prtn->mlid) > return; > > - p_mgrp = (osm_mgrp_t *) cl_qmap_get( > - &p_qos_policy->p_subn->mgrp_mlid_tbl, > - p_prtn->mlid); > - if (p_mgrp == (osm_mgrp_t *) > - cl_qmap_end(&p_qos_policy->p_subn->mgrp_mlid_tbl)) { > + p_mgrp = p_qos_policy->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(p_prtn->mlid) - IB_LID_MCAST_START_HO]; > + if (!p_mgrp) { > OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, > "ERR AC16: MCast group for partition with " > "pkey 0x%04X not found\n", > diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c > index adf4931..4695bc0 100644 > --- a/opensm/opensm/osm_sa.c > +++ b/opensm/opensm/osm_sa.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -558,12 +559,11 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) > p_mcm_port->scope_state, p_mcm_port->proxy_join); > } > > -static void sa_dump_one_mgrp(cl_map_item_t * p_map_item, void *cxt) > +static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) > { > struct opensm_dump_context dump_context; > osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; > FILE *file = ((struct opensm_dump_context *)cxt)->file; > - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; > > fprintf(file, "MC Group 0x%04x %s:" > " mgid=0x%016" PRIx64 ":0x%016" PRIx64 > @@ -700,13 +700,20 @@ static void sa_dump_one_service(cl_list_item_t * p_list_item, void *cxt) > static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) > { > struct opensm_dump_context dump_context; > + osm_mgrp_t *p_mgrp; > + int index; > > dump_context.p_osm = p_osm; > dump_context.file = file; > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump multicast:\n"); > cl_plock_acquire(&p_osm->lock); > - cl_qmap_apply_func(&p_osm->subn.mgrp_mlid_tbl, > - sa_dump_one_mgrp, &dump_context); > + for (index = 0; > + index <= p_osm->subn.max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > + p_mgrp = p_osm->subn.mgrp_mlid_tbl.mgroup[index]; > + if (p_mgrp) > + sa_dump_one_mgrp(p_mgrp, &dump_context); > + } > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform:\n"); > cl_qlist_apply_func(&p_osm->subn.sa_infr_list, > sa_dump_one_inform, &dump_context); > @@ -729,14 +736,12 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, > unsigned well_known) > { > ib_net64_t comp_mask; > - cl_map_item_t *p_next; > osm_mgrp_t *p_mgrp; > > cl_plock_excl_acquire(&p_osm->lock); > > - if ((p_next = cl_qmap_get(&p_osm->subn.mgrp_mlid_tbl, mlid)) != > - cl_qmap_end(&p_osm->subn.mgrp_mlid_tbl)) { > - p_mgrp = (osm_mgrp_t *) p_next; > + p_mgrp = p_osm->subn.mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > + if (p_mgrp) { > if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, > sizeof(ib_gid_t))) { > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > index 46c87c7..7049fc0 100644 > --- a/opensm/opensm/osm_sa_mcmember_record.c > +++ b/opensm/opensm/osm_sa_mcmember_record.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -96,13 +97,7 @@ typedef struct osm_sa_mcmr_search_ctxt { > static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, > IN ib_net16_t const mlid) > { > - cl_map_item_t *map_item; > - > - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); > - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { > - return NULL; > - } > - return (osm_mgrp_t *) map_item; > + return(sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); > } > > /********************************************************************* > @@ -154,16 +149,20 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO > && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho > - && cl_qmap_get(&p_subn->mgrp_mlid_tbl, > - requested_mlid) == > - cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > + && !p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { > mlid = cl_ntoh16(requested_mlid); > goto Exit; > } > > /* If MCGroups table is empty, first return the min mlid */ > - p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > + max_num_mlids = sa->p_subn->max_multicast_lid_ho - > + IB_LID_MCAST_START_HO + 1; > + for (idx = 0; idx < max_num_mlids; idx++) { > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > + if (p_mgrp) > + break; > + } > + if (!p_mgrp) { > mlid = IB_LID_MCAST_START_HO; > OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, > "No multicast groups found using minimal mlid:0x%04X\n", > @@ -171,9 +170,6 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > goto Exit; > } > > - max_num_mlids = > - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; > - > /* track all used mlids in the array (by mlid index) */ > used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); > if (!used_mlids_array) > @@ -181,9 +177,10 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); > > /* scan all available multicast groups in the DB and fill in the table */ > - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > + for (idx = 0; idx < max_num_mlids; idx++) { > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > /* ignore mgrps marked for deletion */ > - if (p_mgrp->to_be_deleted == FALSE) { > + if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 > " : " "0x%016" PRIx64 "\n", > @@ -202,11 +199,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > cl_ntoh16(p_mgrp->mlid), > max_num_mlids + IB_LID_MCAST_START_HO); > } else { > - used_mlids_array[cl_ntoh16(p_mgrp->mlid) - > - IB_LID_MCAST_START_HO] = 1; > + used_mlids_array[idx] = 1; > } > } > - p_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > } > > /* Find "mlid holes" in the mgrp table */ There were some dirty tricks when qmap was used and finally they did an additional mlid array (that with malloc()) for free mlid searching. But now you already have such array, what is the reason to copy this and to run additional loop? > @@ -247,8 +242,7 @@ __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) > not a well known group */ > if (p_mgrp && cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && > p_mgrp->well_known == FALSE) { > - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, > - (cl_map_item_t *) p_mgrp); > + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; > osm_mgrp_delete(p_mgrp); > } > } > @@ -1033,21 +1027,17 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > /* since we might have an old group by that mlid > one whose deletion was delayed for an idle time > we need to deallocate it first */ > - p_prev_mgrp = > - (osm_mgrp_t *) cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); > - if (p_prev_mgrp != > - (osm_mgrp_t *) cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { > + p_prev_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > + if (p_prev_mgrp) { > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > "Found previous group for mlid:0x%04x - " > "Destroying it first\n", > cl_ntoh16(mlid)); > - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, > - (cl_map_item_t *) p_prev_mgrp); > + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; > osm_mgrp_delete(p_prev_mgrp); > } > > - cl_qmap_insert(&sa->p_subn->mgrp_mlid_tbl, > - mlid, &(*pp_mgrp)->map_item); > + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; > > /* Send a Report to any InformInfo registered for > Trap 66: MCGroup create */ > @@ -1069,9 +1059,8 @@ typedef struct osm_sa_pr_mcmr_search_ctxt { > /********************************************************************** > *********************************************************************/ > static void > -__search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > +__search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) > { > - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; > osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = > (osm_sa_pr_mcmr_search_ctxt_t *) context; > const ib_gid_t *p_recvd_mgid; > @@ -1135,23 +1124,30 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > **********************************************************************/ > ib_api_status_t > osm_get_mgrp_by_mgid(IN osm_sa_t *sa, > - IN ib_gid_t *p_mgid, > - OUT osm_mgrp_t **pp_mgrp) > + IN ib_gid_t *p_mgid, > + OUT osm_mgrp_t **pp_mgrp) > { > osm_sa_pr_mcmr_search_ctxt_t mcmr_search_context; > + osm_mgrp_t *p_mgrp; > + int index; > > mcmr_search_context.p_mgid = p_mgid; > mcmr_search_context.sa = sa; > mcmr_search_context.p_mgrp = NULL; > > - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, > - __search_mgrp_by_mgid, &mcmr_search_context); > - > - if (mcmr_search_context.p_mgrp == NULL) > - return IB_NOT_FOUND; > - > - *pp_mgrp = mcmr_search_context.p_mgrp; > - return IB_SUCCESS; > + for (index = 0; > + index <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[index]; > + if (p_mgrp) { > + __search_mgrp_by_mgid(p_mgrp, &mcmr_search_context); > + if (mcmr_search_context.p_mgrp) { > + *pp_mgrp = mcmr_search_context.p_mgrp; > + return IB_SUCCESS; > + } > + } > + } > + return IB_NOT_FOUND; > } > > /********************************************************************** > @@ -1619,10 +1615,9 @@ Exit: > Match the given mgrp to the requested mcmr > **********************************************************************/ > static void > -__osm_sa_mcm_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, > +__osm_sa_mcm_by_comp_mask_cb(IN osm_mgrp_t * const p_mgrp, > IN void *context) > { > - const osm_mgrp_t *const p_mgrp = (osm_mgrp_t *) p_map_item; > osm_sa_mcmr_search_ctxt_t *const p_ctxt = > (osm_sa_mcmr_search_ctxt_t *) context; > osm_sa_t *sa = p_ctxt->sa; > @@ -1811,6 +1806,8 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, > ib_net64_t comp_mask; > osm_physp_t *p_req_physp; > boolean_t trusted_req; > + osm_mgrp_t *p_mgrp; > + int index; > > OSM_LOG_ENTER(sa->p_log); > > @@ -1846,8 +1843,14 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, > CL_PLOCK_ACQUIRE(sa->p_lock); > > /* simply go over all MCGs and match */ > - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, > - __osm_sa_mcm_by_comp_mask_cb, &context); > + for (index = 0; > + index <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[index]; > + if (p_mgrp) { > + __osm_sa_mcm_by_comp_mask_cb(p_mgrp, &context); > + } > + } > > CL_PLOCK_RELEASE(sa->p_lock); > > diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c > index 2a9155c..fb42afb 100644 > --- a/opensm/opensm/osm_sa_path_record.c > +++ b/opensm/opensm/osm_sa_path_record.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -1472,14 +1473,7 @@ __osm_pr_rcv_process_pair(IN osm_sa_t * sa, > static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, > IN ib_net16_t const mlid) > { > - cl_map_item_t *map_item; > - > - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); > - > - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) > - return NULL; > - > - return (osm_mgrp_t *) map_item; > + return(sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); > } Hmm, I thought there are two instances of __get_mgrp_by_mlid(), this one os third. > > /********************************************************************** > diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c > index 7393ead..c6526e7 100644 > --- a/opensm/opensm/osm_sm.c > +++ b/opensm/opensm/osm_sm.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -498,7 +499,6 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > { > osm_mgrp_t *p_mgrp; > osm_port_t *p_port; > - cl_qmap_t *p_tbl; > ib_api_status_t status = IB_SUCCESS; > osm_mcm_info_t *p_mcm; > > @@ -525,9 +525,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > /* > * If this multicast group does not already exist, create it. > */ > - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; > - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { > + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > + if (!p_mgrp) { > OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, > "Creating group, MLID 0x%X\n", cl_ntoh16(mlid)); > > @@ -540,7 +539,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > goto Exit; > } > > - cl_qmap_insert(p_tbl, mlid, &p_mgrp->map_item); > + p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = p_mgrp; > } else { > /* > * The group already exists. If the port is not a > @@ -603,7 +602,6 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, > { > osm_mgrp_t *p_mgrp; > osm_port_t *p_port; > - cl_qmap_t *p_tbl; > ib_api_status_t status = IB_SUCCESS; > > OSM_LOG_ENTER(p_sm->p_log); > @@ -630,9 +628,8 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, > /* > * Get the multicast group object for this group. > */ > - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; > - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { > + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > + if (!p_mgrp) { > CL_PLOCK_RELEASE(p_sm->p_lock); > OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " > "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); > diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c > index 812b022..3e1e3b2 100644 > --- a/opensm/opensm/osm_subnet.c > +++ b/opensm/opensm/osm_subnet.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -73,6 +74,8 @@ static const char null_str[] = "(null)"; > **********************************************************************/ > void osm_subn_construct(IN osm_subn_t * const p_subn) > { > + int index; > + > memset(p_subn, 0, sizeof(*p_subn)); > cl_ptr_vector_construct(&p_subn->port_lid_tbl); > cl_qmap_init(&p_subn->sw_guid_tbl); > @@ -84,19 +87,23 @@ void osm_subn_construct(IN osm_subn_t * const p_subn) > cl_qlist_init(&p_subn->prefix_routes_list); > cl_qmap_init(&p_subn->rtr_guid_tbl); > cl_qmap_init(&p_subn->prtn_pkey_tbl); > - cl_qmap_init(&p_subn->mgrp_mlid_tbl); > + for (index = 0; > + index <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) > + p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; Not needed, memset() is already here (line 3 in this function). Sasha > } > > /********************************************************************** > **********************************************************************/ > void osm_subn_destroy(IN osm_subn_t * const p_subn) > { > + int index; > osm_node_t *p_node, *p_next_node; > osm_port_t *p_port, *p_next_port; > osm_switch_t *p_sw, *p_next_sw; > osm_remote_sm_t *p_rsm, *p_next_rsm; > osm_prtn_t *p_prtn, *p_next_prtn; > - osm_mgrp_t *p_mgrp, *p_next_mgrp; > + osm_mgrp_t *p_mgrp; > osm_infr_t *p_infr, *p_next_infr; > > /* it might be a good idea to de-allocate all known objects */ > @@ -139,12 +146,13 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) > osm_prtn_delete(&p_prtn); > } > > - p_next_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); > - while (p_next_mgrp != > - (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > - p_mgrp = p_next_mgrp; > - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > - osm_mgrp_delete(p_mgrp); > + for (index = 0; > + index <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > + index++) { > + p_mgrp = p_subn->mgrp_mlid_tbl.mgroup[index]; > + if (p_mgrp) > + osm_mgrp_delete(p_mgrp); > + p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; > } > > p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); > > From hrosenstock at xsigo.com Tue Jun 24 15:06:15 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 15:06:15 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/sim_mad.c: Fix some typos Message-ID: <1214345175.13056.334.camel@hrosenstock-ws.xsigo.com> ibsim/sim_mad.c: Fix some typos Signed-off-by: Hal Rosenstock diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 6c5add9..f24844f 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1128,7 +1128,7 @@ Smpfn *get_handle_fn(ib_rpc_t rpc, int response) return fn; } - return 0; // No MGTCLASS matched . + return 0; // No MGTCLASS matched. } int process_packet(Client * cl, void *p, int size, Client ** dcl) @@ -1287,7 +1287,7 @@ int send_trap(Port * port, int trapnum) Port *destport; if (!encode_trapfn) { - IBWARN("trap number %d not upported", trapnum); + IBWARN("trap number %d not supported", trapnum); return -1; } From hrosenstock at xsigo.com Tue Jun 24 15:07:56 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 15:07:56 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim/TODO: Updated for short RMPP support Message-ID: <1214345276.13056.337.camel@hrosenstock-ws.xsigo.com> ibsim/TODO: Updated for short RMPP support Signed-off-by: Hal Rosenstock diff --git a/TODO b/TODO index 616ed71..b5cbc53 100644 --- a/TODO +++ b/TODO @@ -1,7 +1,7 @@ * transaction management - multiple transactions per client - hops simulation - - RMPP + - long RMPP (larger than one MAD) * more attributes handled * MKey support * cleaner (less) debug/warning messages From hrosenstock at xsigo.com Tue Jun 24 15:08:17 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 15:08:17 -0700 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <20080616085236.GM24365@sashak.voltaire.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> Message-ID: <1214345297.13056.339.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-16 at 11:52 +0300, Sasha Khapyorsky wrote: > Another solution would be to encode packet (or payload) length into > header. For this we even don't need to break ABI - there is unused > 64-bit context field. Old sim clients can't talk to new ones due to the length issue. Anyhow, is this important as most apps are dynamically linked to the sim ? -- Hal From sashak at voltaire.com Tue Jun 24 15:07:45 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 01:07:45 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/sim_mad.c: Fix some typos In-Reply-To: <1214345175.13056.334.camel@hrosenstock-ws.xsigo.com> References: <1214345175.13056.334.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624220745.GZ7341@sashak.voltaire.com> On 15:06 Tue 24 Jun , Hal Rosenstock wrote: > ibsim/sim_mad.c: Fix some typos > > Signed-off-by: Hal Rosenstock Applied, Thanks. Sasha From sashak at voltaire.com Tue Jun 24 15:09:16 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 01:09:16 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/TODO: Updated for short RMPP support In-Reply-To: <1214345276.13056.337.camel@hrosenstock-ws.xsigo.com> References: <1214345276.13056.337.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624220916.GA7341@sashak.voltaire.com> On 15:07 Tue 24 Jun , Hal Rosenstock wrote: > ibsim/TODO: Updated for short RMPP support > > Signed-off-by: Hal Rosenstock > > diff --git a/TODO b/TODO > index 616ed71..b5cbc53 100644 > --- a/TODO > +++ b/TODO > @@ -1,7 +1,7 @@ > * transaction management > - multiple transactions per client > - hops simulation > - - RMPP > + - long RMPP (larger than one MAD) I would leave this as is. What we have now is so far from being RMPP support even partial. Sasha From weiny2 at llnl.gov Tue Jun 24 15:17:28 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 24 Jun 2008 15:17:28 -0700 Subject: [ofa-general] Re: [PATCH v2] Add enum strings and *_str functions for enums In-Reply-To: References: <20080415133548.414aeaea.weiny2@llnl.gov> <200806161705.41183.jackm@dev.mellanox.co.il> <20080616102430.505a2000.weiny2@llnl.gov> Message-ID: <20080624151728.692a2cbd.weiny2@llnl.gov> On Tue, 24 Jun 2008 14:54:14 -0700 Roland Dreier wrote: > How about if we just revert the devinfo.c change that used the new > conversion function? I think "down" is a lot nicer in general than > "PORT_DOWN" so we might as well let new code use that. But I do agree > we don't want to break any apps that are parsing ibv_devinfo output. > > So the following seems like i twould work? I like this idea. Ira > > diff --git a/examples/devinfo.c b/examples/devinfo.c > index 86ad7da..caa5d5f 100644 > --- a/examples/devinfo.c > +++ b/examples/devinfo.c > @@ -67,6 +67,17 @@ static const char *guid_str(uint64_t node_guid, char *str) > return str; > } > > +static const char *port_state_str(enum ibv_port_state pstate) > +{ > + switch (pstate) { > + case IBV_PORT_DOWN: return "PORT_DOWN"; > + case IBV_PORT_INIT: return "PORT_INIT"; > + case IBV_PORT_ARMED: return "PORT_ARMED"; > + case IBV_PORT_ACTIVE: return "PORT_ACTIVE"; > + default: return "invalid state"; > + } > +} > + > static const char *port_phy_state_str(uint8_t phys_state) > { > switch (phys_state) { > @@ -254,7 +265,7 @@ static int print_hca_cap(struct ibv_device *ib_dev, uint8_t ib_port) > } > printf("\t\tport:\t%d\n", port); > printf("\t\t\tstate:\t\t\t%s (%d)\n", > - ibv_port_state_str(port_attr.state), port_attr.state); > + port_state_str(port_attr.state), port_attr.state); > printf("\t\t\tmax_mtu:\t\t%s (%d)\n", > mtu_str(port_attr.max_mtu), port_attr.max_mtu); > printf("\t\t\tactive_mtu:\t\t%s (%d)\n", From hrosenstock at xsigo.com Tue Jun 24 15:19:39 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 15:19:39 -0700 Subject: [ofa-general] Re: [PATCH] opensm: Convert mgrp_mlid_tbl into array In-Reply-To: <20080624215852.GY7341@sashak.voltaire.com> References: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> <20080624215852.GY7341@sashak.voltaire.com> Message-ID: <1214345979.13056.349.camel@hrosenstock-ws.xsigo.com> Hi Sasha, On Wed, 2008-06-25 at 00:58 +0300, Sasha Khapyorsky wrote: > Hi Hal, > > On 04:46 Sun 22 Jun , Hal Rosenstock wrote: > > opensm: Convert mgrp_mlid_tbl into array > > Comments are below. > > > > > Signed-off-by: Hal Rosenstock > > > > diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h > > index ef9996c..75be84a 100644 > > --- a/opensm/include/opensm/osm_subnet.h > > +++ b/opensm/include/opensm/osm_subnet.h > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -446,6 +447,10 @@ typedef struct osm_subn_opt { > > * Subnet object > > *********/ > > > > +struct _osm_mgrp_mlid_tbl { > > + void *mgroup[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; > > +}; > > + > > Why this wrapper struct is needed? It's just for clarity (self documenting). Does it need to be removed ? > > /****s* OpenSM: Subnet/osm_subn_t > > * NAME > > * osm_subn_t > > @@ -467,7 +472,6 @@ typedef struct osm_subn { > > cl_qmap_t rtr_guid_tbl; > > cl_qlist_t prefix_routes_list; > > cl_qmap_t prtn_pkey_tbl; > > - cl_qmap_t mgrp_mlid_tbl; > > cl_qmap_t sm_guid_tbl; > > cl_qlist_t sa_sr_list; > > cl_qlist_t sa_infr_list; > > @@ -490,6 +494,7 @@ typedef struct osm_subn { > > boolean_t first_time_master_sweep; > > boolean_t coming_out_of_standby; > > unsigned need_update; > > + struct _osm_mgrp_mlid_tbl mgrp_mlid_tbl; > > } osm_subn_t; > > /* > > * FIELDS > > @@ -513,10 +518,6 @@ typedef struct osm_subn { > > * Container of pointers to all Partition objects in the subnet. > > * Indexed by P_KEY. > > * > > -* mgrp_mlid_tbl > > -* Container of pointers to all Multicast Group objects in the subnet. > > -* Indexed by MLID. > > -* > > * sm_guid_tbl > > * Container of pointers to SM objects representing other SMs > > * on the subnet. > > @@ -615,6 +616,10 @@ typedef struct osm_subn { > > * This flag should be on during first non-master heavy (including > > * pre-master discovery stage) > > * > > +* mgrp_mlid_tbl > > +* Array of pointers to all Multicast Group objects in the subnet. > > +* Indexed by MLID offset from base MLID. > > +* > > * SEE ALSO > > * Subnet object > > *********/ > > diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c > > index e2f5828..44ee8eb 100644 > > --- a/opensm/opensm/osm_drop_mgr.c > > +++ b/opensm/opensm/osm_drop_mgr.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -163,6 +164,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > > ib_gid_t port_gid; > > ib_mad_notice_attr_t notice; > > ib_api_status_t status; > > + int index; > > Here and below, 'int i' looks much more friendly as loop index variable. OK. > > OSM_LOG_ENTER(sm->p_log); > > > > @@ -206,14 +208,15 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) > > > > p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); > > while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { > > - p_mgrp = > > - (osm_mgrp_t *) cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, > > - p_mcm->mlid); > > - if (p_mgrp != > > - (osm_mgrp_t *) cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { > > - osm_mgrp_remove_port(sm->p_subn, sm->p_log, > > - p_mgrp, p_port->guid); > > - osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > > + for (index = 0; > > + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > + p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgrp) { > > + osm_mgrp_remove_port(sm->p_subn, sm->p_log, > > + p_mgrp, p_port->guid); > > + osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > > + } > > Here you have p_mcm->mlid which is index of multicast group where this > port is member + IB_LID_MCAST_START_HO. So you don't need to loop other > all groups in the array. Actually there only one line require changing: > > p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[p_mcm->mlid - > IB_LID_MCAST_START_HO] > if (p_mgrp) { > osm_mgrp_remove_port(sm->p_subn, sm->p_log, p_mgrp, > p_port->guid); > osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > } Yes, that's much better. > > } > > p_mcm = > > (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); > > diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c > > index d7a0f56..15a585e 100644 > > --- a/opensm/opensm/osm_mcast_mgr.c > > +++ b/opensm/opensm/osm_mcast_mgr.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -1132,6 +1133,8 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, > > IN ib_net64_t port_guid) > > { > > ib_api_status_t status; > > + osm_mgrp_t *p_mgroup; > > + int index; > > > > OSM_LOG_ENTER(sm->p_log); > > > > @@ -1154,8 +1157,15 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, > > /* Send a Report to any InformInfo registered for > > Trap 67 : MCGroup delete */ > > osm_mgrp_send_delete_notice(sm->p_subn, sm->p_log, p_mgrp); > > - cl_qmap_remove_item(&sm->p_subn->mgrp_mlid_tbl, > > - (cl_map_item_t *) p_mgrp); > > + for (index = 0; > > + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > + p_mgroup = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgroup == p_mgrp) { > > + sm->p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; > > + break; > > + } > > + } > > The same is here. Ditto. > > osm_mgrp_delete(p_mgrp); > > } > > > > @@ -1171,16 +1181,14 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > > osm_signal_t signal; > > osm_switch_t *p_sw; > > cl_qmap_t *p_sw_tbl; > > - cl_qmap_t *p_mcast_tbl; > > cl_qlist_t *p_list = &sm->mgrp_list; > > osm_mgrp_t *p_mgrp; > > - osm_mgrp_t *p_next_mgrp; > > boolean_t pending_transactions = FALSE; > > + int index; > > > > OSM_LOG_ENTER(sm->p_log); > > > > p_sw_tbl = &sm->p_subn->sw_guid_tbl; > > - p_mcast_tbl = &sm->p_subn->mgrp_mlid_tbl; > > /* > > While holding the lock, iterate over all the established > > multicast groups, servicing each in turn. > > @@ -1189,17 +1197,19 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > > */ > > CL_PLOCK_EXCL_ACQUIRE(sm->p_lock); > > > > - p_mgrp = (osm_mgrp_t *) cl_qmap_head(p_mcast_tbl); > > - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(p_mcast_tbl)) { > > + for (index = 0; > > + index <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > /* > > We reached here due to some change that caused a heavy sweep > > of the subnet. Not due to a specific multicast request. > > So the request type is subnet_change and the port guid is 0. > > */ > > - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > > - mcast_mgr_process_mgrp(sm, p_mgrp, > > - OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, 0); > > - p_mgrp = p_next_mgrp; > > + p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgrp) > > + mcast_mgr_process_mgrp(sm, p_mgrp, > > + OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, > > + 0); > > } > > > > /* > > @@ -1233,13 +1243,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) > > static > > osm_mgrp_t *__get_mgrp_by_mlid(osm_sm_t * sm, IN ib_net16_t const mlid) > > { > > - cl_map_item_t *map_item; > > - > > - map_item = cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, mlid); > > - if (map_item == cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { > > - return NULL; > > - } > > - return (osm_mgrp_t *) map_item; > > + return(sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); > > } > > Same function exists in osm_sa_mcmember_record.c. It is probably good > time to consolidate it in something like osm_get_mgrp_by_mlid() (it > could be placed in osm_subnet.c where another osm_get_*_by() hosted). Sure; is it OK for that to be an incremental patch on top of this to follow shortly ? > > > > /********************************************************************** > > diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c > > index f1d4e54..5a8dd3b 100644 > > --- a/opensm/opensm/osm_qos_policy.c > > +++ b/opensm/opensm/osm_qos_policy.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2006 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -792,11 +793,8 @@ static void __qos_policy_validate_pkey( > > if (!p_prtn->mlid) > > return; > > > > - p_mgrp = (osm_mgrp_t *) cl_qmap_get( > > - &p_qos_policy->p_subn->mgrp_mlid_tbl, > > - p_prtn->mlid); > > - if (p_mgrp == (osm_mgrp_t *) > > - cl_qmap_end(&p_qos_policy->p_subn->mgrp_mlid_tbl)) { > > + p_mgrp = p_qos_policy->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(p_prtn->mlid) - IB_LID_MCAST_START_HO]; > > + if (!p_mgrp) { > > OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, > > "ERR AC16: MCast group for partition with " > > "pkey 0x%04X not found\n", > > diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c > > index adf4931..4695bc0 100644 > > --- a/opensm/opensm/osm_sa.c > > +++ b/opensm/opensm/osm_sa.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -558,12 +559,11 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) > > p_mcm_port->scope_state, p_mcm_port->proxy_join); > > } > > > > -static void sa_dump_one_mgrp(cl_map_item_t * p_map_item, void *cxt) > > +static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) > > { > > struct opensm_dump_context dump_context; > > osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; > > FILE *file = ((struct opensm_dump_context *)cxt)->file; > > - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; > > > > fprintf(file, "MC Group 0x%04x %s:" > > " mgid=0x%016" PRIx64 ":0x%016" PRIx64 > > @@ -700,13 +700,20 @@ static void sa_dump_one_service(cl_list_item_t * p_list_item, void *cxt) > > static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) > > { > > struct opensm_dump_context dump_context; > > + osm_mgrp_t *p_mgrp; > > + int index; > > > > dump_context.p_osm = p_osm; > > dump_context.file = file; > > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump multicast:\n"); > > cl_plock_acquire(&p_osm->lock); > > - cl_qmap_apply_func(&p_osm->subn.mgrp_mlid_tbl, > > - sa_dump_one_mgrp, &dump_context); > > + for (index = 0; > > + index <= p_osm->subn.max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > + p_mgrp = p_osm->subn.mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgrp) > > + sa_dump_one_mgrp(p_mgrp, &dump_context); > > + } > > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform:\n"); > > cl_qlist_apply_func(&p_osm->subn.sa_infr_list, > > sa_dump_one_inform, &dump_context); > > @@ -729,14 +736,12 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, > > unsigned well_known) > > { > > ib_net64_t comp_mask; > > - cl_map_item_t *p_next; > > osm_mgrp_t *p_mgrp; > > > > cl_plock_excl_acquire(&p_osm->lock); > > > > - if ((p_next = cl_qmap_get(&p_osm->subn.mgrp_mlid_tbl, mlid)) != > > - cl_qmap_end(&p_osm->subn.mgrp_mlid_tbl)) { > > - p_mgrp = (osm_mgrp_t *) p_next; > > + p_mgrp = p_osm->subn.mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > > + if (p_mgrp) { > > if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, > > sizeof(ib_gid_t))) { > > OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, > > diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c > > index 46c87c7..7049fc0 100644 > > --- a/opensm/opensm/osm_sa_mcmember_record.c > > +++ b/opensm/opensm/osm_sa_mcmember_record.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -96,13 +97,7 @@ typedef struct osm_sa_mcmr_search_ctxt { > > static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, > > IN ib_net16_t const mlid) > > { > > - cl_map_item_t *map_item; > > - > > - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); > > - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { > > - return NULL; > > - } > > - return (osm_mgrp_t *) map_item; > > + return(sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); > > } > > > > /********************************************************************* > > @@ -154,16 +149,20 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > > if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO > > && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho > > - && cl_qmap_get(&p_subn->mgrp_mlid_tbl, > > - requested_mlid) == > > - cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > + && !p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { > > mlid = cl_ntoh16(requested_mlid); > > goto Exit; > > } > > > > /* If MCGroups table is empty, first return the min mlid */ > > - p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); > > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > + max_num_mlids = sa->p_subn->max_multicast_lid_ho - > > + IB_LID_MCAST_START_HO + 1; > > + for (idx = 0; idx < max_num_mlids; idx++) { > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > > + if (p_mgrp) > > + break; > > + } > > + if (!p_mgrp) { > > mlid = IB_LID_MCAST_START_HO; > > OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, > > "No multicast groups found using minimal mlid:0x%04X\n", > > @@ -171,9 +170,6 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > goto Exit; > > } > > > > - max_num_mlids = > > - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; > > - > > /* track all used mlids in the array (by mlid index) */ > > used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); > > if (!used_mlids_array) > > @@ -181,9 +177,10 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); > > > > /* scan all available multicast groups in the DB and fill in the table */ > > - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > + for (idx = 0; idx < max_num_mlids; idx++) { > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > > /* ignore mgrps marked for deletion */ > > - if (p_mgrp->to_be_deleted == FALSE) { > > + if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { > > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 > > " : " "0x%016" PRIx64 "\n", > > @@ -202,11 +199,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > cl_ntoh16(p_mgrp->mlid), > > max_num_mlids + IB_LID_MCAST_START_HO); > > } else { > > - used_mlids_array[cl_ntoh16(p_mgrp->mlid) - > > - IB_LID_MCAST_START_HO] = 1; > > + used_mlids_array[idx] = 1; > > } > > } > > - p_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > > } > > > > /* Find "mlid holes" in the mgrp table */ > > There were some dirty tricks when qmap was used and finally they did > an additional mlid array (that with malloc()) for free mlid searching. > But now you already have such array, what is the reason to copy this and > to run additional loop? It can be eliminated in an incremental patch. OK ? > > @@ -247,8 +242,7 @@ __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) > > not a well known group */ > > if (p_mgrp && cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && > > p_mgrp->well_known == FALSE) { > > - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, > > - (cl_map_item_t *) p_mgrp); > > + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; > > osm_mgrp_delete(p_mgrp); > > } > > } > > @@ -1033,21 +1027,17 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, > > /* since we might have an old group by that mlid > > one whose deletion was delayed for an idle time > > we need to deallocate it first */ > > - p_prev_mgrp = > > - (osm_mgrp_t *) cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); > > - if (p_prev_mgrp != > > - (osm_mgrp_t *) cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { > > + p_prev_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > > + if (p_prev_mgrp) { > > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > "Found previous group for mlid:0x%04x - " > > "Destroying it first\n", > > cl_ntoh16(mlid)); > > - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, > > - (cl_map_item_t *) p_prev_mgrp); > > + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; > > osm_mgrp_delete(p_prev_mgrp); > > } > > > > - cl_qmap_insert(&sa->p_subn->mgrp_mlid_tbl, > > - mlid, &(*pp_mgrp)->map_item); > > + sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; > > > > /* Send a Report to any InformInfo registered for > > Trap 66: MCGroup create */ > > @@ -1069,9 +1059,8 @@ typedef struct osm_sa_pr_mcmr_search_ctxt { > > /********************************************************************** > > *********************************************************************/ > > static void > > -__search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > > +__search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) > > { > > - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; > > osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = > > (osm_sa_pr_mcmr_search_ctxt_t *) context; > > const ib_gid_t *p_recvd_mgid; > > @@ -1135,23 +1124,30 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) > > **********************************************************************/ > > ib_api_status_t > > osm_get_mgrp_by_mgid(IN osm_sa_t *sa, > > - IN ib_gid_t *p_mgid, > > - OUT osm_mgrp_t **pp_mgrp) > > + IN ib_gid_t *p_mgid, > > + OUT osm_mgrp_t **pp_mgrp) > > { > > osm_sa_pr_mcmr_search_ctxt_t mcmr_search_context; > > + osm_mgrp_t *p_mgrp; > > + int index; > > > > mcmr_search_context.p_mgid = p_mgid; > > mcmr_search_context.sa = sa; > > mcmr_search_context.p_mgrp = NULL; > > > > - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, > > - __search_mgrp_by_mgid, &mcmr_search_context); > > - > > - if (mcmr_search_context.p_mgrp == NULL) > > - return IB_NOT_FOUND; > > - > > - *pp_mgrp = mcmr_search_context.p_mgrp; > > - return IB_SUCCESS; > > + for (index = 0; > > + index <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgrp) { > > + __search_mgrp_by_mgid(p_mgrp, &mcmr_search_context); > > + if (mcmr_search_context.p_mgrp) { > > + *pp_mgrp = mcmr_search_context.p_mgrp; > > + return IB_SUCCESS; > > + } > > + } > > + } > > + return IB_NOT_FOUND; > > } > > > > /********************************************************************** > > @@ -1619,10 +1615,9 @@ Exit: > > Match the given mgrp to the requested mcmr > > **********************************************************************/ > > static void > > -__osm_sa_mcm_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, > > +__osm_sa_mcm_by_comp_mask_cb(IN osm_mgrp_t * const p_mgrp, > > IN void *context) > > { > > - const osm_mgrp_t *const p_mgrp = (osm_mgrp_t *) p_map_item; > > osm_sa_mcmr_search_ctxt_t *const p_ctxt = > > (osm_sa_mcmr_search_ctxt_t *) context; > > osm_sa_t *sa = p_ctxt->sa; > > @@ -1811,6 +1806,8 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, > > ib_net64_t comp_mask; > > osm_physp_t *p_req_physp; > > boolean_t trusted_req; > > + osm_mgrp_t *p_mgrp; > > + int index; > > > > OSM_LOG_ENTER(sa->p_log); > > > > @@ -1846,8 +1843,14 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, > > CL_PLOCK_ACQUIRE(sa->p_lock); > > > > /* simply go over all MCGs and match */ > > - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, > > - __osm_sa_mcm_by_comp_mask_cb, &context); > > + for (index = 0; > > + index <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgrp) { > > + __osm_sa_mcm_by_comp_mask_cb(p_mgrp, &context); > > + } > > + } > > > > CL_PLOCK_RELEASE(sa->p_lock); > > > > diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c > > index 2a9155c..fb42afb 100644 > > --- a/opensm/opensm/osm_sa_path_record.c > > +++ b/opensm/opensm/osm_sa_path_record.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -1472,14 +1473,7 @@ __osm_pr_rcv_process_pair(IN osm_sa_t * sa, > > static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, > > IN ib_net16_t const mlid) > > { > > - cl_map_item_t *map_item; > > - > > - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); > > - > > - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) > > - return NULL; > > - > > - return (osm_mgrp_t *) map_item; > > + return(sa->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); > > } > > Hmm, I thought there are two instances of __get_mgrp_by_mlid(), this one > os third. Yes, there were/are three instances of this code. > > /********************************************************************** > > diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c > > index 7393ead..c6526e7 100644 > > --- a/opensm/opensm/osm_sm.c > > +++ b/opensm/opensm/osm_sm.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -498,7 +499,6 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > > { > > osm_mgrp_t *p_mgrp; > > osm_port_t *p_port; > > - cl_qmap_t *p_tbl; > > ib_api_status_t status = IB_SUCCESS; > > osm_mcm_info_t *p_mcm; > > > > @@ -525,9 +525,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > > /* > > * If this multicast group does not already exist, create it. > > */ > > - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; > > - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); > > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { > > + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > > + if (!p_mgrp) { > > OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, > > "Creating group, MLID 0x%X\n", cl_ntoh16(mlid)); > > > > @@ -540,7 +539,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, > > goto Exit; > > } > > > > - cl_qmap_insert(p_tbl, mlid, &p_mgrp->map_item); > > + p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = p_mgrp; > > } else { > > /* > > * The group already exists. If the port is not a > > @@ -603,7 +602,6 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, > > { > > osm_mgrp_t *p_mgrp; > > osm_port_t *p_port; > > - cl_qmap_t *p_tbl; > > ib_api_status_t status = IB_SUCCESS; > > > > OSM_LOG_ENTER(p_sm->p_log); > > @@ -630,9 +628,8 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, > > /* > > * Get the multicast group object for this group. > > */ > > - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; > > - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); > > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { > > + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; > > + if (!p_mgrp) { > > CL_PLOCK_RELEASE(p_sm->p_lock); > > OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " > > "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); > > diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c > > index 812b022..3e1e3b2 100644 > > --- a/opensm/opensm/osm_subnet.c > > +++ b/opensm/opensm/osm_subnet.c > > @@ -2,6 +2,7 @@ > > * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > > + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. > > * > > * This software is available to you under a choice of one of two > > * licenses. You may choose to be licensed under the terms of the GNU > > @@ -73,6 +74,8 @@ static const char null_str[] = "(null)"; > > **********************************************************************/ > > void osm_subn_construct(IN osm_subn_t * const p_subn) > > { > > + int index; > > + > > memset(p_subn, 0, sizeof(*p_subn)); > > cl_ptr_vector_construct(&p_subn->port_lid_tbl); > > cl_qmap_init(&p_subn->sw_guid_tbl); > > @@ -84,19 +87,23 @@ void osm_subn_construct(IN osm_subn_t * const p_subn) > > cl_qlist_init(&p_subn->prefix_routes_list); > > cl_qmap_init(&p_subn->rtr_guid_tbl); > > cl_qmap_init(&p_subn->prtn_pkey_tbl); > > - cl_qmap_init(&p_subn->mgrp_mlid_tbl); > > + for (index = 0; > > + index <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) > > + p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; > > Not needed, memset() is already here (line 3 in this function). OK. -- Hal > Sasha > > > } > > > > /********************************************************************** > > **********************************************************************/ > > void osm_subn_destroy(IN osm_subn_t * const p_subn) > > { > > + int index; > > osm_node_t *p_node, *p_next_node; > > osm_port_t *p_port, *p_next_port; > > osm_switch_t *p_sw, *p_next_sw; > > osm_remote_sm_t *p_rsm, *p_next_rsm; > > osm_prtn_t *p_prtn, *p_next_prtn; > > - osm_mgrp_t *p_mgrp, *p_next_mgrp; > > + osm_mgrp_t *p_mgrp; > > osm_infr_t *p_infr, *p_next_infr; > > > > /* it might be a good idea to de-allocate all known objects */ > > @@ -139,12 +146,13 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) > > osm_prtn_delete(&p_prtn); > > } > > > > - p_next_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); > > - while (p_next_mgrp != > > - (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > - p_mgrp = p_next_mgrp; > > - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > > - osm_mgrp_delete(p_mgrp); > > + for (index = 0; > > + index <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; > > + index++) { > > + p_mgrp = p_subn->mgrp_mlid_tbl.mgroup[index]; > > + if (p_mgrp) > > + osm_mgrp_delete(p_mgrp); > > + p_subn->mgrp_mlid_tbl.mgroup[index] = NULL; > > } > > > > p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); > > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Tue Jun 24 15:22:33 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 01:22:33 +0300 Subject: [ofa-general] Re: [PATCH] ibsim: Support short RMPP packets In-Reply-To: <1214345297.13056.339.camel@hrosenstock-ws.xsigo.com> References: <1213192410.14047.320.camel@hrosenstock-ws.xsigo.com> <20080612205157.GZ6256@sashak.voltaire.com> <1213306815.14047.545.camel@hrosenstock-ws.xsigo.com> <20080616085236.GM24365@sashak.voltaire.com> <1214345297.13056.339.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624222233.GB7341@sashak.voltaire.com> On 15:08 Tue 24 Jun , Hal Rosenstock wrote: > > Old sim clients can't talk to new ones due to the length issue. Sure. Packet format is same, but interface was changed. > Anyhow, is this important as most apps are dynamically linked to the > sim ? ibsim and umad2sim are packaged together, I don't expect separate usage. Sasha From hrosenstock at xsigo.com Tue Jun 24 15:31:27 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 15:31:27 -0700 Subject: [ofa-general] [PATCH 1/2] libibmad: add a drpath2str function for easier printing of DR Paths. In-Reply-To: <20080624144838.55ec7e3d.weiny2@llnl.gov> References: <20080624144838.55ec7e3d.weiny2@llnl.gov> Message-ID: <1214346687.13056.352.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-24 at 14:48 -0700, Ira Weiny wrote: > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -622,6 +622,7 @@ enum { > char * portid2str(ib_portid_t *portid); > int portid2portnum(ib_portid_t *portid); > int str2drpath(ib_dr_path_t *path, char *routepath, int drslid, > int drdlid); > +char * drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size); Should drpath2str be added to libibmad.map too ? -- Hal From sashak at voltaire.com Tue Jun 24 15:36:36 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 01:36:36 +0300 Subject: [ofa-general] Re: [PATCH] opensm: Convert mgrp_mlid_tbl into array In-Reply-To: <1214345979.13056.349.camel@hrosenstock-ws.xsigo.com> References: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> <20080624215852.GY7341@sashak.voltaire.com> <1214345979.13056.349.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624223636.GC7341@sashak.voltaire.com> On 15:19 Tue 24 Jun , Hal Rosenstock wrote: > > > > > > +struct _osm_mgrp_mlid_tbl { > > > + void *mgroup[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; > > > +}; > > > + > > > > Why this wrapper struct is needed? > > It's just for clarity (self documenting). Does it need to be removed ? Yes, please - this makes the code less readable (I spent some time in attempts to understand why it was done so). > > Here you have p_mcm->mlid which is index of multicast group where this > > port is member + IB_LID_MCAST_START_HO. So you don't need to loop other > > all groups in the array. Actually there only one line require changing: > > > > p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[p_mcm->mlid - > > IB_LID_MCAST_START_HO] > > if (p_mgrp) { > > osm_mgrp_remove_port(sm->p_subn, sm->p_log, p_mgrp, > > p_port->guid); > > osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > > } > > Yes, that's much better. But check about possibly needed cl_ntoh16(p_mcm->mlid) - I don't remember how mcm stores mlid. > > Same function exists in osm_sa_mcmember_record.c. It is probably good > > time to consolidate it in something like osm_get_mgrp_by_mlid() (it > > could be placed in osm_subnet.c where another osm_get_*_by() hosted). > > Sure; is it OK for that to be an incremental patch on top of this to > follow shortly ? Yes, if it is simpler for you. I would make osm_get_mgrp_by_mlid() as separate patch even before to array converting, but it is really minor. > > > @@ -154,16 +149,20 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > > > > if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO > > > && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho > > > - && cl_qmap_get(&p_subn->mgrp_mlid_tbl, > > > - requested_mlid) == > > > - cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > > + && !p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { > > > mlid = cl_ntoh16(requested_mlid); > > > goto Exit; > > > } > > > > > > /* If MCGroups table is empty, first return the min mlid */ > > > - p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); > > > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > > + max_num_mlids = sa->p_subn->max_multicast_lid_ho - > > > + IB_LID_MCAST_START_HO + 1; > > > + for (idx = 0; idx < max_num_mlids; idx++) { > > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > > > + if (p_mgrp) > > > + break; > > > + } > > > + if (!p_mgrp) { > > > mlid = IB_LID_MCAST_START_HO; > > > OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, > > > "No multicast groups found using minimal mlid:0x%04X\n", > > > @@ -171,9 +170,6 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > goto Exit; > > > } > > > > > > - max_num_mlids = > > > - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; > > > - > > > /* track all used mlids in the array (by mlid index) */ > > > used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); > > > if (!used_mlids_array) > > > @@ -181,9 +177,10 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); > > > > > > /* scan all available multicast groups in the DB and fill in the table */ > > > - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > > + for (idx = 0; idx < max_num_mlids; idx++) { > > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > > > /* ignore mgrps marked for deletion */ > > > - if (p_mgrp->to_be_deleted == FALSE) { > > > + if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { > > > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > > "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 > > > " : " "0x%016" PRIx64 "\n", > > > @@ -202,11 +199,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > cl_ntoh16(p_mgrp->mlid), > > > max_num_mlids + IB_LID_MCAST_START_HO); > > > } else { > > > - used_mlids_array[cl_ntoh16(p_mgrp->mlid) - > > > - IB_LID_MCAST_START_HO] = 1; > > > + used_mlids_array[idx] = 1; > > > } > > > } > > > - p_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > > > } > > > > > > /* Find "mlid holes" in the mgrp table */ > > > > There were some dirty tricks when qmap was used and finally they did > > an additional mlid array (that with malloc()) for free mlid searching. > > But now you already have such array, what is the reason to copy this and > > to run additional loop? > > It can be eliminated in an incremental patch. OK ? I would prefer to have this improved in v2 of the patch. This function was never ideal, but now it looks (and works) really bad. Sasha From hrosenstock at xsigo.com Tue Jun 24 16:02:12 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 16:02:12 -0700 Subject: [ofa-general] Re: [PATCH] opensm: Convert mgrp_mlid_tbl into array In-Reply-To: <20080624223636.GC7341@sashak.voltaire.com> References: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> <20080624215852.GY7341@sashak.voltaire.com> <1214345979.13056.349.camel@hrosenstock-ws.xsigo.com> <20080624223636.GC7341@sashak.voltaire.com> Message-ID: <1214348532.13056.358.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 01:36 +0300, Sasha Khapyorsky wrote: > On 15:19 Tue 24 Jun , Hal Rosenstock wrote: > > > > > > > > +struct _osm_mgrp_mlid_tbl { > > > > + void *mgroup[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; > > > > +}; > > > > + > > > > > > Why this wrapper struct is needed? > > > > It's just for clarity (self documenting). Does it need to be removed ? > > Yes, please - this makes the code less readable (I spent some time in > attempts to understand why it was done so). OK. > > > Here you have p_mcm->mlid which is index of multicast group where this > > > port is member + IB_LID_MCAST_START_HO. So you don't need to loop other > > > all groups in the array. Actually there only one line require changing: > > > > > > p_mgrp = sm->p_subn->mgrp_mlid_tbl.mgroup[p_mcm->mlid - > > > IB_LID_MCAST_START_HO] > > > if (p_mgrp) { > > > osm_mgrp_remove_port(sm->p_subn, sm->p_log, p_mgrp, > > > p_port->guid); > > > osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); > > > } > > > > Yes, that's much better. > > But check about possibly needed cl_ntoh16(p_mcm->mlid) - I don't remember > how mcm stores mlid. It's in net format so does need the conversion. > > > Same function exists in osm_sa_mcmember_record.c. It is probably good > > > time to consolidate it in something like osm_get_mgrp_by_mlid() (it > > > could be placed in osm_subnet.c where another osm_get_*_by() hosted). > > > > Sure; is it OK for that to be an incremental patch on top of this to > > follow shortly ? > > Yes, if it is simpler for you. I would make osm_get_mgrp_by_mlid() as > separate patch even before to array converting, but it is really minor. Sure; it could be done before, during, or after this patch. I do prefer to do it after. > > > > @@ -154,16 +149,20 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > > > > > > if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO > > > > && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho > > > > - && cl_qmap_get(&p_subn->mgrp_mlid_tbl, > > > > - requested_mlid) == > > > > - cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > > > + && !p_subn->mgrp_mlid_tbl.mgroup[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { > > > > mlid = cl_ntoh16(requested_mlid); > > > > goto Exit; > > > > } > > > > > > > > /* If MCGroups table is empty, first return the min mlid */ > > > > - p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); > > > > - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > > > + max_num_mlids = sa->p_subn->max_multicast_lid_ho - > > > > + IB_LID_MCAST_START_HO + 1; > > > > + for (idx = 0; idx < max_num_mlids; idx++) { > > > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > > > > + if (p_mgrp) > > > > + break; > > > > + } > > > > + if (!p_mgrp) { > > > > mlid = IB_LID_MCAST_START_HO; > > > > OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, > > > > "No multicast groups found using minimal mlid:0x%04X\n", > > > > @@ -171,9 +170,6 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > > goto Exit; > > > > } > > > > > > > > - max_num_mlids = > > > > - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; > > > > - > > > > /* track all used mlids in the array (by mlid index) */ > > > > used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); > > > > if (!used_mlids_array) > > > > @@ -181,9 +177,10 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > > memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); > > > > > > > > /* scan all available multicast groups in the DB and fill in the table */ > > > > - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { > > > > + for (idx = 0; idx < max_num_mlids; idx++) { > > > > + p_mgrp = sa->p_subn->mgrp_mlid_tbl.mgroup[idx]; > > > > /* ignore mgrps marked for deletion */ > > > > - if (p_mgrp->to_be_deleted == FALSE) { > > > > + if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { > > > > OSM_LOG(sa->p_log, OSM_LOG_DEBUG, > > > > "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 > > > > " : " "0x%016" PRIx64 "\n", > > > > @@ -202,11 +199,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) > > > > cl_ntoh16(p_mgrp->mlid), > > > > max_num_mlids + IB_LID_MCAST_START_HO); > > > > } else { > > > > - used_mlids_array[cl_ntoh16(p_mgrp->mlid) - > > > > - IB_LID_MCAST_START_HO] = 1; > > > > + used_mlids_array[idx] = 1; > > > > } > > > > } > > > > - p_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); > > > > } > > > > > > > > /* Find "mlid holes" in the mgrp table */ > > > > > > There were some dirty tricks when qmap was used and finally they did > > > an additional mlid array (that with malloc()) for free mlid searching. > > > But now you already have such array, what is the reason to copy this and > > > to run additional loop? > > > > It can be eliminated in an incremental patch. OK ? > > I would prefer to have this improved in v2 of the patch. This function > was never ideal, but now it looks (and works) really bad. It's no worse than it was; can't it wait a little ? -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Tue Jun 24 16:19:23 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 02:19:23 +0300 Subject: [ofa-general] Re: [PATCH] opensm: Convert mgrp_mlid_tbl into array In-Reply-To: <1214348532.13056.358.camel@hrosenstock-ws.xsigo.com> References: <1214135198.13056.217.camel@hrosenstock-ws.xsigo.com> <20080624215852.GY7341@sashak.voltaire.com> <1214345979.13056.349.camel@hrosenstock-ws.xsigo.com> <20080624223636.GC7341@sashak.voltaire.com> <1214348532.13056.358.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624231923.GD7341@sashak.voltaire.com> On 16:02 Tue 24 Jun , Hal Rosenstock wrote: > > > > I would prefer to have this improved in v2 of the patch. This function > > was never ideal, but now it looks (and works) really bad. > > It's no worse than it was; can't it wait a little ? Ok, I can do it during merge, it is not a big deal. Sasha From hrosenstock at xsigo.com Tue Jun 24 16:28:48 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Tue, 24 Jun 2008 16:28:48 -0700 Subject: [ofa-general] [PATCHv2] opensm: Convert mgrp_mlid_tbl into array Message-ID: <1214350128.13056.367.camel@hrosenstock-ws.xsigo.com> opensm: Convert mgrp_mlid_tbl into array Changes from v1: Remove wrapper structure for array of void pointers In osm_drop_mgr.c and osm_mcast_mgr.c, use mlid rather than walking array Eliminate unneeded initialization in osm_subnet.c:osm_subn_construct Additional patches to follow: 1. Consolidate get mgrp by mlid code into osm_get_mgrp_by_mlid function 2. Eliminate no longer needed used_mlids_array in __get_new_mlid Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 61fb700..5e15f0e 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -472,7 +473,6 @@ typedef struct osm_subn { cl_qmap_t rtr_guid_tbl; cl_qlist_t prefix_routes_list; cl_qmap_t prtn_pkey_tbl; - cl_qmap_t mgrp_mlid_tbl; cl_qmap_t sm_guid_tbl; cl_qlist_t sa_sr_list; cl_qlist_t sa_infr_list; @@ -495,6 +495,7 @@ typedef struct osm_subn { boolean_t first_time_master_sweep; boolean_t coming_out_of_standby; unsigned need_update; + void *mgrp_mlid_tbl[IB_LID_MCAST_END_HO - IB_LID_MCAST_START_HO + 1]; } osm_subn_t; /* * FIELDS @@ -518,10 +519,6 @@ typedef struct osm_subn { * Container of pointers to all Partition objects in the subnet. * Indexed by P_KEY. * -* mgrp_mlid_tbl -* Container of pointers to all Multicast Group objects in the subnet. -* Indexed by MLID. -* * sm_guid_tbl * Container of pointers to SM objects representing other SMs * on the subnet. @@ -620,6 +617,10 @@ typedef struct osm_subn { * This flag should be on during first non-master heavy (including * pre-master discovery stage) * +* mgrp_mlid_tbl +* Array of pointers to all Multicast Group objects in the subnet. +* Indexed by MLID offset from base MLID. +* * SEE ALSO * Subnet object *********/ diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index e2f5828..57ec18a 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -206,11 +207,8 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { - p_mgrp = - (osm_mgrp_t *) cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, - p_mcm->mlid); - if (p_mgrp != - (osm_mgrp_t *) cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { + p_mgrp = sm->p_subn->mgrp_mlid_tbl[cl_ntoh16(p_mcm->mlid) - IB_LID_MCAST_START_HO]; + if (p_mgrp) { osm_mgrp_remove_port(sm->p_subn, sm->p_log, p_mgrp, p_port->guid); osm_mcm_info_delete((osm_mcm_info_t *) p_mcm); diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index d7a0f56..1a2be8d 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -1154,8 +1155,7 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, /* Send a Report to any InformInfo registered for Trap 67 : MCGroup delete */ osm_mgrp_send_delete_notice(sm->p_subn, sm->p_log, p_mgrp); - cl_qmap_remove_item(&sm->p_subn->mgrp_mlid_tbl, - (cl_map_item_t *) p_mgrp); + sm->p_subn->mgrp_mlid_tbl[cl_ntoh16(p_mgrp->mlid) - IB_LID_MCAST_START_HO] = NULL; osm_mgrp_delete(p_mgrp); } @@ -1171,16 +1171,14 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) osm_signal_t signal; osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; - cl_qmap_t *p_mcast_tbl; cl_qlist_t *p_list = &sm->mgrp_list; osm_mgrp_t *p_mgrp; - osm_mgrp_t *p_next_mgrp; boolean_t pending_transactions = FALSE; + int i; OSM_LOG_ENTER(sm->p_log); p_sw_tbl = &sm->p_subn->sw_guid_tbl; - p_mcast_tbl = &sm->p_subn->mgrp_mlid_tbl; /* While holding the lock, iterate over all the established multicast groups, servicing each in turn. @@ -1189,17 +1187,19 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) */ CL_PLOCK_EXCL_ACQUIRE(sm->p_lock); - p_mgrp = (osm_mgrp_t *) cl_qmap_head(p_mcast_tbl); - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(p_mcast_tbl)) { + for (i = 0; + i <= sm->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + i++) { /* We reached here due to some change that caused a heavy sweep of the subnet. Not due to a specific multicast request. So the request type is subnet_change and the port guid is 0. */ - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); - mcast_mgr_process_mgrp(sm, p_mgrp, - OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, 0); - p_mgrp = p_next_mgrp; + p_mgrp = sm->p_subn->mgrp_mlid_tbl[i]; + if (p_mgrp) + mcast_mgr_process_mgrp(sm, p_mgrp, + OSM_MCAST_REQ_TYPE_SUBNET_CHANGE, + 0); } /* @@ -1233,13 +1233,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) static osm_mgrp_t *__get_mgrp_by_mlid(osm_sm_t * sm, IN ib_net16_t const mlid) { - cl_map_item_t *map_item; - - map_item = cl_qmap_get(&sm->p_subn->mgrp_mlid_tbl, mlid); - if (map_item == cl_qmap_end(&sm->p_subn->mgrp_mlid_tbl)) { - return NULL; - } - return (osm_mgrp_t *) map_item; + return(sm->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); } /********************************************************************** diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index f1d4e54..2599730 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2006 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -792,11 +793,8 @@ static void __qos_policy_validate_pkey( if (!p_prtn->mlid) return; - p_mgrp = (osm_mgrp_t *) cl_qmap_get( - &p_qos_policy->p_subn->mgrp_mlid_tbl, - p_prtn->mlid); - if (p_mgrp == (osm_mgrp_t *) - cl_qmap_end(&p_qos_policy->p_subn->mgrp_mlid_tbl)) { + p_mgrp = p_qos_policy->p_subn->mgrp_mlid_tbl[cl_ntoh16(p_prtn->mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp) { OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, "ERR AC16: MCast group for partition with " "pkey 0x%04X not found\n", diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index adf4931..602a9fc 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -558,12 +559,11 @@ static void mcast_mgr_dump_one_port(cl_map_item_t * p_map_item, void *cxt) p_mcm_port->scope_state, p_mcm_port->proxy_join); } -static void sa_dump_one_mgrp(cl_map_item_t * p_map_item, void *cxt) +static void sa_dump_one_mgrp(osm_mgrp_t *p_mgrp, void *cxt) { struct opensm_dump_context dump_context; osm_opensm_t *p_osm = ((struct opensm_dump_context *)cxt)->p_osm; FILE *file = ((struct opensm_dump_context *)cxt)->file; - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; fprintf(file, "MC Group 0x%04x %s:" " mgid=0x%016" PRIx64 ":0x%016" PRIx64 @@ -700,13 +700,20 @@ static void sa_dump_one_service(cl_list_item_t * p_list_item, void *cxt) static void sa_dump_all_sa(osm_opensm_t * p_osm, FILE * file) { struct opensm_dump_context dump_context; + osm_mgrp_t *p_mgrp; + int i; dump_context.p_osm = p_osm; dump_context.file = file; OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump multicast:\n"); cl_plock_acquire(&p_osm->lock); - cl_qmap_apply_func(&p_osm->subn.mgrp_mlid_tbl, - sa_dump_one_mgrp, &dump_context); + for (i = 0; + i <= p_osm->subn.max_multicast_lid_ho - IB_LID_MCAST_START_HO; + i++) { + p_mgrp = p_osm->subn.mgrp_mlid_tbl[i]; + if (p_mgrp) + sa_dump_one_mgrp(p_mgrp, &dump_context); + } OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, "Dump inform:\n"); cl_qlist_apply_func(&p_osm->subn.sa_infr_list, sa_dump_one_inform, &dump_context); @@ -729,14 +736,12 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, unsigned well_known) { ib_net64_t comp_mask; - cl_map_item_t *p_next; osm_mgrp_t *p_mgrp; cl_plock_excl_acquire(&p_osm->lock); - if ((p_next = cl_qmap_get(&p_osm->subn.mgrp_mlid_tbl, mlid)) != - cl_qmap_end(&p_osm->subn.mgrp_mlid_tbl)) { - p_mgrp = (osm_mgrp_t *) p_next; + p_mgrp = p_osm->subn.mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (p_mgrp) { if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, sizeof(ib_gid_t))) { OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 46c87c7..1b97aa3 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -96,13 +97,7 @@ typedef struct osm_sa_mcmr_search_ctxt { static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, IN ib_net16_t const mlid) { - cl_map_item_t *map_item; - - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { - return NULL; - } - return (osm_mgrp_t *) map_item; + return(sa->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); } /********************************************************************* @@ -154,16 +149,20 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho - && cl_qmap_get(&p_subn->mgrp_mlid_tbl, - requested_mlid) == - cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { + && !p_subn->mgrp_mlid_tbl[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { mlid = cl_ntoh16(requested_mlid); goto Exit; } /* If MCGroups table is empty, first return the min mlid */ - p_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { + max_num_mlids = sa->p_subn->max_multicast_lid_ho - + IB_LID_MCAST_START_HO + 1; + for (idx = 0; idx < max_num_mlids; idx++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl[idx]; + if (p_mgrp) + break; + } + if (!p_mgrp) { mlid = IB_LID_MCAST_START_HO; OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, "No multicast groups found using minimal mlid:0x%04X\n", @@ -171,9 +170,6 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) goto Exit; } - max_num_mlids = - sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; - /* track all used mlids in the array (by mlid index) */ used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); if (!used_mlids_array) @@ -181,9 +177,10 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); /* scan all available multicast groups in the DB and fill in the table */ - while (p_mgrp != (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { + for (idx = 0; idx < max_num_mlids; idx++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl[idx]; /* ignore mgrps marked for deletion */ - if (p_mgrp->to_be_deleted == FALSE) { + if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 " : " "0x%016" PRIx64 "\n", @@ -202,11 +199,9 @@ __get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) cl_ntoh16(p_mgrp->mlid), max_num_mlids + IB_LID_MCAST_START_HO); } else { - used_mlids_array[cl_ntoh16(p_mgrp->mlid) - - IB_LID_MCAST_START_HO] = 1; + used_mlids_array[idx] = 1; } } - p_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); } /* Find "mlid holes" in the mgrp table */ @@ -247,8 +242,7 @@ __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) not a well known group */ if (p_mgrp && cl_is_qmap_empty(&p_mgrp->mcm_port_tbl) && p_mgrp->well_known == FALSE) { - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, - (cl_map_item_t *) p_mgrp); + sa->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; osm_mgrp_delete(p_mgrp); } } @@ -1033,21 +1027,17 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* since we might have an old group by that mlid one whose deletion was delayed for an idle time we need to deallocate it first */ - p_prev_mgrp = - (osm_mgrp_t *) cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); - if (p_prev_mgrp != - (osm_mgrp_t *) cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) { + p_prev_mgrp = sa->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (p_prev_mgrp) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Found previous group for mlid:0x%04x - " "Destroying it first\n", cl_ntoh16(mlid)); - cl_qmap_remove_item(&sa->p_subn->mgrp_mlid_tbl, - (cl_map_item_t *) p_prev_mgrp); + sa->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = NULL; osm_mgrp_delete(p_prev_mgrp); } - cl_qmap_insert(&sa->p_subn->mgrp_mlid_tbl, - mlid, &(*pp_mgrp)->map_item); + sa->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = *pp_mgrp; /* Send a Report to any InformInfo registered for Trap 66: MCGroup create */ @@ -1069,9 +1059,8 @@ typedef struct osm_sa_pr_mcmr_search_ctxt { /********************************************************************** *********************************************************************/ static void -__search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) +__search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) { - osm_mgrp_t *p_mgrp = (osm_mgrp_t *) p_map_item; osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = (osm_sa_pr_mcmr_search_ctxt_t *) context; const ib_gid_t *p_recvd_mgid; @@ -1135,23 +1124,30 @@ __search_mgrp_by_mgid(IN cl_map_item_t * const p_map_item, IN void *context) **********************************************************************/ ib_api_status_t osm_get_mgrp_by_mgid(IN osm_sa_t *sa, - IN ib_gid_t *p_mgid, - OUT osm_mgrp_t **pp_mgrp) + IN ib_gid_t *p_mgid, + OUT osm_mgrp_t **pp_mgrp) { osm_sa_pr_mcmr_search_ctxt_t mcmr_search_context; + osm_mgrp_t *p_mgrp; + int i; mcmr_search_context.p_mgid = p_mgid; mcmr_search_context.sa = sa; mcmr_search_context.p_mgrp = NULL; - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, - __search_mgrp_by_mgid, &mcmr_search_context); - - if (mcmr_search_context.p_mgrp == NULL) - return IB_NOT_FOUND; - - *pp_mgrp = mcmr_search_context.p_mgrp; - return IB_SUCCESS; + for (i = 0; + i <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + i++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl[i]; + if (p_mgrp) { + __search_mgrp_by_mgid(p_mgrp, &mcmr_search_context); + if (mcmr_search_context.p_mgrp) { + *pp_mgrp = mcmr_search_context.p_mgrp; + return IB_SUCCESS; + } + } + } + return IB_NOT_FOUND; } /********************************************************************** @@ -1619,10 +1615,9 @@ Exit: Match the given mgrp to the requested mcmr **********************************************************************/ static void -__osm_sa_mcm_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, +__osm_sa_mcm_by_comp_mask_cb(IN osm_mgrp_t * const p_mgrp, IN void *context) { - const osm_mgrp_t *const p_mgrp = (osm_mgrp_t *) p_map_item; osm_sa_mcmr_search_ctxt_t *const p_ctxt = (osm_sa_mcmr_search_ctxt_t *) context; osm_sa_t *sa = p_ctxt->sa; @@ -1811,6 +1806,8 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, ib_net64_t comp_mask; osm_physp_t *p_req_physp; boolean_t trusted_req; + osm_mgrp_t *p_mgrp; + int i; OSM_LOG_ENTER(sa->p_log); @@ -1846,8 +1843,13 @@ __osm_mcmr_query_mgrp(IN osm_sa_t * sa, CL_PLOCK_ACQUIRE(sa->p_lock); /* simply go over all MCGs and match */ - cl_qmap_apply_func(&sa->p_subn->mgrp_mlid_tbl, - __osm_sa_mcm_by_comp_mask_cb, &context); + for (i = 0; + i <= sa->p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + i++) { + p_mgrp = sa->p_subn->mgrp_mlid_tbl[i]; + if (p_mgrp) + __osm_sa_mcm_by_comp_mask_cb(p_mgrp, &context); + } CL_PLOCK_RELEASE(sa->p_lock); diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 2a9155c..ab90f75 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -1472,14 +1473,7 @@ __osm_pr_rcv_process_pair(IN osm_sa_t * sa, static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, IN ib_net16_t const mlid) { - cl_map_item_t *map_item; - - map_item = cl_qmap_get(&sa->p_subn->mgrp_mlid_tbl, mlid); - - if (map_item == cl_qmap_end(&sa->p_subn->mgrp_mlid_tbl)) - return NULL; - - return (osm_mgrp_t *) map_item; + return(sa->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); } /********************************************************************** diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index 7393ead..212a2ee 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -498,7 +499,6 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, { osm_mgrp_t *p_mgrp; osm_port_t *p_port; - cl_qmap_t *p_tbl; ib_api_status_t status = IB_SUCCESS; osm_mcm_info_t *p_mcm; @@ -525,9 +525,8 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, /* * If this multicast group does not already exist, create it. */ - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp) { OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, "Creating group, MLID 0x%X\n", cl_ntoh16(mlid)); @@ -540,7 +539,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, goto Exit; } - cl_qmap_insert(p_tbl, mlid, &p_mgrp->map_item); + p_sm->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO] = p_mgrp; } else { /* * The group already exists. If the port is not a @@ -603,7 +602,6 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, { osm_mgrp_t *p_mgrp; osm_port_t *p_port; - cl_qmap_t *p_tbl; ib_api_status_t status = IB_SUCCESS; OSM_LOG_ENTER(p_sm->p_log); @@ -630,9 +628,8 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, /* * Get the multicast group object for this group. */ - p_tbl = &p_sm->p_subn->mgrp_mlid_tbl; - p_mgrp = (osm_mgrp_t *) cl_qmap_get(p_tbl, mlid); - if (p_mgrp == (osm_mgrp_t *) cl_qmap_end(p_tbl)) { + p_mgrp = p_sm->p_subn->mgrp_mlid_tbl[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + if (!p_mgrp) { CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " "No multicast group for MLID 0x%X\n", cl_ntoh16(mlid)); diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 380db8e..79be2e7 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -85,19 +86,19 @@ void osm_subn_construct(IN osm_subn_t * const p_subn) cl_qlist_init(&p_subn->prefix_routes_list); cl_qmap_init(&p_subn->rtr_guid_tbl); cl_qmap_init(&p_subn->prtn_pkey_tbl); - cl_qmap_init(&p_subn->mgrp_mlid_tbl); } /********************************************************************** **********************************************************************/ void osm_subn_destroy(IN osm_subn_t * const p_subn) { + int i; osm_node_t *p_node, *p_next_node; osm_port_t *p_port, *p_next_port; osm_switch_t *p_sw, *p_next_sw; osm_remote_sm_t *p_rsm, *p_next_rsm; osm_prtn_t *p_prtn, *p_next_prtn; - osm_mgrp_t *p_mgrp, *p_next_mgrp; + osm_mgrp_t *p_mgrp; osm_infr_t *p_infr, *p_next_infr; /* it might be a good idea to de-allocate all known objects */ @@ -140,12 +141,14 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) osm_prtn_delete(&p_prtn); } - p_next_mgrp = (osm_mgrp_t *) cl_qmap_head(&p_subn->mgrp_mlid_tbl); - while (p_next_mgrp != - (osm_mgrp_t *) cl_qmap_end(&p_subn->mgrp_mlid_tbl)) { - p_mgrp = p_next_mgrp; - p_next_mgrp = (osm_mgrp_t *) cl_qmap_next(&p_mgrp->map_item); - osm_mgrp_delete(p_mgrp); + for (i = 0; + i <= p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO; + i++) { + p_mgrp = p_subn->mgrp_mlid_tbl[i]; + if (p_mgrp) { + osm_mgrp_delete(p_mgrp); + p_subn->mgrp_mlid_tbl[i] = NULL; + } } p_next_infr = (osm_infr_t *) cl_qlist_head(&p_subn->sa_infr_list); From weiny2 at llnl.gov Tue Jun 24 16:45:57 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 24 Jun 2008 16:45:57 -0700 Subject: [ofa-general] [PATCH 1/2 v2] libibmad: add a drpath2str function for easier printing of DR Paths. In-Reply-To: <1214346687.13056.352.camel@hrosenstock-ws.xsigo.com> References: <20080624144838.55ec7e3d.weiny2@llnl.gov> <1214346687.13056.352.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080624164557.398354bf.weiny2@llnl.gov> On Tue, 24 Jun 2008 15:31:27 -0700 Hal Rosenstock wrote: > On Tue, 2008-06-24 at 14:48 -0700, Ira Weiny wrote: > > --- a/libibmad/include/infiniband/mad.h > > +++ b/libibmad/include/infiniband/mad.h > > @@ -622,6 +622,7 @@ enum { > > char * portid2str(ib_portid_t *portid); > > int portid2portnum(ib_portid_t *portid); > > int str2drpath(ib_dr_path_t *path, char *routepath, int drslid, > > int drdlid); > > +char * drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size); > > Should drpath2str be added to libibmad.map too ? > > -- Hal Ooops... yes... since I was only using it internally right now I did not catch that. Thanks, Ira >From 3e192dabd78f81342106b24ee0407725f8cf78eb Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Tue, 24 Jun 2008 14:42:25 -0700 Subject: [PATCH] libibmad: add a drpath2str function for easier printing of DR Paths. Signed-off-by: Ira K. Weiny --- libibmad/include/infiniband/mad.h | 1 + libibmad/src/libibmad.map | 1 + libibmad/src/portid.c | 12 ++++++++++++ 3 files changed, 14 insertions(+), 0 deletions(-) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index 4f19a31..ae3c644 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -622,6 +622,7 @@ enum { char * portid2str(ib_portid_t *portid); int portid2portnum(ib_portid_t *portid); int str2drpath(ib_dr_path_t *path, char *routepath, int drslid, int drdlid); +char * drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size); static inline int ib_portid_set(ib_portid_t *portid, int lid, int qp, int qkey) diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index c1eb613..5a45a64 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -56,6 +56,7 @@ IBMAD_1.3 { portid2portnum; portid2str; str2drpath; + drpath2str; mad_agent_class; mad_class_agent; mad_register_client; diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c index 056b03d..7279e14 100644 --- a/libibmad/src/portid.c +++ b/libibmad/src/portid.c @@ -114,3 +114,15 @@ str2drpath(ib_dr_path_t *path, char *routepath, int drslid, int drdlid) return path->cnt; } + +char * +drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size) +{ + int i = 0; + int rc = snprintf(dstr, dstr_size, "slid %d; dlid %d; %d", + path->drslid, path->drdlid, path->p[0]); + for (i = 1; i <= path->cnt; i++) { + rc += snprintf(dstr+rc, dstr_size-rc, ",%d", path->p[i]); + } + return (dstr); +} -- 1.5.1 From xma at us.ibm.com Tue Jun 24 17:28:54 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 18:28:54 -0600 Subject: [ofa-general] [RFC][PATCH] last recv race patch In-Reply-To: Message-ID: Hello Roland, We have found several panics and memory leak for IPoIB-CM. I reviewed the code and caught some issues. I will start to submit a list of patches for review. i have tested the combined patch for connectX, it passed 24 hours 16 netperf/netserver bidirctional TCP, UDP stream test for 2.7.26-rc6 kernel. I turned on debug, I found that same QP context being destoryed twice for nonSRQ connection. I reviewed the code and found that there is a window the list could be added after the reap call, so checking the QP context status is needed. My linux email client is broken, so I attach each patch here. Address a possible race ------------------- Signed-off-by: Shirley Ma drivers/infiniband/ulp/ipoib/ipoib_cm.c | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 97e67d3..0886ee7 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -559,9 +559,12 @@ void ipoib_cm_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) else { if (!--p->recv_count) { spin_lock_irqsave(&priv->lock, flags); - list_move(&p->list, &priv->cm.rx_reap_list); - spin_unlock_irqrestore(&priv->lock, flags); - queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); + if (p->state == IPOIB_CM_RX_LIVE) { + list_move(&p->list, &priv->cm.rx_reap_list); + spin_unlock_irqrestore(&priv->lock, flags); + queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); + } else + spin_unlock_irqrestore(&priv->lock, flags); } return; } (See attached file: nonsrq_last_recv_race.patch) Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nonsrq_last_recv_race.patch Type: application/octet-stream Size: 1031 bytes Desc: not available URL: From makc at sgi.com Tue Jun 24 17:33:13 2008 From: makc at sgi.com (Max Matveev) Date: Wed, 25 Jun 2008 10:33:13 +1000 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080624144839.35b6d38f.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> Message-ID: <18529.37449.500476.509711@kuku.melbourne.sgi.com> On Tue, 24 Jun 2008 14:48:39 -0700, Ira Weiny wrote: IW> + static char gid[256]; IW> + static char drpath[256]; IW> + inet_ntop(AF_INET6, dport->gid, gid, 256); IW> + drpath2str(&(dport->drpath), drpath, 256); Couple of nitpicks: 1. you don't need static if all you do is print the strings. 2. longest IPv6 address is is known, so you can declare gid as char gid[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; 3. Use sizeof(str) in the arguments to inet_ntop() and drpath2str. IW> + IBWARN("_do_madrpc failed; dport (lid %d; grh \"%s\"; gid %s; drpath %s)", IW> + dport->lid, dport->grh_present ? "yes":"no", IW> + gid, drpath); IW> return 0; IW> + } IW> mad = umad_get_mad(rcvbuf); IW> @@ -223,8 +232,16 @@ mad_rpc_rmpp(void *port_id, ib_rpc_t *rpc, ib_portid_t *dport, IW> + static char gid[256]; IW> + static char drpath[256]; IW> + inet_ntop(AF_INET6, dport->gid, gid, 256); IW> + drpath2str(&(dport->drpath), drpath, 256); IW> + IBWARN("_do_madrpc failed; dport (lid %d; grh \"%s\"; gid %s; drpath %s)", IW> + dport->lid, dport->grh_present ? "yes":"no", IW> + gid, drpath); IW> return 0; IW> + } Same as above. In fact, consider factoring that bit out. max From xma at us.ibm.com Tue Jun 24 17:34:43 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 18:34:43 -0600 Subject: [ofa-general] [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Hello Roland, This patch addresses a possible race if more than two last wqe reached events have been received. The first reap will reap all list in the drain list, then if there is any other cqe arrives after the first drain WR cqe, then it will crash. Signed-off-by: Shirley Ma --------------------- drivers/infiniband/ulp/ipoib/ipoib.h | 1 - drivers/infiniband/ulp/ipoib/ipoib_cm.c | 38 +++++++++++++++--------------- 2 files changed, 19 insertions(+), 20 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index ca126fc..fc6c811 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -226,7 +226,6 @@ struct ipoib_cm_dev_priv { struct list_head passive_ids; /* state: LIVE */ struct list_head rx_error_list; /* state: ERROR */ struct list_head rx_flush_list; /* state: FLUSH, drain not started */ - struct list_head rx_drain_list; /* state: FLUSH, drain started */ struct list_head rx_reap_list; /* state: FLUSH, drain done */ struct work_struct start_task; struct work_struct reap_task; diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 0886ee7..ae67379 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -210,10 +210,7 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) struct ib_send_wr *bad_wr; struct ipoib_cm_rx *p; - /* We only reserved 1 extra slot in CQ for drain WRs, so - * make sure we have at most 1 outstanding WR. */ - if (list_empty(&priv->cm.rx_flush_list) || - !list_empty(&priv->cm.rx_drain_list)) + if (list_empty(&priv->cm.rx_flush_list)) return; /* @@ -221,10 +218,11 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) * error" WC will be immediately generated for each WR we post. */ p = list_entry(priv->cm.rx_flush_list.next, typeof(*p), list); + /* We only reserved 1 extra slot in CQ for drain WRs, so + * make sure we have at most 1 outstanding WR. */ if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); - list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_drain_list); } static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) @@ -237,9 +235,11 @@ static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) return; spin_lock_irqsave(&priv->lock, flags); - list_move(&p->list, &priv->cm.rx_flush_list); - p->state = IPOIB_CM_RX_FLUSH; - ipoib_cm_start_rx_drain(priv); + if (p->state == IPOIB_CM_RX_LIVE) { + list_move(&p->list, &priv->cm.rx_flush_list); + p->state = IPOIB_CM_RX_FLUSH; + ipoib_cm_start_rx_drain(priv); + } spin_unlock_irqrestore(&priv->lock, flags); } @@ -529,21 +529,25 @@ void ipoib_cm_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) ipoib_dbg_data(priv, "cm recv completion: id %d, status: %d\n", wr_id, wc->status); + p = wc->qp->qp_context; + if (unlikely(wr_id >= ipoib_recvq_size)) { if (wr_id == (IPOIB_CM_RX_DRAIN_WRID & ~(IPOIB_OP_CM | IPOIB_OP_RECV))) { spin_lock_irqsave(&priv->lock, flags); - list_splice_init(&priv->cm.rx_drain_list, &priv->cm.rx_reap_list); - ipoib_cm_start_rx_drain(priv); - queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); - spin_unlock_irqrestore(&priv->lock, flags); + if (p->state == IPOIB_CM_RX_FLUSH) { + list_move(&p->list, &priv->cm.rx_reap_list); + p->state == IPOIB_CM_RX_ERROR; + ipoib_cm_start_rx_drain(priv); + spin_unlock_irqrestore(&priv->lock, flags); + queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); + } else + spin_unlock_irqrestore(&priv->lock, flags); } else ipoib_warn(priv, "cm recv completion event with wrid %d (> %d)\n", wr_id, ipoib_recvq_size); return; } - p = wc->qp->qp_context; - has_srq = ipoib_cm_has_srq(dev); rx_ring = has_srq ? priv->cm.srq_ring : p->rx_ring; @@ -853,8 +857,7 @@ void ipoib_cm_dev_stop(struct net_device *dev) begin = jiffies; while (!list_empty(&priv->cm.rx_error_list) || - !list_empty(&priv->cm.rx_flush_list) || - !list_empty(&priv->cm.rx_drain_list)) { + !list_empty(&priv->cm.rx_flush_list)) { if (time_after(jiffies, begin + 5 * HZ)) { ipoib_warn(priv, "RX drain timing out\n"); @@ -865,8 +868,6 @@ void ipoib_cm_dev_stop(struct net_device *dev) &priv->cm.rx_reap_list); list_splice_init(&priv->cm.rx_error_list, &priv->cm.rx_reap_list); - list_splice_init(&priv->cm.rx_drain_list, - &priv->cm.rx_reap_list); break; } spin_unlock_irq(&priv->lock); @@ -1458,7 +1459,6 @@ int ipoib_cm_dev_init(struct net_device *dev) INIT_LIST_HEAD(&priv->cm.start_list); INIT_LIST_HEAD(&priv->cm.rx_error_list); INIT_LIST_HEAD(&priv->cm.rx_flush_list); - INIT_LIST_HEAD(&priv->cm.rx_drain_list); INIT_LIST_HEAD(&priv->cm.rx_reap_list); INIT_WORK(&priv->cm.start_task, ipoib_cm_tx_start); INIT_WORK(&priv->cm.reap_task, ipoib_cm_tx_reap); (See attached file: last_wqe_race1.patch) Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: last_wqe_race1.patch Type: application/octet-stream Size: 5055 bytes Desc: not available URL: From xma at us.ibm.com Tue Jun 24 17:39:06 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 18:39:06 -0600 Subject: [ofa-general] [RFC][PATCH] release IPoIB-cm stale connections resouce In-Reply-To: Message-ID: Hello Roland, This patch releases all staled connections, otherwise all statled connections will remain untill the nodes down or IPoIB module being removed. It causes waste lots of resource in the long run. Signed-off-by: Shirley Ma --------------------- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index ae67379..442dd49 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -61,6 +61,7 @@ MODULE_PARM_DESC(cm_data_debug_level, #define IPOIB_CM_RX_UPDATE_TIME (256 * HZ) #define IPOIB_CM_RX_TIMEOUT (2 * 256 * HZ) +#define IPOIB_CM_RX_DRAIN (60 * HZ) #define IPOIB_CM_RX_DELAY (3 * 256 * HZ) #define IPOIB_CM_RX_UPDATE_MASK (0x3) @@ -1343,6 +1344,15 @@ static void ipoib_cm_stale_task(struct work_struct *work) int ret; spin_lock_irq(&priv->lock); + /* wait for the CQ drain, and clean up stale connections */ + while (!list_empty(&priv->cm.rx_error_list)) { + /* List is sorted by LRU, start from tail, + * stop when we see a recently used entry */ + p = list_entry(priv->cm.rx_error_list.prev, typeof(*p), list); + if (time_before_eq(jiffies, p->jiffies + IPOIB_CM_RX_DRAIN)) + break; + list_move(&p->list, &priv->cm.rx_reap_list); + } while (!list_empty(&priv->cm.passive_ids)) { /* List is sorted by LRU, start from tail, * stop when we see a recently used entry */ (See attached file: stale_connection_cleanup.patch) Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: stale_connection_cleanup.patch Type: application/octet-stream Size: 1528 bytes Desc: not available URL: From xma at us.ibm.com Tue Jun 24 17:51:42 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 18:51:42 -0600 Subject: [ofa-general] [RFC][PATCH] IPoIB-cm: release connection if post_send drain WR failure In-Reply-To: Message-ID: Hello Roland, In case post_send drain WR failure, release connections in stale task. Signed-off-by: Shirley Ma -------------------------- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 442dd49..06e5933 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -1344,6 +1344,17 @@ static void ipoib_cm_stale_task(struct work_struct *work) int ret; spin_lock_irq(&priv->lock); + /* in case post drain WR failure, clean up connection here */ + while (!list_empty(&priv->cm.rx_flush_list)) { + /* List is sorted by LRU, start from tail, + * stop when we see a recently used entry */ + p = list_entry(priv->cm.rx_flush_list.prev, typeof(*p), list); + p->state = IPOIB_CM_RX_ERROR; + if (time_before_eq(jiffies, p->jiffies + IPOIB_CM_RX_DRAIN)) + break; + list_move(&p->list, &priv->cm.rx_reap_list); + + } /* wait for the CQ drain, and clean up stale connections */ while (!list_empty(&priv->cm.rx_error_list)) { /* List is sorted by LRU, start from tail, (See attached file: post_drain_wr_failure.patch) Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: post_drain_wr_failure.patch Type: application/octet-stream Size: 1190 bytes Desc: not available URL: From xma at us.ibm.com Tue Jun 24 17:57:56 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 18:57:56 -0600 Subject: [ofa-general] [RFC][PATCH] release IPoIB-cm stale connections resouce In-Reply-To: Message-ID: Hello Roland, I am sorry I missed one line during splitting so regenrate this patch. This patch releases all staled connections, otherwise all statled connections will remain untill the nodes down or IPoIB module being removed. It causes waste lots of resource in the long run. Signed-off-by: Shirley Ma --------------------- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index ae67379..9bc12ea 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -61,6 +61,7 @@ MODULE_PARM_DESC(cm_data_debug_level, #define IPOIB_CM_RX_UPDATE_TIME (256 * HZ) #define IPOIB_CM_RX_TIMEOUT (2 * 256 * HZ) +#define IPOIB_CM_RX_DRAIN (60 * HZ) #define IPOIB_CM_RX_DELAY (3 * 256 * HZ) #define IPOIB_CM_RX_UPDATE_MASK (0x3) @@ -1343,6 +1344,15 @@ static void ipoib_cm_stale_task(struct work_struct *work) int ret; spin_lock_irq(&priv->lock); + /* wait for the CQ drain, and clean up stale connections */ + while (!list_empty(&priv->cm.rx_error_list)) { + /* List is sorted by LRU, start from tail, + * stop when we see a recently used entry */ + p = list_entry(priv->cm.rx_error_list.prev, typeof(*p), list); + if (time_before_eq(jiffies, p->jiffies + IPOIB_CM_RX_DRAIN)) + break; + list_move(&p->list, &priv->cm.rx_reap_list); + } while (!list_empty(&priv->cm.passive_ids)) { /* List is sorted by LRU, start from tail, * stop when we see a recently used entry */ @@ -1351,6 +1361,7 @@ static void ipoib_cm_stale_task(struct work_struct *work) break; list_move(&p->list, &priv->cm.rx_error_list); p->state = IPOIB_CM_RX_ERROR; + p->jiffies = jiffies; spin_unlock_irq(&priv->lock); ret = ib_modify_qp(p->qp, &ipoib_cm_err_attr, IB_QP_STATE); if (ret) (See attached file: stale_connection_cleanup.patch) Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: stale_connection_cleanup.patch Type: application/octet-stream Size: 1835 bytes Desc: not available URL: From sashak at voltaire.com Tue Jun 24 18:36:45 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 04:36:45 +0300 Subject: [ofa-general] Re: [PATCHv2] opensm: Convert mgrp_mlid_tbl into array In-Reply-To: <1214350128.13056.367.camel@hrosenstock-ws.xsigo.com> References: <1214350128.13056.367.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080625013645.GF7341@sashak.voltaire.com> On 16:28 Tue 24 Jun , Hal Rosenstock wrote: > opensm: Convert mgrp_mlid_tbl into array > > Changes from v1: > Remove wrapper structure for array of void pointers > In osm_drop_mgr.c and osm_mcast_mgr.c, use mlid rather than walking array > Eliminate unneeded initialization in osm_subnet.c:osm_subn_construct > > Additional patches to follow: > 1. Consolidate get mgrp by mlid code into osm_get_mgrp_by_mlid function > 2. Eliminate no longer needed used_mlids_array in __get_new_mlid Actually did both. Will post patches. Also changes mgrp_mlid_tbl[] to just mgroups[] since most fields named *_tbl are qmap tables. > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Jun 24 18:37:55 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 04:37:55 +0300 Subject: [ofa-general] [PATCH] opensm/osm_sa_mcmember_record: cleanup __get_new_mlid() function In-Reply-To: <20080625013645.GF7341@sashak.voltaire.com> References: <1214350128.13056.367.camel@hrosenstock-ws.xsigo.com> <20080625013645.GF7341@sashak.voltaire.com> Message-ID: <20080625013755.GG7341@sashak.voltaire.com> Remove unneded flow, buffer allocation, speedup and simplify. Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_sa_mcmember_record.c | 98 ++++---------------------------- 1 files changed, 12 insertions(+), 86 deletions(-) diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 6647a2f..231f482 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -132,98 +132,25 @@ static void __free_mlid(IN osm_sa_t * sa, IN uint16_t mlid) /********************************************************************* Get a new unused mlid by scanning all the used ones in the subnet. - TODO: Implement a more scalable - O(1) solution based on pool of - available mlids. **********************************************************************/ -static ib_net16_t -__get_new_mlid(IN osm_sa_t * sa, IN ib_net16_t requested_mlid) +static ib_net16_t __get_new_mlid(osm_sa_t *sa, ib_net16_t requested_mlid) { osm_subn_t *p_subn = sa->p_subn; - osm_mgrp_t *p_mgrp; - uint8_t *used_mlids_array; - uint16_t idx; - uint16_t mlid; /* the result */ - uint16_t max_num_mlids; - - OSM_LOG_ENTER(sa->p_log); + unsigned i, max; if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho - && !p_subn->mgroups[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) { - mlid = cl_ntoh16(requested_mlid); - goto Exit; + && !p_subn->mgroups[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) + return requested_mlid; + + max = p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; + for (i = 0; i < max; i++) { + osm_mgrp_t *p_mgrp = sa->p_subn->mgroups[i]; + if (!p_mgrp || p_mgrp->to_be_deleted) + return cl_hton16(i + IB_LID_MCAST_START_HO); } - /* If MCGroups table is empty, first return the min mlid */ - max_num_mlids = sa->p_subn->max_multicast_lid_ho - - IB_LID_MCAST_START_HO + 1; - for (idx = 0; idx < max_num_mlids; idx++) { - p_mgrp = sa->p_subn->mgroups[idx]; - if (p_mgrp) - break; - } - if (!p_mgrp) { - mlid = IB_LID_MCAST_START_HO; - OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No multicast groups found using minimal mlid:0x%04X\n", - mlid); - goto Exit; - } - - /* track all used mlids in the array (by mlid index) */ - used_mlids_array = (uint8_t *) malloc(sizeof(uint8_t) * max_num_mlids); - if (!used_mlids_array) - return 0; - memset(used_mlids_array, 0, sizeof(uint8_t) * max_num_mlids); - - /* scan all available multicast groups in the DB and fill in the table */ - for (idx = 0; idx < max_num_mlids; idx++) { - p_mgrp = sa->p_subn->mgroups[idx]; - /* ignore mgrps marked for deletion */ - if (p_mgrp && p_mgrp->to_be_deleted == FALSE) { - OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Found mgrp with lid:0x%X MGID: 0x%016" PRIx64 - " : " "0x%016" PRIx64 "\n", - cl_ntoh16(p_mgrp->mlid), - cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast. - prefix), - cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast. - interface_id)); - - /* Map in table */ - if (cl_ntoh16(p_mgrp->mlid) > - sa->p_subn->max_multicast_lid_ho) { - OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B27: " - "Found mgrp with mlid:0x%04X > " - "max allowed mlid:0x%04X\n", - cl_ntoh16(p_mgrp->mlid), - max_num_mlids + IB_LID_MCAST_START_HO); - } else { - used_mlids_array[idx] = 1; - } - } - } - - /* Find "mlid holes" in the mgrp table */ - for (idx = 0; - (idx < max_num_mlids) && (used_mlids_array[idx] == 1); idx++) ; - - /* did it go above the maximal mlid allowed */ - if (idx < max_num_mlids) { - mlid = idx + IB_LID_MCAST_START_HO; - OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "Found available mlid:0x%04X at idx:%u\n", mlid, idx); - } else { - OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B23: " - "All available:%u mlids are taken\n", max_num_mlids); - mlid = 0; - } - - free(used_mlids_array); - -Exit: - OSM_LOG_EXIT(sa->p_log); - return cl_hton16(mlid); + return 0; } /********************************************************************* @@ -233,8 +160,7 @@ Exit: we silently drop it. Since it was an intermediate group no need to re-route it. **********************************************************************/ -static void -__cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) +static void __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) { osm_mgrp_t *p_mgrp = __get_mgrp_by_mlid(sa, mlid); -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Tue Jun 24 18:39:11 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 04:39:11 +0300 Subject: [ofa-general] [PATCH] opensm: merge all __get_mgrp_by_mlid() instances In-Reply-To: <20080625013755.GG7341@sashak.voltaire.com> References: <1214350128.13056.367.camel@hrosenstock-ws.xsigo.com> <20080625013645.GF7341@sashak.voltaire.com> <20080625013755.GG7341@sashak.voltaire.com> Message-ID: <20080625013911.GH7341@sashak.voltaire.com> Merge all __get_mgrp_by_mlid() instances into single osm_get_mgrp_by_mlid() function. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_subnet.h | 29 +++++++++++++++++++++++++++++ opensm/opensm/osm_drop_mgr.c | 2 +- opensm/opensm/osm_mcast_mgr.c | 10 +--------- opensm/opensm/osm_qos_policy.c | 2 +- opensm/opensm/osm_sa.c | 2 +- opensm/opensm/osm_sa_mcmember_record.c | 17 ++++------------- opensm/opensm/osm_sa_path_record.c | 10 +--------- opensm/opensm/osm_sm.c | 4 ++-- 8 files changed, 40 insertions(+), 36 deletions(-) diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 6924b40..b4f130e 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -721,6 +721,7 @@ struct osm_log; struct osm_switch; struct osm_physp; struct osm_port; +struct osm_mgrp; /****f* OpenSM: Helper/osm_get_gid_by_mad_addr * NAME @@ -917,6 +918,34 @@ struct osm_port *osm_get_port_by_guid(IN osm_subn_t const *p_subn, * osm_port_t *********/ +/****f* OpenSM: Subnet/osm_get_mgrp_by_mlid +* NAME +* osm_get_mgrp_by_mlid +* +* DESCRIPTION +* The looks for the given multicast group in the subnet table by mlid. +* NOTE: this code is not thread safe. Need to grab the lock before +* calling it. +* +* SYNOPSIS +*/ +static inline +struct osm_mgrp *osm_get_mgrp_by_mlid(osm_subn_t const *p_subn, ib_net16_t mlid) +{ + return p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; +} +/* +* PARAMETERS +* p_subn +* [in] Pointer to an osm_subn_t object +* +* mlid +* [in] The multicast group mlid in network order +* +* RETURN VALUES +* The multicast group structure pointer if found. NULL otherwise. +*********/ + /****f* OpenSM: Helper/osm_get_physp_by_mad_addr * NAME * osm_get_physp_by_mad_addr diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index 5ebb2d5..2d8d6e9 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -207,7 +207,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) p_mcm = (osm_mcm_info_t *) cl_qlist_remove_head(&p_port->mcm_list); while (p_mcm != (osm_mcm_info_t *) cl_qlist_end(&p_port->mcm_list)) { - p_mgrp = sm->p_subn->mgroups[cl_ntoh16(p_mcm->mlid) - IB_LID_MCAST_START_HO]; + p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, p_mcm->mlid); if (p_mgrp) { osm_mgrp_remove_port(sm->p_subn, sm->p_log, p_mgrp, p_port->guid); diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index 111e411..ac45cf3 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -1229,14 +1229,6 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) } /********************************************************************** - **********************************************************************/ -static -osm_mgrp_t *__get_mgrp_by_mlid(osm_sm_t * sm, IN ib_net16_t const mlid) -{ - return(sm->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); -} - -/********************************************************************** This is the function that is invoked during idle time to handle the process request for mcast groups where join/leave/delete was required. **********************************************************************/ @@ -1270,7 +1262,7 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) /* since we delayed the execution we prefer to pass the mlid as the mgrp identifier and then find it or abort */ - p_mgrp = __get_mgrp_by_mlid(sm, mlid); + p_mgrp = osm_get_mgrp_by_mlid(sm->p_subn, mlid); if (!p_mgrp) continue; diff --git a/opensm/opensm/osm_qos_policy.c b/opensm/opensm/osm_qos_policy.c index 223aba6..ecc38b3 100644 --- a/opensm/opensm/osm_qos_policy.c +++ b/opensm/opensm/osm_qos_policy.c @@ -793,7 +793,7 @@ static void __qos_policy_validate_pkey( if (!p_prtn->mlid) return; - p_mgrp = p_qos_policy->p_subn->mgroups[cl_ntoh16(p_prtn->mlid) - IB_LID_MCAST_START_HO]; + p_mgrp = osm_get_mgrp_by_mlid(p_qos_policy->p_subn, p_prtn->mlid); if (!p_mgrp) { OSM_LOG(&p_qos_policy->p_subn->p_osm->log, OSM_LOG_ERROR, "ERR AC16: MCast group for partition with " diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index 45e0d22..355e1c2 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -740,7 +740,7 @@ static osm_mgrp_t *load_mcgroup(osm_opensm_t * p_osm, ib_net16_t mlid, cl_plock_excl_acquire(&p_osm->lock); - p_mgrp = p_osm->subn.mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + p_mgrp = osm_get_mgrp_by_mlid(&p_osm->subn, mlid); if (p_mgrp) { if (!memcmp(&p_mgrp->mcmember_rec.mgid, &p_mcm_rec->mgid, sizeof(ib_gid_t))) { diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 231f482..3b41d03 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -91,15 +91,6 @@ typedef struct osm_sa_mcmr_search_ctxt { boolean_t trusted_req; } osm_sa_mcmr_search_ctxt_t; -/********************************************************************** - Look for a MGRP in the mgroups by mlid -**********************************************************************/ -static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, - IN ib_net16_t const mlid) -{ - return(sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); -} - /********************************************************************* Copy certain fields between two mcmember records used during the process of join request to copy data from the mgrp @@ -140,7 +131,7 @@ static ib_net16_t __get_new_mlid(osm_sa_t *sa, ib_net16_t requested_mlid) if (requested_mlid && cl_ntoh16(requested_mlid) >= IB_LID_MCAST_START_HO && cl_ntoh16(requested_mlid) <= p_subn->max_multicast_lid_ho - && !p_subn->mgroups[cl_ntoh16(requested_mlid) - IB_LID_MCAST_START_HO]) + && !osm_get_mgrp_by_mlid(p_subn, requested_mlid)) return requested_mlid; max = p_subn->max_multicast_lid_ho - IB_LID_MCAST_START_HO + 1; @@ -162,7 +153,7 @@ static ib_net16_t __get_new_mlid(osm_sa_t *sa, ib_net16_t requested_mlid) **********************************************************************/ static void __cleanup_mgrp(IN osm_sa_t * sa, IN ib_net16_t const mlid) { - osm_mgrp_t *p_mgrp = __get_mgrp_by_mlid(sa, mlid); + osm_mgrp_t *p_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); /* Remove MGRP only if osm_mcm_port_t count is 0 and not a well known group */ @@ -953,7 +944,7 @@ osm_mcmr_rcv_create_new_mgrp(IN osm_sa_t * sa, /* since we might have an old group by that mlid one whose deletion was delayed for an idle time we need to deallocate it first */ - p_prev_mgrp = sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + p_prev_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); if (p_prev_mgrp) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Found previous group for mlid:0x%04x - " @@ -1480,7 +1471,7 @@ __osm_mcmr_rcv_join_mgrp(IN osm_sa_t * sa, CL_PLOCK_EXCL_ACQUIRE(sa->p_lock); /* the request for routing failed so we need to remove the port */ - p_mgrp = __get_mgrp_by_mlid(sa, mlid); + p_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, mlid); if (p_mgrp != NULL) { osm_mgrp_remove_port(sa->p_subn, sa->p_log, diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index a38fb60..9af8a4f 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -1470,14 +1470,6 @@ __osm_pr_rcv_process_pair(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ -static osm_mgrp_t *__get_mgrp_by_mlid(IN osm_sa_t * sa, - IN ib_net16_t const mlid) -{ - return(sa->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]); -} - -/********************************************************************** - **********************************************************************/ static void __osm_pr_get_mgrp(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, OUT osm_mgrp_t ** pp_mgrp) @@ -1520,7 +1512,7 @@ __osm_pr_get_mgrp(IN osm_sa_t * sa, goto Exit; } } else { - *pp_mgrp = __get_mgrp_by_mlid(sa, p_pr->dlid); + *pp_mgrp = osm_get_mgrp_by_mlid(sa->p_subn, p_pr->dlid); if (*pp_mgrp == NULL) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F11: " "No MC group found for PathRecord " diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index e677fa0..3548935 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -525,7 +525,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, /* * If this multicast group does not already exist, create it. */ - p_mgrp = p_sm->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); if (!p_mgrp) { OSM_LOG(p_sm->p_log, OSM_LOG_VERBOSE, "Creating group, MLID 0x%X\n", cl_ntoh16(mlid)); @@ -628,7 +628,7 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, /* * Get the multicast group object for this group. */ - p_mgrp = p_sm->p_subn->mgroups[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + p_mgrp = osm_get_mgrp_by_mlid(p_sm->p_subn, mlid); if (!p_mgrp) { CL_PLOCK_RELEASE(p_sm->p_lock); OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E08: " -- 1.5.5.1.178.g1f811 From rdreier at cisco.com Tue Jun 24 18:42:00 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 18:42:00 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last recv race patch In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 18:28:54 -0600") References: Message-ID: This all looks like good work, especially if it's fixing crasher problems. However, it would really make it a lot easier to review if the changelog had more detail: > I turned on debug, I found that same QP context being destoryed twice > for nonSRQ connection. I reviewed the code and found that there is a window > the list could be added after the reap call, so checking the QP context > status is needed. > Address a possible race in other words if your could diagram the race in the standard way, eg Thread 1 Thread 2 foo(); list_add(); do_something(); blah(); check_it(); oops_because_thread_2_messed_us_up(); then it avoids me having to reverse engineer the debugging work you did and makes it much easier to see why this patch makes sense. Basically the patch description should explain the problem with the current code in enough detail that it is easy to understand how and when things go wrong, and explain the fix enough so that it is easy to understand why you are changing things as you do. I'm guessing in this patch it's a race with the stale task moving the connection to the error state exactly when the last receive completes? Anyway this applies to all the patches, thanks. - R. From weiny2 at llnl.gov Tue Jun 24 19:22:51 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 24 Jun 2008 19:22:51 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <18529.37449.500476.509711@kuku.melbourne.sgi.com> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> Message-ID: <20080624192251.38214cf1.weiny2@llnl.gov> On Wed, 25 Jun 2008 10:33:13 +1000 Max Matveev wrote: > On Tue, 24 Jun 2008 14:48:39 -0700, Ira Weiny wrote: > > IW> + static char gid[256]; > IW> + static char drpath[256]; > IW> + inet_ntop(AF_INET6, dport->gid, gid, 256); > IW> + drpath2str(&(dport->drpath), drpath, 256); > Couple of nitpicks: > > 1. you don't need static if all you do is print the strings. > 2. longest IPv6 address is is known, so you can declare gid as > char gid[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; > 3. Use sizeof(str) in the arguments to inet_ntop() and drpath2str. Good points, v2 is attached. > > IW> + IBWARN("_do_madrpc failed; dport (lid %d; grh \"%s\"; gid %s; drpath %s)", > IW> + dport->lid, dport->grh_present ? "yes":"no", > IW> + gid, drpath); > IW> return 0; > IW> + } > > IW> mad = umad_get_mad(rcvbuf); > > IW> @@ -223,8 +232,16 @@ mad_rpc_rmpp(void *port_id, ib_rpc_t *rpc, ib_portid_t *dport, > > IW> + static char gid[256]; > IW> + static char drpath[256]; > IW> + inet_ntop(AF_INET6, dport->gid, gid, 256); > IW> + drpath2str(&(dport->drpath), drpath, 256); > IW> + IBWARN("_do_madrpc failed; dport (lid %d; grh \"%s\"; gid %s; drpath %s)", > IW> + dport->lid, dport->grh_present ? "yes":"no", > IW> + gid, drpath); > IW> return 0; > IW> + } > Same as above. In fact, consider factoring that bit out. Ah, yes... Actually... there is already a portid2str functions... :-( I will have to leave this till tomorrow to make sure I don't mess anything else up tonight. Sorry Sasha, Ira From rdreier at cisco.com Tue Jun 24 20:11:16 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 20:11:16 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 18:34:43 -0600") References: Message-ID: I don't understand the problem this patch is solving (insufficient detail in the changelog), but I don't see how it can be correct: > struct list_head rx_flush_list; /* state: FLUSH, drain not started */ > - struct list_head rx_drain_list; /* state: FLUSH, drain started */ > struct list_head rx_reap_list; /* state: FLUSH, drain done */ It seems to me we need all three of these states to keep track of QPs properly: first one means "last WQE reached", second one means "send posted after last WQE reached" and last on means "completion seen for send posted after last WQE reached". - R. From rdreier at cisco.com Tue Jun 24 20:11:56 2008 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Jun 2008 20:11:56 -0700 Subject: [ofa-general] [RFC][PATCH] IPoIB-cm: release connection if post_send drain WR failure In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 18:51:42 -0600") References: Message-ID: > In case post_send drain WR failure, release connections in stale task. What is "post_send drain WR failure"? When can it happen? - R. From jackm at dev.mellanox.co.il Tue Jun 24 22:33:00 2008 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Wed, 25 Jun 2008 08:33:00 +0300 Subject: [ofa-general] Re: [PATCH v2] Add enum strings and *_str functions for enums In-Reply-To: References: <20080616102430.505a2000.weiny2@llnl.gov> Message-ID: <200806250833.01371.jackm@dev.mellanox.co.il> On Wednesday 25 June 2008 00:54, Roland Dreier wrote: > How about if we just revert the devinfo.c change that used the new > conversion function? I agree -- that seems best. New apps can use the new string function, and we'll preserve ibv_devinfo. - Jack From xma at us.ibm.com Tue Jun 24 22:53:31 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 22:53:31 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/24/2008 08:11:16 PM: > I don't understand the problem this patch is solving (insufficient > detail in the changelog), but I don't see how it can be correct: > > > struct list_head rx_flush_list; /* state: FLUSH, drain notstarted */ > > - struct list_head rx_drain_list; /* state: FLUSH, drain started */ > > struct list_head rx_reap_list; /* state: FLUSH, drain done */ > > It seems to me we need all three of these states to keep track of QPs > properly: first one means "last WQE reached", second one means "send > posted after last WQE reached" and last on means "completion seen for > send posted after last WQE reached". > > - R. The issue is if rx_drain_list has only one QP context on that list, then it's not necessary to have a rx_drain_list; if rx_drain_list has mulitple QP contexts on that list, then move all elements from that list to reap list from a particular QP drain WR cqe is wrong. if (unlikely(wr_id >= ipoib_recvq_size)) { if (wr_id == (IPOIB_CM_RX_DRAIN_WRID & ~(IPOIB_OP_CM | IPOIB_OP_RECV))) { spin_lock_irqsave(&priv->lock, flags); list_splice_init(&priv->cm.rx_drain_list, &priv->cm.rx_reap_list); ipoib_cm_start_rx_drain(priv); queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); spin_unlock_irqrestore(&priv->lock, flags); Also there is a possible race between timeout reap and last wqe reached reap, so check the status of that QP context is necessary. Hopefully it's clear. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Tue Jun 24 23:16:16 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 23:16:16 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Hello Roland, Let's assume this senarios: 1. QP100 last WQE reached event, QP100 context is added into flush_list, and then it is put into drain_list, and does post_send of a drain WR. 2. QP200 last WQE reached event, QP200 context is added into flush_list, but not drain_list since only one drain WR will be posted 3. QP300 ...., QP300 context is added into flush_list, but not drain_list So QP100 is on drain_list, QP200, QP300 are on flush_list In rcq poll_cq, 1. QP 100 drain WR cqe is polled, it will put QP100 into reap_list then call ipoib_cm_start_rx_drain(), post_send of QP200 drain WR, and QP200, QP300 are both moved from flush_list to drain_list 2. QP 200 drain WR cqe is polled, it will move both QP200 and QP300 from drain_list to reap_list 3. QP300 cqe comes, but QP300 context has been freed, ---> panic. Does that make sense? Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Tue Jun 24 23:22:06 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 23:22:06 -0700 Subject: [ofa-general] [RFC][PATCH] IPoIB-cm: release connection if post_send drain WR failure In-Reply-To: Message-ID: Hello Roland, Roland Dreier wrote on 06/24/2008 08:11:56 PM: > > In case post_send drain WR failure, release connections in stale task. > What is "post_send drain WR failure"? When can it happen? > - R. I saw it happened from below ipoib debug output on testing ehca2: static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) { .... /* * QPs on flush list are error state. This way, a "flush * error" WC will be immediately generated for each WR we post. */ p = list_entry(priv->cm.rx_flush_list.next, typeof(*p), list); if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); ... } Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Tue Jun 24 23:29:21 2008 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 24 Jun 2008 23:29:21 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last recv race patch In-Reply-To: Message-ID: > Anyway this applies to all the patches, thanks. > - R. Thanks Roland, will improve it next time. :) Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From kliteyn at dev.mellanox.co.il Tue Jun 24 23:46:31 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 25 Jun 2008 09:46:31 +0300 Subject: [ofa-general] [PATCH] opensm/osm_vendor_select.h: undoing change in default vendor selection Message-ID: <4861E9C7.1010802@dev.mellanox.co.il> Hi Sasha, Undoing a change in the default vendor selection. The patch broke ibmgtsim compilation. Signed-off-by: Yevgeny Kliteynik --- opensm/include/vendor/osm_vendor_select.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/include/vendor/osm_vendor_select.h b/opensm/include/vendor/osm_vendor_select.h index a7d12dc..8dcc05b 100644 --- a/opensm/include/vendor/osm_vendor_select.h +++ b/opensm/include/vendor/osm_vendor_select.h @@ -58,7 +58,7 @@ #ifndef OSM_VENDOR_INTF_TS #ifndef OSM_VENDOR_INTF_SIM #ifndef OSM_VENDOR_INTF_AL -#define OSM_VENDOR_INTF_OPENIB +#define OSM_VENDOR_INTF_AL #endif /* AL */ #endif /* TS */ #endif /* SIM */ -- 1.5.1.4 From Sumit.Gaur at Sun.COM Tue Jun 24 23:41:36 2008 From: Sumit.Gaur at Sun.COM (Sumit Gaur - Sun Microsystem) Date: Wed, 25 Jun 2008 12:11:36 +0530 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> Message-ID: <4861E8A0.7000704@Sun.COM> Hi Hal/Sashak, Hal Rosenstock wrote: > Hi Sumit, > > On Tue, 2008-06-24 at 14:21 +0530, Sumit Gaur - Sun Microsystem wrote: > >>Hi, >>I am using OFED 2.5.* > > ^^^^^ > 1.2.5.* ? > Sorry for typo .. it is 1.2.5.* > >> and observing that my SMI requests are serving very fast >>and response time is very less on the contrary my GSI requests were served in >>more time and response time sometime goes more than 2 sec. Any light on this >>different behavior. > > > What are the specific GS requests which are slow in response ? Are they > compute intensive ? I am sending only request for rpc.mgtclass = IB_PERFORMANCE_CLASS; rpc.method = IB_MAD_METHOD_GET; at every one second. > > In general, there are a few possibilities (which can cause this). SM > traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most > subnets). > > Some possibilities are: > 1. Timeout/retry being hit for some GS traffic (GS request or response > lost/corrupted) Yes, this is also happening, Sometimes I am getting corrupt data back, and if I retry to send same request again it fails or send corrupted data back again. > 2. Data VL busy (is there anything else utilizing VL0 ?) Not sure about it. Is there anything to verify it? I am running openSM also but not using it.. > 3. CPU busy (what other things are competing for CPU on the responder ?) CPU utilization is ok. and all requests are lid routed routed. > > -- Hal > > >>Is it expected behavior if yes then why ? >> >>Thanks and Regards >>sumit >>_______________________________________________ >>general mailing list >>general at lists.openfabrics.org >>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From kliteyn at dev.mellanox.co.il Wed Jun 25 00:07:49 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 25 Jun 2008 10:07:49 +0300 Subject: [ofa-general] [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing some data types and log messages Message-ID: <4861EEC5.8060300@dev.mellanox.co.il> Hi Sasha, Fixing some data types and corresponding log messages, thus solving compiler warnings. [ sorry for the dup, my previous message is being held hostage by the openib anti-spam filter, probably because of syntax errors in the subject :) ] Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_state_mgr.c | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index f15a4af..6ca276b 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -863,7 +863,8 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) osm_port_t *p_port; osm_port_t *p_next_port; cl_ptr_vector_t *p_port_lid_tbl; - size_t max_lid, ref_size, curr_size, lid; + size_t ref_size, curr_size; + uint16_t max_lid, lid; osm_port_t *p_port_ref, *p_port_stored; cl_ptr_vector_t ref_port_lid_tbl; uint16_t min_lid_ho; @@ -921,7 +922,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * with the new lid we wanted to give it in our * port_lid_tbl. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3322: " - "lid %u is wrongly assigned to port 0x%016" + "lid 0x%04x is wrongly assigned to port 0x%016" PRIx64 " (\'%s\' port %u) in port_lid_tbl\n", lid, cl_ntoh64(osm_port_get_guid(p_port_stored)), @@ -934,7 +935,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * original lid. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3323: " "port 0x%016" PRIx64 " (\'%s\' port %u)" - " exists in new port_lid_tbl under lid %u," + " exists in new port_lid_tbl under lid 0x%04x," " but missing in subnet port_lid_tbl db\n", cl_ntoh64(osm_port_get_guid(p_port_ref)), p_port_ref->p_node->print_desc, @@ -945,7 +946,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * it didn't reach it, and p_port_ref also didn't get * the lid update. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3324: " - "lid %u has port 0x%016" PRIx64 + "lid 0x%04x has port 0x%016" PRIx64 " (\'%s\' port %u) in new port_lid_tbl db, " "and port 0x%016" PRIx64 " (\'%s\' port %u)" " in subnet port_lid_tbl db\n", lid, -- 1.5.1.4 From kliteyn at mellanox.co.il Wed Jun 25 00:53:51 2008 From: kliteyn at mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 25 Jun 2008 10:53:51 +0300 Subject: [ofa-general] [PATCH] opensm/osm_ucast_mgr: code consolidation and cleanup In-Reply-To: <20080624204509.GS7341@sashak.voltaire.com> References: <1214252698.5369.537.camel@cardanus.llnl.gov> <20080624130950.GL7341@sashak.voltaire.com> <20080624204340.GR7341@sashak.voltaire.com> <20080624204509.GS7341@sashak.voltaire.com> Message-ID: <4861F98F.8080308@mellanox.co.il> Hi Sasha, Sasha Khapyorsky wrote: > There are some code consolidation, functionality merging and removing > intermediate lists. Also osm_port_t structure has list_item field now > which is used to maintain port order list for routing. > > This is a preparation for routing port ordering based on leaf switches > reverse sorted by number of active links. > > Signed-off-by: Sasha Khapyorsky > --- > opensm/include/opensm/osm_port.h | 4 + > opensm/include/opensm/osm_ucast_mgr.h | 5 + > opensm/opensm/osm_ucast_mgr.c | 254 ++++++--------------------------- > 3 files changed, 55 insertions(+), 208 deletions(-) OpenSM crashed in cl_qlist_insert_tail() on the following assert: CL_ASSERT(p_list_item->p_list != p_list); See below: > -static void > -__osm_ucast_mgr_build_fwd_tables(IN osm_ucast_mgr_t * const p_mgr) > +static void add_port_to_order_list(cl_map_item_t * const p_map_item, void *ctx) > { > - struct osm_ucast_port_context ctx; > - osm_opensm_t *p_osm; > - cl_qmap_t *p_sw_guid_tbl; > - cl_qlist_t guid_routing_order_list; > - cl_qlist_t port_routing_order_list; > - unsigned int lists_created = 0; > - unsigned int use_routing_order = 0; > - int ret = 0; > + osm_port_t *port = (osm_port_t *)p_map_item; > + osm_ucast_mgr_t *m = ctx; > > - p_osm = p_mgr->p_subn->p_osm; > - p_sw_guid_tbl = &p_mgr->p_subn->sw_guid_tbl; Crashed during cl_qlist_insert_tail(...) here: > + if (!port->flag) > + cl_qlist_insert_tail(&m->port_order_list, &port->list_item); > + else > + port->flag = 0; > +} ... > +static void ucast_mgr_build_lfts(osm_ucast_mgr_t *p_mgr) > +{ > + cl_qlist_init(&p_mgr->port_order_list); > > - OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, > + if (p_mgr->p_subn->opt.guid_routing_order_file) { > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > "Fetching guid routing order file %s\n", > - p_osm->subn.opt.guid_routing_order_file); > - > - cl_qlist_init(&guid_routing_order_list); > - cl_qlist_init(&port_routing_order_list); > - > - lists_created++; > + p_mgr->p_subn->opt.guid_routing_order_file); > > - list_ctx.p_mgr = p_mgr; > - list_ctx.p_list = &guid_routing_order_list; > - > - ret = parse_node_map(p_osm->subn.opt.guid_routing_order_file, > - __guid_qlist_insert, > - &list_ctx); > - if (!ret && cl_qlist_count(&guid_routing_order_list)) { > - if (__port_qlist_create(p_mgr, > - &guid_routing_order_list, > - &port_routing_order_list) < 0) > - goto cleanup; > - use_routing_order++; > - } > - else > - OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR : " > + if (parse_node_map(p_mgr->p_subn->opt.guid_routing_order_file, > + add_guid_to_order_list, p_mgr)) > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR : " > "cannot parse guid routing order file %s\n", > - p_osm->subn.opt.guid_routing_order_file); > + p_mgr->p_subn->opt.guid_routing_order_file); > } > > - if (use_routing_order) { > - ctx.p_mgr = p_mgr; > - ctx.p_port_tbl = NULL; > - ctx.p_port_list = &port_routing_order_list; > - } > - else { > - ctx.p_mgr = p_mgr; > - ctx.p_port_tbl = &p_mgr->p_subn->port_guid_tbl; > - ctx.p_port_list = NULL; > - } Entered to add_port_to_order_list from here: > + cl_qmap_apply_func(&p_mgr->p_subn->port_guid_tbl, > + add_port_to_order_list, p_mgr); -- Yevgeny -- Yevgeny Kliteynik Mellanox Technologies LTD. SW Phone: +972 (4) 909 7200 (ext 394) Mobile: +972 (52) 693 0010 E-mail: kliteyn at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From eli at mellanox.co.il Wed Jun 25 01:25:52 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Wed, 25 Jun 2008 11:25:52 +0300 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: References: Message-ID: <1214382352.23583.54.camel@mtls03> What you describe makes sense to me but it looks like your fix opens another hole. Now you removed the limit on the number of drain WRs that may be posted and you risk a CQ overflow. Wouldn't it be simpler to just move one entry from the flush list to the drain list instead of moving appending the whole flush list to the drain list? diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 6223fc3..1fcbf7b 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -222,7 +222,7 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); - list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_drain_list); + list_move(priv->cm.rx_flush_list.next, &priv->cm.rx_drain_list); } static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) On Tue, 2008-06-24 at 23:16 -0700, Shirley Ma wrote: > Hello Roland, > > Let's assume this senarios: > > 1. QP100 last WQE reached event, QP100 context is added into > flush_list, and then it is put into drain_list, and does post_send of > a drain WR. > 2. QP200 last WQE reached event, QP200 context is added into > flush_list, but not drain_list since only one drain WR will be posted > 3. QP300 ...., QP300 context is added into flush_list, but not > drain_list > > So QP100 is on drain_list, QP200, QP300 are on flush_list > > In rcq poll_cq, > 1. QP 100 drain WR cqe is polled, it will put QP100 into reap_list > then call ipoib_cm_start_rx_drain(), post_send of QP200 drain WR, and > QP200, QP300 are both moved from flush_list to drain_list > 2. QP 200 drain WR cqe is polled, it will move both QP200 and QP300 > from drain_list to reap_list > 3. QP300 cqe comes, but QP300 context has been freed, ---> panic. > > Does that make sense? > > Thanks > Shirley > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Eli Cohen Mellanox Technologies LTD. SW- Linux Phone: +972 (4) 909 7200 (ext 276) Mobile: +972 (54) 650 1206 E-mail: eli at mellanox.co.il ---------------------------------------------------------------------- Emails belong on computers, trees belong in forests; if you must print this, do it on recycled paper. http://www.greenpeace.org/international/ ---------------------------------------------------------------------- Disclaimer added by CodeTwo Exchange Rules http://www.codetwo.com From ogerlitz at voltaire.com Wed Jun 25 01:38:12 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 11:38:12 +0300 Subject: [ofa-general] [PATCH v1 for-2.6.27] IPOIB: add LRO support In-Reply-To: <4860E37C.20502@dev.mellanox.co.il> References: <4860A5BE.7090608@dev.mellanox.co.il> <4860C150.8070006@voltaire.com> <4860E37C.20502@dev.mellanox.co.il> Message-ID: <486203F4.2050409@voltaire.com> Vladimir Sokolovsky wrote: > +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, > + void **tcph, u64 *hdr_flags, void *priv) > +{ > ... > + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) > + return -1; > > This will return -1 if HCA is not capable of doing checksum offload. Is there any reason to relate the checksum offload capability with LRO? Or. From ogerlitz at voltaire.com Wed Jun 25 02:26:12 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 12:26:12 +0300 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <1214318386.23583.37.camel@mtls03> References: <1214318386.23583.37.camel@mtls03> Message-ID: <48620F34.6090901@voltaire.com> Eli Cohen wrote: > When an SKB cannot be chained to a session, the current code attempts to "restore" its ip_summed field from lro_mgr->ip_summed. However, lro_mgr->ip_summed does not hold the original value; in fact, we'd better not touch skb->ip_summed since it is not modified by the code in the path leading to a failure to chain it. > > --- a/net/ipv4/inet_lro.c > +++ b/net/ipv4/inet_lro.c > @@ -383,8 +383,7 @@ static int __lro_proc_skb(struct net_lro_mgr *lro_mgr, struct sk_buff *skb, > out2: /* send aggregated SKBs to stack */ > lro_flush(lro_mgr, lro_desc); > > -out: /* Original SKB has to be posted to stack */ > - skb->ip_summed = lro_mgr->ip_summed; > +out: > return 1; > } Jan-Bernd, I understand from your response that lro_mgr->ip_summed is not needed, so I guess it should removed from all other places that (eg its definition and usage in inet_lro.[ch] and under drivers/net. Second, if lro_mgr->aggr_ip_summed is indeed needed, I tend to think it need to be derived per received packet from skb->ip_summed, since the kernel allows for drivers ti have different checksum offload capabilities which for some drivers might be impossible to be encoded in one global value (lro_mgr->aggr_ip_summed), what's your thinking here? Third, consider a case where the receiver gets some very small data chunks (eg file/block target that has to serve lots of IOPS for some clients but also large IOs for other clients), that is some senders set TCP_NODELAY, etc. Now, looking in the code _lro_proc_skb() (below) and doing reading some reads at the archives, my understanding is that its very possible that a large set of small packets would be gathered and sent up to the stack only by the consumer calling lro_flush_all in the end of its NAPI poll loop. Am I correct? Or > static int __lro_proc_skb(struct net_lro_mgr *lro_mgr, struct sk_buff *skb, > struct vlan_group *vgrp, u16 vlan_tag, void *priv) > { > struct net_lro_desc *lro_desc; > struct iphdr *iph; > struct tcphdr *tcph; > u64 flags; > int vlan_hdr_len = 0; > > if (!lro_mgr->get_skb_header > || lro_mgr->get_skb_header(skb, (void *)&iph, (void *)&tcph, > &flags, priv)) > goto out; > > if (!(flags & LRO_IPV4) || !(flags & LRO_TCP)) > goto out; > > lro_desc = lro_get_desc(lro_mgr, lro_mgr->lro_arr, iph, tcph); > if (!lro_desc) > goto out; > > if ((skb->protocol == htons(ETH_P_8021Q)) > && !(lro_mgr->features & LRO_F_EXTRACT_VLAN_ID)) > vlan_hdr_len = VLAN_HLEN; > > if (!lro_desc->active) { /* start new lro session */ > if (lro_tcp_ip_check(iph, tcph, skb->len - vlan_hdr_len, NULL)) > goto out; > > skb->ip_summed = lro_mgr->ip_summed_aggr; > lro_init_desc(lro_desc, skb, iph, tcph, vlan_tag, vgrp); > LRO_INC_STATS(lro_mgr, aggregated); > return 0; > } > > if (lro_desc->tcp_next_seq != ntohl(tcph->seq)) > goto out2; > > if (lro_tcp_ip_check(iph, tcph, skb->len, lro_desc)) > goto out2; > > lro_add_packet(lro_desc, skb, iph, tcph); > LRO_INC_STATS(lro_mgr, aggregated); > > if ((lro_desc->pkt_aggr_cnt >= lro_mgr->max_aggr) || > lro_desc->parent->len > (0xFFFF - lro_mgr->dev->mtu)) > lro_flush(lro_mgr, lro_desc); > > return 0; > > out2: /* send aggregated SKBs to stack */ > lro_flush(lro_mgr, lro_desc); > > out: /* Original SKB has to be posted to stack */ > skb->ip_summed = lro_mgr->ip_summed; > return 1; > } From vlad at lists.openfabrics.org Wed Jun 25 02:38:16 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 25 Jun 2008 02:38:16 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080625-0200 daily build status Message-ID: <20080625093816.37644E6072A@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080625-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From tziporet at dev.mellanox.co.il Wed Jun 25 02:44:02 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Wed, 25 Jun 2008 12:44:02 +0300 Subject: [ofa-general] performance drop for datagram mode with the newconnectx FW In-Reply-To: References: <20080623114236.GA11852@mtls03> Message-ID: <48621362.6090808@mellanox.co.il> Suresh Shelvapille wrote: > I also get about 650MB for datagram mode with 2.3 FW. > > And how much with 2.5.0? Tziporet From sashak at voltaire.com Wed Jun 25 03:17:34 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 13:17:34 +0300 Subject: [ofa-general] [PATCH] opensm/osm_ucast_mgr: code consolidation and cleanup In-Reply-To: <4861F98F.8080308@mellanox.co.il> References: <1214252698.5369.537.camel@cardanus.llnl.gov> <20080624130950.GL7341@sashak.voltaire.com> <20080624204340.GR7341@sashak.voltaire.com> <20080624204509.GS7341@sashak.voltaire.com> <4861F98F.8080308@mellanox.co.il> Message-ID: <20080625101734.GB22159@sashak.voltaire.com> Hi Yevgeny, On 10:53 Wed 25 Jun , Yevgeny Kliteynik wrote: > OpenSM crashed in cl_qlist_insert_tail() on the following assert: > > CL_ASSERT(p_list_item->p_list != p_list); Yes, I see why it does. Thanks for finding this. Something like this: diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index b9e484e..39ce2bd 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -743,6 +743,9 @@ static void ucast_mgr_build_lfts(osm_ucast_mgr_t *p_mgr) cl_qmap_apply_func(&p_mgr->p_subn->sw_guid_tbl, __osm_ucast_mgr_process_tbl, p_mgr); + + while(cl_is_qlist_empty(&p_mgr->port_order_list)) + cl_qlist_remove_head(&p_mgr->port_order_list); } /********************************************************************** , solves it "conventionally". OTOH I am not sure that this CL_ACCERT() is optimal. As far as I can understand it check that list element is not entered list twice (and it is ok), but as side effect it also doesn't permit to reinitialize and reuse the qlist without explicit removing all members (as was done in this patch and I guess in some other places too) and this is bad. Personally I would vote for removing this assert at all - a bugs which it tries to prevent are fatal in runtime anyway. Thoughts? Sasha From THEMANN at de.ibm.com Wed Jun 25 04:28:05 2008 From: THEMANN at de.ibm.com (Jan-Bernd Themann) Date: Wed, 25 Jun 2008 13:28:05 +0200 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <48620F34.6090901@voltaire.com> Message-ID: Hi Or Gerlitz wrote on 25.06.2008 11:26:12: > Jan-Bernd, > > I understand from your response that lro_mgr->ip_summed is not needed, > so I guess it should removed from all other places that (eg its > definition and usage in inet_lro.[ch] and under drivers/net. > no, what I meant is that it is only not needed at that particular place as the packet is not handled by LRO. Without this line the driver can set an individual value for each SKB that is not aggregated if wished. For example when the packet is not a valid IP packet. However, removing all ip_summed fields impacts the fragment lro mode. There we have to set some value for not aggregated packets. The SKBs are generated within the LRO engine. If desired (and if there is HW that wants to use that) we can pass that value for each provided fragment. This would add one additional paramter to the already 8 parameters of __lro_proc_segment. That is of course possible. > Second, if lro_mgr->aggr_ip_summed is indeed needed, I tend to think it > need to be derived per received packet from skb->ip_summed, since the > kernel allows for drivers ti have different checksum offload > capabilities which for some drivers might be impossible to be encoded in > one global value (lro_mgr->aggr_ip_summed), what's your thinking here? I think that for valid TCP/IP packets this value should always be the same as the hardware either support the set ip_summed_aggr value for TCP/IPv4 packets, or not. Maybe that assumption is not right, but so far I haven't seen any hardware that behaves in a different way. > > Third, consider a case where the receiver gets some very small data > chunks (eg file/block target that has to serve lots of IOPS for some > clients but also large IOs for other clients), that is some senders set > TCP_NODELAY, etc. Now, looking in the code _lro_proc_skb() (below) and > doing reading some reads at the archives, my understanding is that its > very possible that a large set of small packets would be gathered and > sent up to the stack only by the consumer calling lro_flush_all in the > end of its NAPI poll loop. Am I correct? yes, that is possible. An increased delay is the prise of LRO :-) Regards, Jan-Bernd -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at voltaire.com Wed Jun 25 04:36:16 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 14:36:16 +0300 Subject: [ofa-general] Re: [PATCH 2/3 v2] mlx4: multicast cast loopback block In-Reply-To: References: <4860C216.3010400@voltaire.com> Message-ID: <48622DB0.6090604@voltaire.com> Roland Dreier wrote: > sigh... I can fix it up this time but it's such a pointless waste to send patches with problems when a trivial, easy-to-run script can tell you how to fix them. Understood. Will act to have myself and people here run it before sending next times ... Don't bother to fix it yourself, there's still time, Ron will fix and resend. > not a big deal either way. I prefer my way because if I have > if (something) > func(lots, of, parameters, all, the, same, except, 1); > else > func(lots, of, parameters, all, the, same, except, 0); > then it's not always so clear what the difference between the two > function calls really is without reading carefully. By putting > !!something into the call: > > func(lots, of, parameters, all, the, same, except, !!something); > > it becomes obvious how the value of something is used. yes, I see the point, once you get used to it, !! code is clearer than func(lots, of, parameters, all, the, same, except, X). So Ron will fix this as well. Or. Or. From jean-vincent.ficet at bull.net Wed Jun 25 04:36:58 2008 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Wed, 25 Jun 2008 13:36:58 +0200 Subject: [ofa-general] Issues with osm_state_mgr.c:__osm_state_mgr_get_remote_port_info() Message-ID: <48622DDA.40509@bull.net> Hi, In the file opensm/osm_state_mgr.c, function __osm_state_mgr_get_remote_port_info(), we have the following code: mad_context.pi_context.port_guid = cl_hton64(osm_physp_get_port_num(p_physp)); The port_guid's type is ib_net64_t, which is OK w.r.t cl_hton64 type cast. However, the osm_physp_get_port_num() function in include/opensm/osm_port.h is implemented as follows: static inline uint8_t osm_physp_get_port_num(IN const osm_physp_t * const p_physp) { CL_ASSERT(p_physp); CL_ASSERT(osm_physp_is_valid(p_physp)); return (p_physp->port_num); } And the port_num field in the osm_physp_t is declared as an uint8_t. Am I right in saying that feeding a port number into mad_context.pi_context.port_guid is incorrect ? If so, the cl_hton64 type cast is hiding the compiler warning ;-) Thanks, Vincent From ogerlitz at voltaire.com Wed Jun 25 04:47:25 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 14:47:25 +0300 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: References: Message-ID: <4862304D.2050306@voltaire.com> Jan-Bernd Themann wrote: > no, what I meant is that it is only not needed at that particular > place as the packet is not handled by LRO. Without this line the > driver can set an individual value for each SKB that is not > aggregated if wished. For example when the packet is not a valid IP > packet. However, removing all ip_summed fields impacts the fragment > lro mode. There we have to set some value for not aggregated packets. > The SKBs are generated within the LRO engine. If desired (and if there > is HW that wants to use that) we can pass that value for each provided > fragment. This would add one additional paramter to the already 8 > parameters of __lro_proc_segment. That is of course possible. OK, understood, both points. Eli, lets add to this patch a comment in inet_lro.h saying that the value of lro_mgr->ip_summed is ignored by the core lro code for drivers that use the non fragmented mode. Also for the ipoib patch, lets not set this value. > I think that for valid TCP/IP packets this value should always be the > same as the hardware either support the set ip_summed_aggr value for > TCP/IPv4 packets, or not. Maybe that assumption is not right, but so > far I haven't seen any hardware that behaves in a different way. Yes, for TCP/IPv4 you seem to be right and here the problem was in the lro patch to ipoib which set this value blindly regardless of the HW capabilities, I asked Vlad to change this in the next version of the patch. As for other types of traffic, I was thinking that allowing the driver to set it per packet makes a better isolation between the core lro code to the driver, but this is not major issue. > yes, that is possible. An increased delay is the prise of LRO :-) > Is there some pointer you might be able to provide on LRO benchmark for small packets and/or mixed small/large packet streams? Or. From sashak at voltaire.com Wed Jun 25 04:53:07 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 14:53:07 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_vendor_select.h: undoing change in default vendor selection In-Reply-To: <4861E9C7.1010802@dev.mellanox.co.il> References: <4861E9C7.1010802@dev.mellanox.co.il> Message-ID: <20080625115307.GF22159@sashak.voltaire.com> On 09:46 Wed 25 Jun , Yevgeny Kliteynik wrote: > > Undoing a change in the default vendor selection. > The patch broke ibmgtsim compilation. This patch breaks external plugin compilation where osm_vendor.h is included implicitly (requires complicated vendor selection for installed vendor layer, although it should be transparent for event plugins). I tried both: cd ibmgtsim && ./configure --prefix=${ROOT} --with-ibdm=${ROOT} \ && make && make install , and cd opensm && ${OSM_DIR}/configure --with-osmv=sim --with-sim=${ROOT} \ --prefix=${ROOT} && make && make install And it works fine for me. Could you provide more details about the failure? How to reproduce it? Another solutions can be to make osm_vendor_select.h to be autogenerated by ./configure script or even better - to make vendor independent vendor API. But both are much larger change. Sasha From THEMANN at de.ibm.com Wed Jun 25 05:10:02 2008 From: THEMANN at de.ibm.com (Jan-Bernd Themann) Date: Wed, 25 Jun 2008 14:10:02 +0200 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <4862304D.2050306@voltaire.com> Message-ID: Or Gerlitz wrote on 25.06.2008 13:47:25: > Jan-Bernd Themann wrote: > > no, what I meant is that it is only not needed at that particular > > place as the packet is not handled by LRO. Without this line the > > driver can set an individual value for each SKB that is not > > aggregated if wished. For example when the packet is not a valid IP > > packet. However, removing all ip_summed fields impacts the fragment > > lro mode. There we have to set some value for not aggregated packets. > > The SKBs are generated within the LRO engine. If desired (and if there > > is HW that wants to use that) we can pass that value for each provided > > fragment. This would add one additional paramter to the already 8 > > parameters of __lro_proc_segment. That is of course possible. > OK, understood, both points. > > Eli, lets add to this patch a comment in inet_lro.h saying that the > value of lro_mgr->ip_summed is ignored by the core lro code for drivers > that use the non fragmented mode. Also for the ipoib patch, lets not set > this value. > > > I think that for valid TCP/IP packets this value should always be the > > same as the hardware either support the set ip_summed_aggr value for > > TCP/IPv4 packets, or not. Maybe that assumption is not right, but so > > far I haven't seen any hardware that behaves in a different way. > Yes, for TCP/IPv4 you seem to be right and here the problem was in the > lro patch to ipoib which set this value blindly regardless of the HW > capabilities, I asked Vlad to change this in the next version of the > patch. As for other types of traffic, I was thinking that allowing the > driver to set it per packet makes a better isolation between the core > lro code to the driver, but this is not major issue. > > yes, that is possible. An increased delay is the prise of LRO :-) > > > Is there some pointer you might be able to provide on LRO benchmark for > small packets and/or mixed small/large packet streams? > > Or. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From THEMANN at de.ibm.com Wed Jun 25 05:15:19 2008 From: THEMANN at de.ibm.com (Jan-Bernd Themann) Date: Wed, 25 Jun 2008 14:15:19 +0200 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <4862304D.2050306@voltaire.com> Message-ID: Hi Or, sorry, had a problem with the last mail... Or Gerlitz wrote on 25.06.2008 13:47:25: > Jan-Bernd Themann wrote: > > no, what I meant is that it is only not needed at that particular > > place as the packet is not handled by LRO. Without this line the > > driver can set an individual value for each SKB that is not > > aggregated if wished. For example when the packet is not a valid IP > > packet. However, removing all ip_summed fields impacts the fragment > > lro mode. There we have to set some value for not aggregated packets. > > The SKBs are generated within the LRO engine. If desired (and if there > > is HW that wants to use that) we can pass that value for each provided > > fragment. This would add one additional paramter to the already 8 > > parameters of __lro_proc_segment. That is of course possible. > OK, understood, both points. > > Eli, lets add to this patch a comment in inet_lro.h saying that the > value of lro_mgr->ip_summed is ignored by the core lro code for drivers > that use the non fragmented mode. Also for the ipoib patch, lets not set > this value. > > > I think that for valid TCP/IP packets this value should always be the > > same as the hardware either support the set ip_summed_aggr value for > > TCP/IPv4 packets, or not. Maybe that assumption is not right, but so > > far I haven't seen any hardware that behaves in a different way. > Yes, for TCP/IPv4 you seem to be right and here the problem was in the > lro patch to ipoib which set this value blindly regardless of the HW > capabilities, I asked Vlad to change this in the next version of the > patch. As for other types of traffic, I was thinking that allowing the > driver to set it per packet makes a better isolation between the core > lro code to the driver, but this is not major issue. > > yes, that is possible. An increased delay is the prise of LRO :-) > > > Is there some pointer you might be able to provide on LRO benchmark for > small packets and/or mixed small/large packet streams? I don't have any benchmarks handy, but Andrew Gallatin from myrinet posted his results on the mailing list some month ago. I'll let you know if I find a link. Regards, Jan-Bernd -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Wed Jun 25 05:30:05 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 25 Jun 2008 15:30:05 +0300 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <4862304D.2050306@voltaire.com> References: <4862304D.2050306@voltaire.com> Message-ID: <1214397005.23583.60.camel@mtls03> On Wed, 2008-06-25 at 14:47 +0300, Or Gerlitz wrote: > Jan-Bernd Themann wrote: > > no, what I meant is that it is only not needed at that particular > > place as the packet is not handled by LRO. Without this line the > > driver can set an individual value for each SKB that is not > > aggregated if wished. For example when the packet is not a valid IP > > packet. However, removing all ip_summed fields impacts the fragment > > lro mode. There we have to set some value for not aggregated packets. > > The SKBs are generated within the LRO engine. If desired (and if there > > is HW that wants to use that) we can pass that value for each provided > > fragment. This would add one additional paramter to the already 8 > > parameters of __lro_proc_segment. That is of course possible. > OK, understood, both points. > > Eli, lets add to this patch a comment in inet_lro.h saying that the > value of lro_mgr->ip_summed is ignored by the core lro code for drivers > that use the non fragmented mode. We already have this comment: struct net_lro_mgr { ... u32 ip_summed; /* Set in non generated SKBs in page mode */ I think we should use have it something like that: ... /* * Set for generated SKBs in that are not added to * the frag list in fragmented mode */ u32 ip_summed; What do you think? > Also for the ipoib patch, lets not set > this value. Agree. From kliteyn at dev.mellanox.co.il Tue Jun 24 23:41:00 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 25 Jun 2008 09:41:00 +0300 Subject: [ofa-general] ***SPAM*** [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages Message-ID: <4861E87C.3000007@dev.mellanox.co.il> Hi Sasha, Fixing some data types and corresponding log messages, thus solving compiler warnings. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_state_mgr.c | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index f15a4af..6ca276b 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -863,7 +863,8 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) osm_port_t *p_port; osm_port_t *p_next_port; cl_ptr_vector_t *p_port_lid_tbl; - size_t max_lid, ref_size, curr_size, lid; + size_t ref_size, curr_size; + uint16_t max_lid, lid; osm_port_t *p_port_ref, *p_port_stored; cl_ptr_vector_t ref_port_lid_tbl; uint16_t min_lid_ho; @@ -921,7 +922,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * with the new lid we wanted to give it in our * port_lid_tbl. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3322: " - "lid %u is wrongly assigned to port 0x%016" + "lid 0x%04x is wrongly assigned to port 0x%016" PRIx64 " (\'%s\' port %u) in port_lid_tbl\n", lid, cl_ntoh64(osm_port_get_guid(p_port_stored)), @@ -934,7 +935,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * original lid. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3323: " "port 0x%016" PRIx64 " (\'%s\' port %u)" - " exists in new port_lid_tbl under lid %u," + " exists in new port_lid_tbl under lid 0x%04x," " but missing in subnet port_lid_tbl db\n", cl_ntoh64(osm_port_get_guid(p_port_ref)), p_port_ref->p_node->print_desc, @@ -945,7 +946,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * it didn't reach it, and p_port_ref also didn't get * the lid update. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3324: " - "lid %u has port 0x%016" PRIx64 + "lid 0x%04x has port 0x%016" PRIx64 " (\'%s\' port %u) in new port_lid_tbl db, " "and port 0x%016" PRIx64 " (\'%s\' port %u)" " in subnet port_lid_tbl db\n", lid, -- 1.5.1.4 From sashak at voltaire.com Wed Jun 25 05:33:57 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 15:33:57 +0300 Subject: [ofa-general] Issues with osm_state_mgr.c:__osm_state_mgr_get_remote_port_info() In-Reply-To: <48622DDA.40509@bull.net> References: <48622DDA.40509@bull.net> Message-ID: <20080625123357.GG22159@sashak.voltaire.com> Hi Vincent, On 13:36 Wed 25 Jun , Vincent Ficet wrote: > > In the file opensm/osm_state_mgr.c, function > __osm_state_mgr_get_remote_port_info(), we have the following code: > > mad_context.pi_context.port_guid = > cl_hton64(osm_physp_get_port_num(p_physp)); > > The port_guid's type is ib_net64_t, which is OK w.r.t cl_hton64 type cast. > > However, the osm_physp_get_port_num() function in include/opensm/osm_port.h > is implemented as follows: > > static inline uint8_t > osm_physp_get_port_num(IN const osm_physp_t * const p_physp) > { > CL_ASSERT(p_physp); > CL_ASSERT(osm_physp_is_valid(p_physp)); > return (p_physp->port_num); > } > > And the port_num field in the osm_physp_t is declared as an uint8_t. > > Am I right in saying that feeding a port number into > mad_context.pi_context.port_guid is incorrect ? Yes, it is wrong, there should be: mad_context.pi_context.port_guid = cl_hton64(p_physp->port_guid); This function was broken, we didn't see that since it is almost unused flow (with stable network). Thanks for the finding. Want to post the patch? Sasha From sashak at voltaire.com Wed Jun 25 05:51:34 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 15:51:34 +0300 Subject: [ofa-general] Issues with osm_state_mgr.c:__osm_state_mgr_get_remote_port_info() In-Reply-To: <20080625123357.GG22159@sashak.voltaire.com> References: <48622DDA.40509@bull.net> <20080625123357.GG22159@sashak.voltaire.com> Message-ID: <20080625125134.GH22159@sashak.voltaire.com> On 15:33 Wed 25 Jun , Sasha Khapyorsky wrote: > > mad_context.pi_context.port_guid = cl_hton64(p_physp->port_guid); cl_hnon64() is not needed. > This function was broken, we didn't see that since it is almost unused > flow (with stable network). I checked the code and found that this value was not used in context of light sweep, so it worked without problem. But thanks for finding anyway. Sasha From monis at Voltaire.COM Wed Jun 25 06:00:24 2008 From: monis at Voltaire.COM (Moni Shoua) Date: Wed, 25 Jun 2008 16:00:24 +0300 Subject: [ofa-general] [PATCH V2] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: <484F8A20.9070101@Voltaire.COM> References: <484F8A20.9070101@Voltaire.COM> Message-ID: <48624168.3050305@Voltaire.COM> The patch tries to solve the problem of device going down and paths being flushed on SM change event. The method is to mark the paths as candidates for refresh (with a valid flag) and wait for ARP probe to start a flow of path lookup that leads to patch query which ends up in path (and address handle) refresh. The solution requires a different and less intrusive handling of SM change event. For that, the second argument of the flush function changed it's meaning from boolean flag to a level. In most cases, SM failover doesn't cause LID change so traffic won't stop. In the rare cases of LID change, the remote host (the one that hadn't changed its LID) will lose connectivity until paths are refreshed. This is no worse than the current state. In fact, preventing the device from going down saves packets that otherwise would be lost. Signed-off-by: Moni Levy Signed-off-by: Moni Shoua --- drivers/infiniband/ulp/ipoib/ipoib.h | 17 +++++++++-- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 42 ++++++++++++++++++----------- drivers/infiniband/ulp/ipoib/ipoib_main.c | 38 ++++++++++++++++++++++---- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 19 ++++++------- 4 files changed, 84 insertions(+), 32 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index ca126fc..bb24e34 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -55,6 +55,12 @@ /* constants */ +enum ipoib_flush_level { + IPOIB_FLUSH_LIGHT, + IPOIB_FLUSH_NORMAL, + IPOIB_FLUSH_HEAVY +}; + enum { IPOIB_ENCAP_LEN = 4, @@ -276,10 +282,11 @@ struct ipoib_dev_priv { struct delayed_work pkey_poll_task; struct delayed_work mcast_task; - struct work_struct flush_task; + struct work_struct flush_light; + struct work_struct flush_normal; + struct work_struct flush_heavy; struct work_struct restart_task; struct delayed_work ah_reap_task; - struct work_struct pkey_event_task; struct ib_device *ca; u8 port; @@ -358,6 +365,7 @@ struct ipoib_path { struct completion done; struct rb_node rb_node; + int valid; struct list_head list; }; @@ -423,11 +431,14 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_ah *address, u32 qpn); void ipoib_reap_ah(struct work_struct *work); +void ipoib_mark_paths_invalid(struct net_device *dev); void ipoib_flush_paths(struct net_device *dev); struct ipoib_dev_priv *ipoib_intf_alloc(const char *format); int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port); -void ipoib_ib_dev_flush(struct work_struct *work); +void ipoib_ib_dev_flush_light(struct work_struct *work); +void ipoib_ib_dev_flush_normal(struct work_struct *work); +void ipoib_ib_dev_flush_heavy(struct work_struct *work); void ipoib_pkey_event(struct work_struct *work); void ipoib_ib_dev_cleanup(struct net_device *dev); diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index f429bce..146ee31 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -898,7 +898,8 @@ int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port) return 0; } -static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) +static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, + enum ipoib_flush_level level) { struct ipoib_dev_priv *cpriv; struct net_device *dev = priv->dev; @@ -911,7 +912,7 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) * the parent is down. */ list_for_each_entry(cpriv, &priv->child_intfs, list) - __ipoib_ib_dev_flush(cpriv, pkey_event); + __ipoib_ib_dev_flush(cpriv, level); mutex_unlock(&priv->vlan_mutex); @@ -925,7 +926,7 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) return; } - if (pkey_event) { + if (level == IPOIB_FLUSH_HEAVY) { if (ib_find_pkey(priv->ca, priv->port, priv->pkey, &new_index)) { clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags); ipoib_ib_dev_down(dev, 0); @@ -943,11 +944,15 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) priv->pkey_index = new_index; } - ipoib_dbg(priv, "flushing\n"); + if (level == IPOIB_FLUSH_LIGHT) { + ipoib_mark_paths_invalid(dev); + ipoib_mcast_dev_flush(dev); + } - ipoib_ib_dev_down(dev, 0); + if (level >= IPOIB_FLUSH_NORMAL) + ipoib_ib_dev_down(dev, 0); - if (pkey_event) { + if (level == IPOIB_FLUSH_HEAVY) { ipoib_ib_dev_stop(dev, 0); ipoib_ib_dev_open(dev); } @@ -957,27 +962,34 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, int pkey_event) * we get here, don't bring it back up if it's not configured up */ if (test_bit(IPOIB_FLAG_ADMIN_UP, &priv->flags)) { - ipoib_ib_dev_up(dev); + if (level >= IPOIB_FLUSH_NORMAL) + ipoib_ib_dev_up(dev); ipoib_mcast_restart_task(&priv->restart_task); } } -void ipoib_ib_dev_flush(struct work_struct *work) +void ipoib_ib_dev_flush_light(struct work_struct *work) +{ + struct ipoib_dev_priv *priv = + container_of(work, struct ipoib_dev_priv, flush_light); + + __ipoib_ib_dev_flush(priv, IPOIB_FLUSH_LIGHT); +} + +void ipoib_ib_dev_flush_normal(struct work_struct *work) { struct ipoib_dev_priv *priv = - container_of(work, struct ipoib_dev_priv, flush_task); + container_of(work, struct ipoib_dev_priv, flush_normal); - ipoib_dbg(priv, "Flushing %s\n", priv->dev->name); - __ipoib_ib_dev_flush(priv, 0); + __ipoib_ib_dev_flush(priv, IPOIB_FLUSH_NORMAL); } -void ipoib_pkey_event(struct work_struct *work) +void ipoib_ib_dev_flush_heavy(struct work_struct *work) { struct ipoib_dev_priv *priv = - container_of(work, struct ipoib_dev_priv, pkey_event_task); + container_of(work, struct ipoib_dev_priv, flush_heavy); - ipoib_dbg(priv, "Flushing %s and restarting its QP\n", priv->dev->name); - __ipoib_ib_dev_flush(priv, 1); + __ipoib_ib_dev_flush(priv, IPOIB_FLUSH_HEAVY); } void ipoib_ib_dev_cleanup(struct net_device *dev) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index 2442090..3b6df37 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -350,6 +350,23 @@ void ipoib_path_iter_read(struct ipoib_path_iter *iter, #endif /* CONFIG_INFINIBAND_IPOIB_DEBUG */ +void ipoib_mark_paths_invalid(struct net_device *dev) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + struct ipoib_path *path, *tp; + spin_lock_irq(&priv->tx_lock); + spin_lock(&priv->lock); + + list_for_each_entry_safe(path, tp, &priv->path_list, list) { + ipoib_dbg(priv, "mark path LID 0x%04x GID " IPOIB_GID_FMT " invalid\n", + be16_to_cpu(path->pathrec.dlid), + IPOIB_GID_ARG(path->pathrec.dgid)); + path->valid = 0; + } + spin_unlock(&priv->lock); + spin_unlock_irq(&priv->tx_lock); +} + void ipoib_flush_paths(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -386,6 +403,7 @@ static void path_rec_completion(int status, struct net_device *dev = path->dev; struct ipoib_dev_priv *priv = netdev_priv(dev); struct ipoib_ah *ah = NULL; + struct ipoib_ah *old_ah = NULL; struct ipoib_neigh *neigh, *tn; struct sk_buff_head skqueue; struct sk_buff *skb; @@ -409,6 +427,9 @@ static void path_rec_completion(int status, spin_lock_irqsave(&priv->lock, flags); + if (path->ah) + ipoib_put_ah(path->ah); + path->ah = ah; if (ah) { @@ -421,6 +442,8 @@ static void path_rec_completion(int status, __skb_queue_tail(&skqueue, skb); list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) { + if (neigh->ah) + old_ah = neigh->ah; kref_get(&path->ah->ref); neigh->ah = path->ah; memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, @@ -443,13 +466,15 @@ static void path_rec_completion(int status, while ((skb = __skb_dequeue(&neigh->queue))) __skb_queue_tail(&skqueue, skb); } + path->valid = 1; } path->query = NULL; complete(&path->done); spin_unlock_irqrestore(&priv->lock, flags); - + if (old_ah) + ipoib_put_ah(old_ah); while ((skb = __skb_dequeue(&skqueue))) { skb->dev = dev; if (dev_queue_xmit(skb)) @@ -471,6 +496,7 @@ static struct ipoib_path *path_rec_create(struct net_device *dev, void *gid) return NULL; path->dev = dev; + path->valid = 0; skb_queue_head_init(&path->queue); @@ -623,8 +649,9 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev, spin_lock(&priv->lock); path = __path_find(dev, phdr->hwaddr + 4); - if (!path) { - path = path_rec_create(dev, phdr->hwaddr + 4); + if (!path || !path->valid) { + if (!path) + path = path_rec_create(dev, phdr->hwaddr + 4); if (path) { /* put pseudoheader back on for next time */ skb_push(skb, sizeof *phdr); @@ -989,9 +1016,10 @@ static void ipoib_setup(struct net_device *dev) INIT_LIST_HEAD(&priv->multicast_list); INIT_DELAYED_WORK(&priv->pkey_poll_task, ipoib_pkey_poll); - INIT_WORK(&priv->pkey_event_task, ipoib_pkey_event); INIT_DELAYED_WORK(&priv->mcast_task, ipoib_mcast_join_task); - INIT_WORK(&priv->flush_task, ipoib_ib_dev_flush); + INIT_WORK(&priv->flush_light, ipoib_ib_dev_flush_light); + INIT_WORK(&priv->flush_normal, ipoib_ib_dev_flush_normal); + INIT_WORK(&priv->flush_heavy, ipoib_ib_dev_flush_heavy); INIT_WORK(&priv->restart_task, ipoib_mcast_restart_task); INIT_DELAYED_WORK(&priv->ah_reap_task, ipoib_reap_ah); } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c index 8766d29..4ab06ea 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c @@ -289,15 +289,16 @@ void ipoib_event(struct ib_event_handler *handler, if (record->element.port_num != priv->port) return; - if (record->event == IB_EVENT_PORT_ERR || - record->event == IB_EVENT_PORT_ACTIVE || - record->event == IB_EVENT_LID_CHANGE || - record->event == IB_EVENT_SM_CHANGE || - record->event == IB_EVENT_CLIENT_REREGISTER) { - ipoib_dbg(priv, "Port state change event\n"); - queue_work(ipoib_workqueue, &priv->flush_task); + ipoib_dbg(priv, "Event %d on device %s port %d\n", record->event, + record->device->name, record->element.port_num); + if (record->event == IB_EVENT_SM_CHANGE || + record->event == IB_EVENT_CLIENT_REREGISTER) { + queue_work(ipoib_workqueue, &priv->flush_light); + } else if (record->event == IB_EVENT_PORT_ERR || + record->event == IB_EVENT_PORT_ACTIVE || + record->event == IB_EVENT_LID_CHANGE) { + queue_work(ipoib_workqueue, &priv->flush_normal); } else if (record->event == IB_EVENT_PKEY_CHANGE) { - ipoib_dbg(priv, "P_Key change event on port:%d\n", priv->port); - queue_work(ipoib_workqueue, &priv->pkey_event_task); + queue_work(ipoib_workqueue, &priv->flush_heavy); } } From ogerlitz at voltaire.com Wed Jun 25 06:01:18 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 16:01:18 +0300 Subject: [ofa-general] [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <1214397005.23583.60.camel@mtls03> References: <4862304D.2050306@voltaire.com> <1214397005.23583.60.camel@mtls03> Message-ID: <4862419E.5040307@voltaire.com> Eli Cohen wrote: > We already have this comment: > u32 ip_summed; /* Set in non generated SKBs in page mode */ > > I think we should use have it something like that: > /* > * Set for generated SKBs in that are not added to > * the frag list in fragmented mode > */ > u32 ip_summed; > > What do you think? > yes, this could be a little more clear. Or From 271sinoto at haos.com.tr Wed Jun 25 04:19:41 2008 From: 271sinoto at haos.com.tr (georges bert) Date: Wed, 25 Jun 2008 11:19:41 +0000 Subject: [ofa-general] Best of Chloe, Burberry, UGG Message-ID: <000901c8d6c4$036762a2$d60066ba@xqjesax> The international generous de luxe store for shoes and bags is just one click away. Estimated by tens thousands of grateful customers all around the world, we carry many of populars brands including: Chanel Armani Chanel Shoes Paul Smith Prada Shoes Here you willc find hundred thousands of best designs for shoes, and leather products, at at temp't1ng priceE. Sale ends this week, so visit shop today and start pampering yourself and your loved ones. - Visit our site: www.ladenume[DOT]com (copy this link and replace "[DOT]" to ".") From monis at Voltaire.COM Wed Jun 25 05:51:45 2008 From: monis at Voltaire.COM (Moni Shoua) Date: Wed, 25 Jun 2008 15:51:45 +0300 Subject: ***SPAM*** Re: [ofa-general] [PATCH] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: References: <484F8A20.9070101@Voltaire.COM> <4860D860.1080805@Voltaire.COM> Message-ID: <48623F61.8070005@Voltaire.COM> > Too-long lines are OK with me, but I get the following with the patch I > was replying to: I was working with 2.6.26-rc2 which produced less output with checkpatch.pl I pulled rc8 and got the same output as yours. I already sent answers to your other comments. I will resend V2 of the patch with respect to all issues. thanks From zonalxpl at bergezentrum-trebsen.de Wed Jun 25 06:14:10 2008 From: zonalxpl at bergezentrum-trebsen.de (Delia Freeman) Date: Wed, 25 Jun 2008 09:14:10 -0400 Subject: [ofa-general] Do you have the experience but not the Degree? gskvr fuvv g Message-ID: <198244873.52488678933963@bergezentrum-trebsen.de> Want the deg hsp ree but can not find the ti nrn me? WHAT A GRE oa AT ID yfi EA! We pro gji vide a co zr ncept that will allow anyone with sufficient work experience to obtain a fully verifiable Un vbc ivers js ity Deg jn ree. Ba upb chel msr ors, Ma tg ste tdp rs or ev hpr en a Do wj cto rv rate. Think of it, within four to six weeks, you too could be a co rt lle lkn ge gr pb adu dsf ate. Many people sh le are the same fru pu stra nk tion, they are all doing the work of the person that has the de bjw gree and the person that has the d cn egree is getting all the mo ack ney. Do not you think that it is time you were paid fair comp otg ensa eue tion for the level of work you are already doing? This is your chance t dwd o fin qd ally make the right move and receive your due benefits. If you are like most people, you are more than qu tqy alif pa ied with your experience, but are lacking that pre jd stig gaa ious piece of paper known as a di mn plo plq ma that is often the pas wse sp rh ort to suc uvg cess. CA qt LL US TODAY AND GIVE YOUR WORK EXP th ERIE uuw NCE THE CHANCE TO EARN YOU THE HIG uam HER CO fzh MPENS er ATION YOU DE mj SER ool VE! 1-5 fiy 2 aa 0-2 sv 0 at 3-0 plu 21 jtr 3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at voltaire.com Wed Jun 25 06:55:40 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 16:55:40 +0300 (IDT) Subject: [ofa-general] msi-x seems to cause large performance variation for connectx / datagram mode Message-ID: While doing a synthetic network benchmark (netperf) of 64K message size, on two nodes each with two HCAs: Arbel & ConnectX (where the idea was to have an apples-to-apples comparison), I saw a nice advatange with connectx, but there is kind of large variation in the performance it gives. The system spec is: RH5 x86_64 2.6.18-8.el5 SMP, 4GB RAM, Intel 1.6GHz four CPUs (or two CPUS and two cores each, I don't know) - I used datagram mode, where connectx has checksum and LSO offloads - the code used is not the mainline kernel but rather git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel commit 39e1dc833f98e5134f91fcf7f33df402adf4bc0c digging a little more, I see now that if I disable msi-x, the performance gets to a fixed value of about 620MB/s with connectx which is the lower range that msi-x gives me... HCA FW MB/s msi_x ============================================ Arbel-Arbel 4.8.2 450 0 (ib_mthca) Arbel-Arbel 4.8.2 450 1 (ib_mthca) ConnectX-ConnectX 2.3 620-850 1 (mlx4_core) ConnectX-ConnectX 2.3 620 0 (mlx4_core) Any idea if there's something in the system settings that can explain this? I have pasted below some lspci info and netperf results, it might help. Note that its FW 2.3 and not 2.5 on which I see other issues... Or. # modprobe mlx4_core msi_x=0 # lspci -v | grep -A 15 634a 03:00.0 InfiniBand: Mellanox Technologies Unknown device 634a (rev a0) Subsystem: Mellanox Technologies Unknown device 634a Flags: bus master, fast devsel, latency 0, IRQ 169 Memory at c8300000 (64-bit, non-prefetchable) [size=1M] Memory at c9800000 (64-bit, prefetchable) [size=8M] Memory at c8200000 (64-bit, non-prefetchable) [size=8K] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable- Mask- TabSize=256 Capabilities: [60] Express Endpoint IRQ 0 # modprobe mlx4_core msi_x=1 # lspci -v | grep -A 10 634a 03:00.0 InfiniBand: Mellanox Technologies Unknown device 634a (rev a0) Subsystem: Mellanox Technologies Unknown device 634a Flags: bus master, fast devsel, latency 0, IRQ 169 Memory at c8300000 (64-bit, non-prefetchable) [size=1M] Memory at c9800000 (64-bit, prefetchable) [size=8M] Memory at c8200000 (64-bit, non-prefetchable) [size=8K] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256 Capabilities: [60] Express Endpoint IRQ 0 - Connectx / Connectx msi_x = 1 # netperf -H 192.168.2.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.152 (192.168.2.152) port 0 AF_INET : demo Interim result: 648.10 MBytes/s over 1.02 seconds Interim result: 630.23 MBytes/s over 1.03 seconds Interim result: 629.91 MBytes/s over 1.00 seconds Interim result: 630.24 MBytes/s over 1.00 seconds Interim result: 630.08 MBytes/s over 1.00 seconds Interim result: 630.40 MBytes/s over 1.00 seconds Interim result: 630.19 MBytes/s over 1.00 seconds Interim result: 629.88 MBytes/s over 1.00 seconds Interim result: 630.18 MBytes/s over 1.00 seconds Interim result: 630.17 MBytes/s over 1.00 seconds Interim result: 646.35 MBytes/s over 1.00 seconds Interim result: 659.88 MBytes/s over 1.00 seconds Interim result: 662.61 MBytes/s over 1.00 seconds Interim result: 663.06 MBytes/s over 1.00 seconds Interim result: 662.32 MBytes/s over 1.00 seconds Interim result: 661.93 MBytes/s over 1.00 seconds Interim result: 662.78 MBytes/s over 1.00 seconds Interim result: 662.59 MBytes/s over 1.00 seconds Interim result: 664.89 MBytes/s over 1.00 seconds Interim result: 865.67 MBytes/s over 1.00 seconds Interim result: 726.10 MBytes/s over 1.19 seconds Interim result: 623.57 MBytes/s over 1.16 seconds Interim result: 623.10 MBytes/s over 1.00 seconds Interim result: 622.99 MBytes/s over 1.00 seconds Interim result: 623.47 MBytes/s over 1.00 seconds Interim result: 622.95 MBytes/s over 1.00 seconds Interim result: 623.52 MBytes/s over 1.00 seconds Interim result: 623.26 MBytes/s over 1.00 seconds Interim result: 623.06 MBytes/s over 1.00 seconds Interim result: 623.40 MBytes/s over 1.00 seconds Interim result: 825.80 MBytes/s over 1.00 seconds Interim result: 862.10 MBytes/s over 1.00 seconds Interim result: 862.41 MBytes/s over 1.00 seconds Interim result: 862.98 MBytes/s over 1.00 seconds Interim result: 862.83 MBytes/s over 1.00 seconds Interim result: 862.66 MBytes/s over 1.00 seconds Interim result: 862.08 MBytes/s over 1.00 seconds Interim result: 861.96 MBytes/s over 1.00 seconds Interim result: 861.52 MBytes/s over 1.00 seconds Interim result: 861.32 MBytes/s over 1.00 seconds Interim result: 650.12 MBytes/s over 1.32 seconds Interim result: 623.67 MBytes/s over 1.04 seconds Interim result: 623.60 MBytes/s over 1.00 seconds Interim result: 623.16 MBytes/s over 1.00 seconds Interim result: 623.38 MBytes/s over 1.00 seconds Interim result: 623.34 MBytes/s over 1.00 seconds Interim result: 622.93 MBytes/s over 1.00 seconds Interim result: 622.67 MBytes/s over 1.00 seconds Interim result: 623.09 MBytes/s over 1.00 seconds Interim result: 674.60 MBytes/s over 1.00 seconds Interim result: 862.36 MBytes/s over 1.00 seconds Interim result: 862.32 MBytes/s over 1.00 seconds Interim result: 862.60 MBytes/s over 1.00 seconds Interim result: 862.77 MBytes/s over 1.00 seconds Interim result: 862.42 MBytes/s over 1.00 seconds Interim result: 862.97 MBytes/s over 1.00 seconds Interim result: 862.56 MBytes/s over 1.00 seconds Interim result: 863.27 MBytes/s over 1.00 seconds Interim result: 862.66 MBytes/s over 1.00 seconds Interim result: 794.91 MBytes/s over 1.09 seconds Interim result: 622.52 MBytes/s over 1.28 seconds Interim result: 622.43 MBytes/s over 1.00 seconds Interim result: 622.68 MBytes/s over 1.00 seconds Interim result: 622.51 MBytes/s over 1.00 seconds Interim result: 622.34 MBytes/s over 1.00 seconds Interim result: 622.11 MBytes/s over 1.00 seconds Interim result: 622.11 MBytes/s over 1.00 seconds Interim result: 621.71 MBytes/s over 1.00 seconds Interim result: 621.85 MBytes/s over 1.00 seconds Interim result: 761.27 MBytes/s over 1.00 seconds Interim result: 861.58 MBytes/s over 1.00 seconds Interim result: 861.76 MBytes/s over 1.00 seconds Interim result: 861.10 MBytes/s over 1.00 seconds Interim result: 862.02 MBytes/s over 1.00 seconds Interim result: 861.63 MBytes/s over 1.00 seconds - Arbel/Arbel msi_x = 0 # netperf -H 192.168.1.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.152 (192.168.1.152) port 0 AF_INET : demo Interim result: 464.73 MBytes/s over 1.00 seconds Interim result: 462.77 MBytes/s over 1.00 seconds Interim result: 463.02 MBytes/s over 1.00 seconds Interim result: 463.07 MBytes/s over 1.00 seconds Interim result: 462.93 MBytes/s over 1.00 seconds Interim result: 462.85 MBytes/s over 1.00 seconds Interim result: 462.83 MBytes/s over 1.00 seconds Interim result: 463.21 MBytes/s over 1.00 seconds Interim result: 464.18 MBytes/s over 1.00 seconds Interim result: 464.03 MBytes/s over 1.00 seconds Interim result: 464.59 MBytes/s over 1.00 seconds Interim result: 464.19 MBytes/s over 1.00 seconds Interim result: 464.56 MBytes/s over 1.00 seconds Interim result: 464.50 MBytes/s over 1.00 seconds Interim result: 464.45 MBytes/s over 1.00 seconds Interim result: 464.44 MBytes/s over 1.00 seconds Interim result: 464.38 MBytes/s over 1.00 seconds Interim result: 464.22 MBytes/s over 1.00 seconds Interim result: 464.30 MBytes/s over 1.00 seconds Interim result: 463.98 MBytes/s over 1.00 seconds Interim result: 464.34 MBytes/s over 1.00 seconds Interim result: 463.79 MBytes/s over 1.00 seconds Interim result: 464.23 MBytes/s over 1.00 seconds Interim result: 464.40 MBytes/s over 1.00 seconds Interim result: 464.18 MBytes/s over 1.00 seconds Interim result: 464.78 MBytes/s over 1.00 seconds Interim result: 464.59 MBytes/s over 1.00 seconds Interim result: 464.71 MBytes/s over 1.00 seconds Interim result: 464.40 MBytes/s over 1.00 seconds Interim result: 463.94 MBytes/s over 1.00 seconds Interim result: 464.40 MBytes/s over 1.00 seconds Interim result: 464.39 MBytes/s over 1.00 seconds Interim result: 464.65 MBytes/s over 1.00 seconds Interim result: 464.49 MBytes/s over 1.00 seconds Interim result: 464.14 MBytes/s over 1.00 seconds Interim result: 464.45 MBytes/s over 1.00 seconds Interim result: 464.66 MBytes/s over 1.00 seconds Interim result: 464.64 MBytes/s over 1.00 seconds Interim result: 464.23 MBytes/s over 1.00 seconds Interim result: 464.20 MBytes/s over 1.00 seconds Interim result: 464.16 MBytes/s over 1.00 seconds Interim result: 463.59 MBytes/s over 1.00 seconds Interim result: 464.33 MBytes/s over 1.00 seconds Interim result: 464.07 MBytes/s over 1.00 seconds Interim result: 463.66 MBytes/s over 1.00 seconds - Connectx / Connectx msi_x = 0 # netperf -H 192.168.2.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.152 (192.168.2.152) port 0 AF_INET : demo Interim result: 627.10 MBytes/s over 1.00 seconds Interim result: 626.37 MBytes/s over 1.00 seconds Interim result: 626.76 MBytes/s over 1.00 seconds Interim result: 626.71 MBytes/s over 1.00 seconds Interim result: 626.54 MBytes/s over 1.00 seconds Interim result: 626.86 MBytes/s over 1.00 seconds Interim result: 626.68 MBytes/s over 1.00 seconds Interim result: 626.63 MBytes/s over 1.00 seconds Interim result: 627.74 MBytes/s over 1.00 seconds Interim result: 628.28 MBytes/s over 1.00 seconds Interim result: 627.92 MBytes/s over 1.00 seconds Interim result: 627.79 MBytes/s over 1.00 seconds Interim result: 628.00 MBytes/s over 1.00 seconds Interim result: 627.68 MBytes/s over 1.00 seconds Interim result: 627.32 MBytes/s over 1.00 seconds Interim result: 627.51 MBytes/s over 1.00 seconds Interim result: 627.72 MBytes/s over 1.00 seconds Interim result: 627.67 MBytes/s over 1.00 seconds Interim result: 627.39 MBytes/s over 1.00 seconds Interim result: 627.24 MBytes/s over 1.00 seconds Interim result: 627.06 MBytes/s over 1.00 seconds Interim result: 627.00 MBytes/s over 1.00 seconds Interim result: 627.24 MBytes/s over 1.00 seconds Interim result: 627.87 MBytes/s over 1.00 seconds Interim result: 622.81 MBytes/s over 1.01 seconds Interim result: 623.41 MBytes/s over 1.00 seconds Interim result: 623.24 MBytes/s over 1.00 seconds Interim result: 623.51 MBytes/s over 1.00 seconds Interim result: 623.01 MBytes/s over 1.00 seconds Interim result: 623.16 MBytes/s over 1.00 seconds Interim result: 622.87 MBytes/s over 1.00 seconds Interim result: 622.58 MBytes/s over 1.00 seconds Interim result: 623.25 MBytes/s over 1.00 seconds Interim result: 622.91 MBytes/s over 1.00 seconds Interim result: 622.96 MBytes/s over 1.00 seconds Interim result: 622.93 MBytes/s over 1.00 seconds Interim result: 622.85 MBytes/s over 1.00 seconds Interim result: 623.21 MBytes/s over 1.00 seconds Interim result: 622.83 MBytes/s over 1.00 seconds Interim result: 622.61 MBytes/s over 1.00 seconds Interim result: 622.99 MBytes/s over 1.00 seconds Interim result: 622.83 MBytes/s over 1.00 seconds Interim result: 622.68 MBytes/s over 1.00 seconds Interim result: 622.67 MBytes/s over 1.00 seconds Interim result: 622.59 MBytes/s over 1.00 seconds Interim result: 623.04 MBytes/s over 1.00 seconds Interim result: 623.28 MBytes/s over 1.00 seconds Interim result: 622.82 MBytes/s over 1.00 seconds Interim result: 622.69 MBytes/s over 1.00 seconds Interim result: 622.62 MBytes/s over 1.00 seconds Interim result: 622.47 MBytes/s over 1.00 seconds Interim result: 622.47 MBytes/s over 1.00 seconds Interim result: 623.14 MBytes/s over 1.00 seconds Interim result: 622.68 MBytes/s over 1.00 seconds Interim result: 622.61 MBytes/s over 1.00 seconds Interim result: 622.59 MBytes/s over 1.00 seconds - Arbel/Arbel msi_x = 1 # netperf -H 192.168.1.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.152 (192.168.1.152) port 0 AF_INET : demo Interim result: 457.39 MBytes/s over 1.00 seconds Interim result: 456.53 MBytes/s over 1.00 seconds Interim result: 456.67 MBytes/s over 1.00 seconds Interim result: 456.79 MBytes/s over 1.00 seconds Interim result: 456.52 MBytes/s over 1.00 seconds Interim result: 456.26 MBytes/s over 1.00 seconds Interim result: 456.78 MBytes/s over 1.00 seconds Interim result: 456.49 MBytes/s over 1.00 seconds Interim result: 456.33 MBytes/s over 1.00 seconds Interim result: 456.32 MBytes/s over 1.00 seconds Interim result: 456.75 MBytes/s over 1.00 seconds Interim result: 456.75 MBytes/s over 1.00 seconds Interim result: 457.30 MBytes/s over 1.00 seconds Interim result: 456.40 MBytes/s over 1.00 seconds Interim result: 456.60 MBytes/s over 1.00 seconds Interim result: 456.39 MBytes/s over 1.00 seconds Interim result: 456.53 MBytes/s over 1.00 seconds Interim result: 456.32 MBytes/s over 1.00 seconds Interim result: 456.13 MBytes/s over 1.00 seconds Interim result: 456.58 MBytes/s over 1.00 seconds Interim result: 456.71 MBytes/s over 1.00 seconds Interim result: 456.39 MBytes/s over 1.00 seconds Interim result: 457.01 MBytes/s over 1.00 seconds Interim result: 456.41 MBytes/s over 1.00 seconds Interim result: 445.10 MBytes/s over 1.03 seconds Interim result: 444.25 MBytes/s over 1.00 seconds Interim result: 445.65 MBytes/s over 1.00 seconds Interim result: 446.40 MBytes/s over 1.00 seconds Interim result: 445.32 MBytes/s over 1.00 seconds Interim result: 445.59 MBytes/s over 1.00 seconds Interim result: 445.71 MBytes/s over 1.00 seconds Interim result: 445.84 MBytes/s over 1.00 seconds Interim result: 445.74 MBytes/s over 1.00 seconds Interim result: 445.70 MBytes/s over 1.00 seconds Interim result: 445.22 MBytes/s over 1.00 seconds Interim result: 445.22 MBytes/s over 1.00 seconds Interim result: 445.44 MBytes/s over 1.00 seconds Interim result: 445.72 MBytes/s over 1.00 seconds Interim result: 445.80 MBytes/s over 1.00 seconds Interim result: 445.90 MBytes/s over 1.00 seconds Interim result: 445.87 MBytes/s over 1.00 seconds Interim result: 445.89 MBytes/s over 1.00 seconds Interim result: 445.33 MBytes/s over 1.00 seconds Interim result: 445.36 MBytes/s over 1.00 seconds Interim result: 445.83 MBytes/s over 1.00 seconds Interim result: 445.66 MBytes/s over 1.00 seconds Interim result: 445.62 MBytes/s over 1.00 seconds Interim result: 445.90 MBytes/s over 1.00 seconds Interim result: 445.84 MBytes/s over 1.00 seconds Interim result: 445.51 MBytes/s over 1.00 seconds Interim result: 446.18 MBytes/s over 1.00 seconds Interim result: 446.01 MBytes/s over 1.00 seconds Interim result: 445.87 MBytes/s over 1.00 seconds Interim result: 445.56 MBytes/s over 1.00 seconds Interim result: 446.26 MBytes/s over 1.00 seconds Interim result: 445.56 MBytes/s over 1.00 seconds Interim result: 445.89 MBytes/s over 1.00 seconds Interim result: 445.73 MBytes/s over 1.00 seconds Interim result: 445.56 MBytes/s over 1.00 seconds Interim result: 445.98 MBytes/s over 1.00 seconds Interim result: 446.15 MBytes/s over 1.00 seconds From sashak at voltaire.com Wed Jun 25 06:57:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 16:57:02 +0300 Subject: [ofa-general] [PATCH] opensm: add --version command line option Message-ID: <20080625135702.GL22159@sashak.voltaire.com> With this new '--version' option OpenSM will print its version and exit. Signed-off-by: Sasha Khapyorsky --- opensm/man/opensm.8.in | 4 ++++ opensm/opensm/main.c | 6 ++++++ 2 files changed, 10 insertions(+), 0 deletions(-) diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index d844590..964fbd3 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -5,6 +5,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA) .SH SYNOPSIS .B opensm +[\-\-version]] [\-c(ache-options)] [\-g(uid)[=]] [\-l(mc) ] [\-p(riority) ] [\-smkey ] [\-r(eassign_lids)] [\-R | \-\-routing_engine ] @@ -63,6 +64,9 @@ setup the subnet correctly. .PP .TP +\fB\-\-version\fR +Prints OpenSM version and exits. +.TP \fB\-c\fR, \fB\-\-cache-options\fR Write out a list of all tunable OpenSM parameters, including their current values from the command line diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 505ef57..248171a 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -126,6 +126,8 @@ static void show_usage(void) printf("\n------- OpenSM - Usage and options ----------------------\n"); printf("Usage: opensm [options]\n"); printf("Options:\n"); + printf("--version\n" + " Prints OpenSM version and exits.\n\n"); printf("-F , --config \n" " The name of the OpenSM config file. It has a same format\n" " as opensm.opts option cache file. When not specified\n" @@ -620,6 +622,7 @@ int main(int argc, char *argv[]) 2: optional */ const struct option long_option[] = { + {"version", 0, NULL, 12}, {"config", 1, NULL, 'F'}, {"debug", 1, NULL, 'd'}, {"guid", 1, NULL, 'g'}, @@ -696,6 +699,9 @@ int main(int argc, char *argv[]) next_option = getopt_long_only(argc, argv, short_option, long_option, NULL); switch (next_option) { + case 12: /* --version - already printed above */ + exit(0); + break; case 'F': if (config_file_done) break; -- 1.5.5.1.178.g1f811 From jean-vincent.ficet at bull.net Wed Jun 25 07:09:39 2008 From: jean-vincent.ficet at bull.net (Vincent Ficet) Date: Wed, 25 Jun 2008 16:09:39 +0200 Subject: [ofa-general] Issues with osm_state_mgr.c:__osm_state_mgr_get_remote_port_info() In-Reply-To: <20080625125134.GH22159@sashak.voltaire.com> References: <48622DDA.40509@bull.net> <20080625123357.GG22159@sashak.voltaire.com> <20080625125134.GH22159@sashak.voltaire.com> Message-ID: <486251A3.6010405@bull.net> Hi Sasha, >> mad_context.pi_context.port_guid = cl_hton64(p_physp->port_guid); >> > > cl_hnon64() is not needed. > I generated the patch [1] according to your comments. Please let me know if there is any problem. Cheers, Vincent patch [1]: diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index f15a4af..780d8db 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -202,8 +202,7 @@ __osm_state_mgr_get_remote_port_info(IN osm_sm_t * sm, mad_context.pi_context.node_guid = osm_node_get_node_guid(osm_physp_get_node_ptr(p_physp)); - mad_context.pi_context.port_guid = - cl_hton64(osm_physp_get_port_num(p_physp)); + mad_context.pi_context.port_guid = p_physp->port_guid; mad_context.pi_context.set_method = FALSE; mad_context.pi_context.light_sweep = TRUE; mad_context.pi_context.update_master_sm_base_lid = FALSE; From vlad at dev.mellanox.co.il Wed Jun 25 07:11:53 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Wed, 25 Jun 2008 17:11:53 +0300 Subject: [ofa-general] Re: [PATCH v2 for-2.6.27] IPOIB: add LRO support. In-Reply-To: References: <48611FEB.1030709@dev.mellanox.co.il> Message-ID: <1e336d250806250711l320ec086i8457b4a19cd7102e@mail.gmail.com> On Tue, Jun 24, 2008 at 11:56 PM, Roland Dreier wrote: > To be honest this looks like a step backwards... > > > +static ssize_t show_use_lro(struct device *d, struct device_attribute > *attr, > > + char *buf) > > +{ > > + struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(d)); > > + > > + if (priv->lro.use_lro) > > + return sprintf(buf, "yes\n"); > > + else > > + return sprintf(buf, "no\n"); > > +} > > This looks like a crazy interface ("yes"/"no" vs 1/0??) and in any case > I think using the existing ethtool support to control LRO would be much > simpler and better for users. > > - R. > > I can change "yes" to "1" and "no" to "0". I don't see that there is LRO support in the userspace ethtool. Regards, VLadimir -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sagir at mellanox.co.il Wed Jun 25 07:16:17 2008 From: Sagir at mellanox.co.il (Sagi Rotem) Date: Wed, 25 Jun 2008 17:16:17 +0300 Subject: [ofa-general] RE: msi-x seems to cause large performance variation for connectx / datagram mode In-Reply-To: Message-ID: <40FA0A8088E8A441973D37502F00933E257FA0@mtlexch01.mtl.com> Do you get the same variance with the irqdaemon off ? How does the interrupts on both cases spread between the cores ? Sagi -----Original Message----- From: Or Gerlitz [mailto:ogerlitz at voltaire.com] Sent: Wednesday, June 25, 2008 4:56 PM To: Roland Dreier; Eli Cohen; Sagi Rotem Cc: general at lists.openfabrics.org Subject: msi-x seems to cause large performance variation for connectx / datagram mode While doing a synthetic network benchmark (netperf) of 64K message size, on two nodes each with two HCAs: Arbel & ConnectX (where the idea was to have an apples-to-apples comparison), I saw a nice advatange with connectx, but there is kind of large variation in the performance it gives. The system spec is: RH5 x86_64 2.6.18-8.el5 SMP, 4GB RAM, Intel 1.6GHz four CPUs (or two CPUS and two cores each, I don't know) - I used datagram mode, where connectx has checksum and LSO offloads - the code used is not the mainline kernel but rather git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel commit 39e1dc833f98e5134f91fcf7f33df402adf4bc0c digging a little more, I see now that if I disable msi-x, the performance gets to a fixed value of about 620MB/s with connectx which is the lower range that msi-x gives me... HCA FW MB/s msi_x ============================================ Arbel-Arbel 4.8.2 450 0 (ib_mthca) Arbel-Arbel 4.8.2 450 1 (ib_mthca) ConnectX-ConnectX 2.3 620-850 1 (mlx4_core) ConnectX-ConnectX 2.3 620 0 (mlx4_core) Any idea if there's something in the system settings that can explain this? I have pasted below some lspci info and netperf results, it might help. Note that its FW 2.3 and not 2.5 on which I see other issues... Or. # modprobe mlx4_core msi_x=0 # lspci -v | grep -A 15 634a 03:00.0 InfiniBand: Mellanox Technologies Unknown device 634a (rev a0) Subsystem: Mellanox Technologies Unknown device 634a Flags: bus master, fast devsel, latency 0, IRQ 169 Memory at c8300000 (64-bit, non-prefetchable) [size=1M] Memory at c9800000 (64-bit, prefetchable) [size=8M] Memory at c8200000 (64-bit, non-prefetchable) [size=8K] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable- Mask- TabSize=256 Capabilities: [60] Express Endpoint IRQ 0 # modprobe mlx4_core msi_x=1 # lspci -v | grep -A 10 634a 03:00.0 InfiniBand: Mellanox Technologies Unknown device 634a (rev a0) Subsystem: Mellanox Technologies Unknown device 634a Flags: bus master, fast devsel, latency 0, IRQ 169 Memory at c8300000 (64-bit, non-prefetchable) [size=1M] Memory at c9800000 (64-bit, prefetchable) [size=8M] Memory at c8200000 (64-bit, non-prefetchable) [size=8K] Capabilities: [40] Power Management version 3 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256 Capabilities: [60] Express Endpoint IRQ 0 - Connectx / Connectx msi_x = 1 # netperf -H 192.168.2.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.152 (192.168.2.152) port 0 AF_INET : demo Interim result: 648.10 MBytes/s over 1.02 seconds Interim result: 630.23 MBytes/s over 1.03 seconds Interim result: 629.91 MBytes/s over 1.00 seconds Interim result: 630.24 MBytes/s over 1.00 seconds Interim result: 630.08 MBytes/s over 1.00 seconds Interim result: 630.40 MBytes/s over 1.00 seconds Interim result: 630.19 MBytes/s over 1.00 seconds Interim result: 629.88 MBytes/s over 1.00 seconds Interim result: 630.18 MBytes/s over 1.00 seconds Interim result: 630.17 MBytes/s over 1.00 seconds Interim result: 646.35 MBytes/s over 1.00 seconds Interim result: 659.88 MBytes/s over 1.00 seconds Interim result: 662.61 MBytes/s over 1.00 seconds Interim result: 663.06 MBytes/s over 1.00 seconds Interim result: 662.32 MBytes/s over 1.00 seconds Interim result: 661.93 MBytes/s over 1.00 seconds Interim result: 662.78 MBytes/s over 1.00 seconds Interim result: 662.59 MBytes/s over 1.00 seconds Interim result: 664.89 MBytes/s over 1.00 seconds Interim result: 865.67 MBytes/s over 1.00 seconds Interim result: 726.10 MBytes/s over 1.19 seconds Interim result: 623.57 MBytes/s over 1.16 seconds Interim result: 623.10 MBytes/s over 1.00 seconds Interim result: 622.99 MBytes/s over 1.00 seconds Interim result: 623.47 MBytes/s over 1.00 seconds Interim result: 622.95 MBytes/s over 1.00 seconds Interim result: 623.52 MBytes/s over 1.00 seconds Interim result: 623.26 MBytes/s over 1.00 seconds Interim result: 623.06 MBytes/s over 1.00 seconds Interim result: 623.40 MBytes/s over 1.00 seconds Interim result: 825.80 MBytes/s over 1.00 seconds Interim result: 862.10 MBytes/s over 1.00 seconds Interim result: 862.41 MBytes/s over 1.00 seconds Interim result: 862.98 MBytes/s over 1.00 seconds Interim result: 862.83 MBytes/s over 1.00 seconds Interim result: 862.66 MBytes/s over 1.00 seconds Interim result: 862.08 MBytes/s over 1.00 seconds Interim result: 861.96 MBytes/s over 1.00 seconds Interim result: 861.52 MBytes/s over 1.00 seconds Interim result: 861.32 MBytes/s over 1.00 seconds Interim result: 650.12 MBytes/s over 1.32 seconds Interim result: 623.67 MBytes/s over 1.04 seconds Interim result: 623.60 MBytes/s over 1.00 seconds Interim result: 623.16 MBytes/s over 1.00 seconds Interim result: 623.38 MBytes/s over 1.00 seconds Interim result: 623.34 MBytes/s over 1.00 seconds Interim result: 622.93 MBytes/s over 1.00 seconds Interim result: 622.67 MBytes/s over 1.00 seconds Interim result: 623.09 MBytes/s over 1.00 seconds Interim result: 674.60 MBytes/s over 1.00 seconds Interim result: 862.36 MBytes/s over 1.00 seconds Interim result: 862.32 MBytes/s over 1.00 seconds Interim result: 862.60 MBytes/s over 1.00 seconds Interim result: 862.77 MBytes/s over 1.00 seconds Interim result: 862.42 MBytes/s over 1.00 seconds Interim result: 862.97 MBytes/s over 1.00 seconds Interim result: 862.56 MBytes/s over 1.00 seconds Interim result: 863.27 MBytes/s over 1.00 seconds Interim result: 862.66 MBytes/s over 1.00 seconds Interim result: 794.91 MBytes/s over 1.09 seconds Interim result: 622.52 MBytes/s over 1.28 seconds Interim result: 622.43 MBytes/s over 1.00 seconds Interim result: 622.68 MBytes/s over 1.00 seconds Interim result: 622.51 MBytes/s over 1.00 seconds Interim result: 622.34 MBytes/s over 1.00 seconds Interim result: 622.11 MBytes/s over 1.00 seconds Interim result: 622.11 MBytes/s over 1.00 seconds Interim result: 621.71 MBytes/s over 1.00 seconds Interim result: 621.85 MBytes/s over 1.00 seconds Interim result: 761.27 MBytes/s over 1.00 seconds Interim result: 861.58 MBytes/s over 1.00 seconds Interim result: 861.76 MBytes/s over 1.00 seconds Interim result: 861.10 MBytes/s over 1.00 seconds Interim result: 862.02 MBytes/s over 1.00 seconds Interim result: 861.63 MBytes/s over 1.00 seconds - Arbel/Arbel msi_x = 0 # netperf -H 192.168.1.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.152 (192.168.1.152) port 0 AF_INET : demo Interim result: 464.73 MBytes/s over 1.00 seconds Interim result: 462.77 MBytes/s over 1.00 seconds Interim result: 463.02 MBytes/s over 1.00 seconds Interim result: 463.07 MBytes/s over 1.00 seconds Interim result: 462.93 MBytes/s over 1.00 seconds Interim result: 462.85 MBytes/s over 1.00 seconds Interim result: 462.83 MBytes/s over 1.00 seconds Interim result: 463.21 MBytes/s over 1.00 seconds Interim result: 464.18 MBytes/s over 1.00 seconds Interim result: 464.03 MBytes/s over 1.00 seconds Interim result: 464.59 MBytes/s over 1.00 seconds Interim result: 464.19 MBytes/s over 1.00 seconds Interim result: 464.56 MBytes/s over 1.00 seconds Interim result: 464.50 MBytes/s over 1.00 seconds Interim result: 464.45 MBytes/s over 1.00 seconds Interim result: 464.44 MBytes/s over 1.00 seconds Interim result: 464.38 MBytes/s over 1.00 seconds Interim result: 464.22 MBytes/s over 1.00 seconds Interim result: 464.30 MBytes/s over 1.00 seconds Interim result: 463.98 MBytes/s over 1.00 seconds Interim result: 464.34 MBytes/s over 1.00 seconds Interim result: 463.79 MBytes/s over 1.00 seconds Interim result: 464.23 MBytes/s over 1.00 seconds Interim result: 464.40 MBytes/s over 1.00 seconds Interim result: 464.18 MBytes/s over 1.00 seconds Interim result: 464.78 MBytes/s over 1.00 seconds Interim result: 464.59 MBytes/s over 1.00 seconds Interim result: 464.71 MBytes/s over 1.00 seconds Interim result: 464.40 MBytes/s over 1.00 seconds Interim result: 463.94 MBytes/s over 1.00 seconds Interim result: 464.40 MBytes/s over 1.00 seconds Interim result: 464.39 MBytes/s over 1.00 seconds Interim result: 464.65 MBytes/s over 1.00 seconds Interim result: 464.49 MBytes/s over 1.00 seconds Interim result: 464.14 MBytes/s over 1.00 seconds Interim result: 464.45 MBytes/s over 1.00 seconds Interim result: 464.66 MBytes/s over 1.00 seconds Interim result: 464.64 MBytes/s over 1.00 seconds Interim result: 464.23 MBytes/s over 1.00 seconds Interim result: 464.20 MBytes/s over 1.00 seconds Interim result: 464.16 MBytes/s over 1.00 seconds Interim result: 463.59 MBytes/s over 1.00 seconds Interim result: 464.33 MBytes/s over 1.00 seconds Interim result: 464.07 MBytes/s over 1.00 seconds Interim result: 463.66 MBytes/s over 1.00 seconds - Connectx / Connectx msi_x = 0 # netperf -H 192.168.2.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.152 (192.168.2.152) port 0 AF_INET : demo Interim result: 627.10 MBytes/s over 1.00 seconds Interim result: 626.37 MBytes/s over 1.00 seconds Interim result: 626.76 MBytes/s over 1.00 seconds Interim result: 626.71 MBytes/s over 1.00 seconds Interim result: 626.54 MBytes/s over 1.00 seconds Interim result: 626.86 MBytes/s over 1.00 seconds Interim result: 626.68 MBytes/s over 1.00 seconds Interim result: 626.63 MBytes/s over 1.00 seconds Interim result: 627.74 MBytes/s over 1.00 seconds Interim result: 628.28 MBytes/s over 1.00 seconds Interim result: 627.92 MBytes/s over 1.00 seconds Interim result: 627.79 MBytes/s over 1.00 seconds Interim result: 628.00 MBytes/s over 1.00 seconds Interim result: 627.68 MBytes/s over 1.00 seconds Interim result: 627.32 MBytes/s over 1.00 seconds Interim result: 627.51 MBytes/s over 1.00 seconds Interim result: 627.72 MBytes/s over 1.00 seconds Interim result: 627.67 MBytes/s over 1.00 seconds Interim result: 627.39 MBytes/s over 1.00 seconds Interim result: 627.24 MBytes/s over 1.00 seconds Interim result: 627.06 MBytes/s over 1.00 seconds Interim result: 627.00 MBytes/s over 1.00 seconds Interim result: 627.24 MBytes/s over 1.00 seconds Interim result: 627.87 MBytes/s over 1.00 seconds Interim result: 622.81 MBytes/s over 1.01 seconds Interim result: 623.41 MBytes/s over 1.00 seconds Interim result: 623.24 MBytes/s over 1.00 seconds Interim result: 623.51 MBytes/s over 1.00 seconds Interim result: 623.01 MBytes/s over 1.00 seconds Interim result: 623.16 MBytes/s over 1.00 seconds Interim result: 622.87 MBytes/s over 1.00 seconds Interim result: 622.58 MBytes/s over 1.00 seconds Interim result: 623.25 MBytes/s over 1.00 seconds Interim result: 622.91 MBytes/s over 1.00 seconds Interim result: 622.96 MBytes/s over 1.00 seconds Interim result: 622.93 MBytes/s over 1.00 seconds Interim result: 622.85 MBytes/s over 1.00 seconds Interim result: 623.21 MBytes/s over 1.00 seconds Interim result: 622.83 MBytes/s over 1.00 seconds Interim result: 622.61 MBytes/s over 1.00 seconds Interim result: 622.99 MBytes/s over 1.00 seconds Interim result: 622.83 MBytes/s over 1.00 seconds Interim result: 622.68 MBytes/s over 1.00 seconds Interim result: 622.67 MBytes/s over 1.00 seconds Interim result: 622.59 MBytes/s over 1.00 seconds Interim result: 623.04 MBytes/s over 1.00 seconds Interim result: 623.28 MBytes/s over 1.00 seconds Interim result: 622.82 MBytes/s over 1.00 seconds Interim result: 622.69 MBytes/s over 1.00 seconds Interim result: 622.62 MBytes/s over 1.00 seconds Interim result: 622.47 MBytes/s over 1.00 seconds Interim result: 622.47 MBytes/s over 1.00 seconds Interim result: 623.14 MBytes/s over 1.00 seconds Interim result: 622.68 MBytes/s over 1.00 seconds Interim result: 622.61 MBytes/s over 1.00 seconds Interim result: 622.59 MBytes/s over 1.00 seconds - Arbel/Arbel msi_x = 1 # netperf -H 192.168.1.152 -D 1, -l 600 -fM -t TCP_STREAM -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.152 (192.168.1.152) port 0 AF_INET : demo Interim result: 457.39 MBytes/s over 1.00 seconds Interim result: 456.53 MBytes/s over 1.00 seconds Interim result: 456.67 MBytes/s over 1.00 seconds Interim result: 456.79 MBytes/s over 1.00 seconds Interim result: 456.52 MBytes/s over 1.00 seconds Interim result: 456.26 MBytes/s over 1.00 seconds Interim result: 456.78 MBytes/s over 1.00 seconds Interim result: 456.49 MBytes/s over 1.00 seconds Interim result: 456.33 MBytes/s over 1.00 seconds Interim result: 456.32 MBytes/s over 1.00 seconds Interim result: 456.75 MBytes/s over 1.00 seconds Interim result: 456.75 MBytes/s over 1.00 seconds Interim result: 457.30 MBytes/s over 1.00 seconds Interim result: 456.40 MBytes/s over 1.00 seconds Interim result: 456.60 MBytes/s over 1.00 seconds Interim result: 456.39 MBytes/s over 1.00 seconds Interim result: 456.53 MBytes/s over 1.00 seconds Interim result: 456.32 MBytes/s over 1.00 seconds Interim result: 456.13 MBytes/s over 1.00 seconds Interim result: 456.58 MBytes/s over 1.00 seconds Interim result: 456.71 MBytes/s over 1.00 seconds Interim result: 456.39 MBytes/s over 1.00 seconds Interim result: 457.01 MBytes/s over 1.00 seconds Interim result: 456.41 MBytes/s over 1.00 seconds Interim result: 445.10 MBytes/s over 1.03 seconds Interim result: 444.25 MBytes/s over 1.00 seconds Interim result: 445.65 MBytes/s over 1.00 seconds Interim result: 446.40 MBytes/s over 1.00 seconds Interim result: 445.32 MBytes/s over 1.00 seconds Interim result: 445.59 MBytes/s over 1.00 seconds Interim result: 445.71 MBytes/s over 1.00 seconds Interim result: 445.84 MBytes/s over 1.00 seconds Interim result: 445.74 MBytes/s over 1.00 seconds Interim result: 445.70 MBytes/s over 1.00 seconds Interim result: 445.22 MBytes/s over 1.00 seconds Interim result: 445.22 MBytes/s over 1.00 seconds Interim result: 445.44 MBytes/s over 1.00 seconds Interim result: 445.72 MBytes/s over 1.00 seconds Interim result: 445.80 MBytes/s over 1.00 seconds Interim result: 445.90 MBytes/s over 1.00 seconds Interim result: 445.87 MBytes/s over 1.00 seconds Interim result: 445.89 MBytes/s over 1.00 seconds Interim result: 445.33 MBytes/s over 1.00 seconds Interim result: 445.36 MBytes/s over 1.00 seconds Interim result: 445.83 MBytes/s over 1.00 seconds Interim result: 445.66 MBytes/s over 1.00 seconds Interim result: 445.62 MBytes/s over 1.00 seconds Interim result: 445.90 MBytes/s over 1.00 seconds Interim result: 445.84 MBytes/s over 1.00 seconds Interim result: 445.51 MBytes/s over 1.00 seconds Interim result: 446.18 MBytes/s over 1.00 seconds Interim result: 446.01 MBytes/s over 1.00 seconds Interim result: 445.87 MBytes/s over 1.00 seconds Interim result: 445.56 MBytes/s over 1.00 seconds Interim result: 446.26 MBytes/s over 1.00 seconds Interim result: 445.56 MBytes/s over 1.00 seconds Interim result: 445.89 MBytes/s over 1.00 seconds Interim result: 445.73 MBytes/s over 1.00 seconds Interim result: 445.56 MBytes/s over 1.00 seconds Interim result: 445.98 MBytes/s over 1.00 seconds Interim result: 446.15 MBytes/s over 1.00 seconds From sashak at voltaire.com Wed Jun 25 07:23:05 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 17:23:05 +0300 Subject: [ofa-general] Issues with osm_state_mgr.c:__osm_state_mgr_get_remote_port_info() In-Reply-To: <486251A3.6010405@bull.net> References: <48622DDA.40509@bull.net> <20080625123357.GG22159@sashak.voltaire.com> <20080625125134.GH22159@sashak.voltaire.com> <486251A3.6010405@bull.net> Message-ID: <20080625142305.GM22159@sashak.voltaire.com> On 16:09 Wed 25 Jun , Vincent Ficet wrote: > > diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c > index f15a4af..780d8db 100644 > --- a/opensm/opensm/osm_state_mgr.c > +++ b/opensm/opensm/osm_state_mgr.c > @@ -202,8 +202,7 @@ __osm_state_mgr_get_remote_port_info(IN osm_sm_t * sm, > > mad_context.pi_context.node_guid = > osm_node_get_node_guid(osm_physp_get_node_ptr(p_physp)); > - mad_context.pi_context.port_guid = > - cl_hton64(osm_physp_get_port_num(p_physp)); > + mad_context.pi_context.port_guid = p_physp->port_guid; > mad_context.pi_context.set_method = FALSE; > mad_context.pi_context.light_sweep = TRUE; > mad_context.pi_context.update_master_sm_base_lid = FALSE; As you can see the patch is totally broken (your mailer or cut&past does bad :( ). I applied it by hand, added log message and preserved authorship. Also next time remember to add 'Sihged-off-by:' line. Sasha commit 233b46a410f6d9d7dbd4e762f0b9c9066dcb0a4a Author: Vincent Ficet Date: Wed Jun 25 16:51:04 2008 +0300 opensm: fix wrong port_guid initialization Fix wrong port_guid field initialization when re-querying PortInfo in light sweep. The issue was pointed out by Vincent Ficet. Signed-off-by: Sasha Khapyorsky diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index f15a4af..780d8db 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -202,8 +202,7 @@ __osm_state_mgr_get_remote_port_info(IN osm_sm_t * sm, mad_context.pi_context.node_guid = osm_node_get_node_guid(osm_physp_get_node_ptr(p_physp)); - mad_context.pi_context.port_guid = - cl_hton64(osm_physp_get_port_num(p_physp)); + mad_context.pi_context.port_guid = p_physp->port_guid; mad_context.pi_context.set_method = FALSE; mad_context.pi_context.light_sweep = TRUE; mad_context.pi_context.update_master_sm_base_lid = FALSE; From hrosenstock at xsigo.com Wed Jun 25 07:48:30 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 07:48:30 -0700 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <4861E8A0.7000704@Sun.COM> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> Message-ID: <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> Hi Sumit, On Wed, 2008-06-25 at 12:11 +0530, Sumit Gaur - Sun Microsystem wrote: > Hi Hal/Sashak, > > Hal Rosenstock wrote: > > Hi Sumit, > > > > On Tue, 2008-06-24 at 14:21 +0530, Sumit Gaur - Sun Microsystem wrote: > > > >>Hi, > >>I am using OFED 2.5.* > > > > ^^^^^ > > 1.2.5.* ? > > > Sorry for typo .. it is 1.2.5.* > > > >> and observing that my SMI requests are serving very fast > >>and response time is very less on the contrary my GSI requests were served in > >>more time and response time sometime goes more than 2 sec. Any light on this > >>different behavior. > > > > > > What are the specific GS requests which are slow in response ? Are they > > compute intensive ? > I am sending only request for > > rpc.mgtclass = IB_PERFORMANCE_CLASS; > rpc.method = IB_MAD_METHOD_GET; > > at every one second. > > > > > In general, there are a few possibilities (which can cause this). SM > > traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most > > subnets). > > > > Some possibilities are: > > 1. Timeout/retry being hit for some GS traffic (GS request or response > > lost/corrupted) > Yes, this is also happening, Sometimes I am getting corrupt data back, Is there an error indicated ? > and if I retry to send same request again it fails or send corrupted data back again. > > 2. Data VL busy (is there anything else utilizing VL0 ?) > Not sure about it. Is there anything to verify it? There's an optional counter not commonly implemented so maybe starting by verifying all the PortCounters along the path from requester to responder to see whether there are any low level issues with your subnet. > I am running openSM also but > not using it.. Not sure what you mean by running it but not using it. -- Hal > > 3. CPU busy (what other things are competing for CPU on the responder ?) > CPU utilization is ok. > > and all requests are lid routed routed. > > > > > -- Hal > > > > > >>Is it expected behavior if yes then why ? > >> > >>Thanks and Regards > >>sumit > >>_______________________________________________ > >>general mailing list > >>general at lists.openfabrics.org > >>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >> > >>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From Sumit.Gaur at Sun.COM Wed Jun 25 07:51:07 2008 From: Sumit.Gaur at Sun.COM (Sumit Gaur - Sun Microsystem) Date: Wed, 25 Jun 2008 20:21:07 +0530 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> Message-ID: <48625B5B.7080207@Sun.COM> Hi Hal, Hal Rosenstock wrote: > Hi Sumit, > > On Wed, 2008-06-25 at 12:11 +0530, Sumit Gaur - Sun Microsystem wrote: > >>Hi Hal/Sashak, >> >>Hal Rosenstock wrote: >> >>>Hi Sumit, >>> >>>On Tue, 2008-06-24 at 14:21 +0530, Sumit Gaur - Sun Microsystem wrote: >>> >>> >>>>Hi, >>>>I am using OFED 2.5.* >>> >>> ^^^^^ >>> 1.2.5.* ? >>> >> >>Sorry for typo .. it is 1.2.5.* >> >>>>and observing that my SMI requests are serving very fast >>>>and response time is very less on the contrary my GSI requests were served in >>>>more time and response time sometime goes more than 2 sec. Any light on this >>>>different behavior. >>> >>> >>>What are the specific GS requests which are slow in response ? Are they >>>compute intensive ? >> >>I am sending only request for >> >> rpc.mgtclass = IB_PERFORMANCE_CLASS; >> rpc.method = IB_MAD_METHOD_GET; >> >>at every one second. >> >> >>>In general, there are a few possibilities (which can cause this). SM >>>traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most >>>subnets). >>> >>>Some possibilities are: >>>1. Timeout/retry being hit for some GS traffic (GS request or response >>>lost/corrupted) >> >>Yes, this is also happening, Sometimes I am getting corrupt data back, > > > Is there an error indicated ? For such packets I am getting umad_status as 110. > > >>and if I retry to send same request again it fails or send corrupted data back again. > > >>>2. Data VL busy (is there anything else utilizing VL0 ?) >> >>Not sure about it. Is there anything to verify it? > > > There's an optional counter not commonly implemented so maybe starting > by verifying all the PortCounters along the path from requester to > responder to see whether there are any low level issues with your > subnet. > From ogerlitz at voltaire.com Wed Jun 25 08:00:08 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 18:00:08 +0300 Subject: [ofa-general] Re: msi-x seems to cause large performance variation for connectx / datagram mode In-Reply-To: <40FA0A8088E8A441973D37502F00933E257FA0@mtlexch01.mtl.com> References: <40FA0A8088E8A441973D37502F00933E257FA0@mtlexch01.mtl.com> Message-ID: <48625D78.8070306@voltaire.com> Sagi Rotem wrote: > Do you get the same variance with the irqdaemon off ? How does the interrupts on both cases spread between the cores ? Under MSI-X, when I disable the irq daemon the mlx4 interrupts on each node go to one CPU on each node. I guess there might be an issue with the CPU affinity of the process / network stack kernel thread and the CPU affinity of the hard irq handler used by ipoib. What's your recommendation regarding the affinity of those three elements? Or. From hrosenstock at xsigo.com Wed Jun 25 08:07:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 08:07:38 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <20080624003321.GC7341@sashak.voltaire.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> <20080624003321.GC7341@sashak.voltaire.com> Message-ID: <1214406458.13056.411.camel@hrosenstock-ws.xsigo.com> On Tue, 2008-06-24 at 03:33 +0300, Sasha Khapyorsky wrote: > On 07:01 Mon 23 Jun , Hal Rosenstock wrote: > > > > Inlining is only a hint to the compiler and given the over(ab)use of > > inline in OpenSM (by my count almost 500 instances) I doubt this has the > > intended effect. > > I asked in order to understand about are there any specific reasons for > this patch or just it is "nice to have" stuff (and not to mark my > disagreement). Right now, these are nice to have but maybe are needed for future changes and trying to judge the waters in terms of inline use. I was also experimenting with some header inclusion issues I ran into. > > Are the inlines really needed in these two cases ? > > Inlining is potential optimization and as you said it is optional, so > word "needed" is not the best which describes this :) Yes, in the sense you indicate needed doesn't apply but the sense I meant are those specific uses of the inline needed but I don't want to argue semantics. > I likely fine about the first case (especially about > osm_port_prof_set_ignored_port()), but the second function really looks > as "one-line stuff" for me. By second function, I presume you are referring to osm_port_prof_is_ignored_port. I'm not sure what you mean by "one line stuff" but maybe that also applies to the other patch relating to inline (opensm/osm_switch: Don't inline osm_switch_sp0_is_lmc_capable function). -- Hal > > In general, I think OpenSM needs a more careful look as to what really > > needs inlining. > > Sure. Agreed here. > > > > Would be nice to not mix in one patch. > > > > Already done on list. > > Thanks. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Wed Jun 25 08:16:59 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 08:16:59 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes Message-ID: <1214407019.13056.415.camel@hrosenstock-ws.xsigo.com> opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index 0376686..ec45c3a 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -114,7 +114,6 @@ static inline void osm_port_prof_construct(IN osm_port_profile_t * const p_prof) CL_ASSERT(p_prof); memset(p_prof, 0, sizeof(*p_prof)); } - /* * PARAMETERS * p_prof @@ -144,7 +143,6 @@ osm_port_prof_path_count_inc(IN osm_port_profile_t * const p_prof) CL_ASSERT(p_prof); p_prof->num_paths++; } - /* * PARAMETERS * p_prof @@ -172,7 +170,6 @@ osm_port_prof_path_count_get(IN const osm_port_profile_t * const p_prof) { return (p_prof->num_paths); } - /* * PARAMETERS * p_prof @@ -211,7 +208,6 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, } return FALSE; } - /* * PARAMETERS * p_subn @@ -256,7 +252,6 @@ osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, value = value | (1 << port_num); cl_map_insert(p_map, port_guid, (void *)value); } - /* * PARAMETERS * p_subn From eli at dev.mellanox.co.il Wed Jun 25 08:29:12 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Wed, 25 Jun 2008 18:29:12 +0300 Subject: [ofa-general] [PATCH v2] net/inet_lro: remove setting skb->ip_summed when not LRO-able Message-ID: <1214407752.23583.69.camel@mtls03> >From 9c5223b8574f364629de4cc8607d97b32f742a7c Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Tue, 24 Jun 2008 17:17:14 +0300 Subject: [PATCH] net/inet_lro: remove setting skb->ip_summed when not LRO-able When an SKB cannot be chained to a session, the current code attempts to "restore" its ip_summed field from lro_mgr->ip_summed. However, lro_mgr->ip_summed does not hold the original value; in fact, we'd better not touch skb->ip_summed since it is not modified by the code in the path leading to a failure to chain it. Also use a cleaer comment to the describe the ip_summed field of struct net_lro_mgr. Issue raised by Or Gerlitz Signed-off-by: Eli Cohen --- Changes since previous post: Updated comment on inet_lro.h for more clarity. include/linux/inet_lro.h | 6 +++++- net/ipv4/inet_lro.c | 3 +-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/inet_lro.h b/include/linux/inet_lro.h index 80335b7..c4335fa 100644 --- a/include/linux/inet_lro.h +++ b/include/linux/inet_lro.h @@ -84,7 +84,11 @@ struct net_lro_mgr { from received packets and eth protocol is still ETH_P_8021Q */ - u32 ip_summed; /* Set in non generated SKBs in page mode */ + /* + * Set for generated SKBs that are not added to + * the frag list in fragmented mode + */ + u32 ip_summed; u32 ip_summed_aggr; /* Set in aggregated SKBs: CHECKSUM_UNNECESSARY * or CHECKSUM_NONE */ diff --git a/net/ipv4/inet_lro.c b/net/ipv4/inet_lro.c index 4a4d49f..cfd034a 100644 --- a/net/ipv4/inet_lro.c +++ b/net/ipv4/inet_lro.c @@ -383,8 +383,7 @@ static int __lro_proc_skb(struct net_lro_mgr *lro_mgr, struct sk_buff *skb, out2: /* send aggregated SKBs to stack */ lro_flush(lro_mgr, lro_desc); -out: /* Original SKB has to be posted to stack */ - skb->ip_summed = lro_mgr->ip_summed; +out: return 1; } -- 1.5.6 From ogerlitz at voltaire.com Wed Jun 25 08:33:36 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 25 Jun 2008 18:33:36 +0300 (IDT) Subject: [ofa-general] Re: msi-x seems to cause large performance variation for connectx / datagram mode In-Reply-To: References: Message-ID: > digging a little more, I see now that if I disable msi-x, the > performance gets to a fixed value of about 620MB/s with connectx > which is the lower range that msi-x gives me... setting the irq daemon off, I see that if I set the cpu affinity of the netperf process to be as the one of the mlx4 irq I get fixed performance of 650 MB/s If I set it to different CPU, then eventually I get 480MB/s constantly and 620MB/s constantly if is a different core on the same CPU... HCA MB/s msi_x irqdaemon netperf affinity ======================================================================= ConnectX-ConnectX 620 0 on none ConnectX-ConnectX 620-850 1 on none ConnectX-ConnectX 650 1 off same as irq ConnectX-ConnectX 620 1 off different core ConnectX-ConnectX 480 1 off different cpu Or. From hrosenstock at xsigo.com Wed Jun 25 08:55:31 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 08:55:31 -0700 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <48625B5B.7080207@Sun.COM> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> Message-ID: <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> Hi Sumit, On Wed, 2008-06-25 at 20:21 +0530, Sumit Gaur - Sun Microsystem wrote: > Hi Hal, > > Hal Rosenstock wrote: > > Hi Sumit, > > > > On Wed, 2008-06-25 at 12:11 +0530, Sumit Gaur - Sun Microsystem wrote: > > > >>Hi Hal/Sashak, > >> > >>Hal Rosenstock wrote: > >> > >>>Hi Sumit, > >>> > >>>On Tue, 2008-06-24 at 14:21 +0530, Sumit Gaur - Sun Microsystem wrote: > >>> > >>> > >>>>Hi, > >>>>I am using OFED 2.5.* > >>> > >>> ^^^^^ > >>> 1.2.5.* ? > >>> > >> > >>Sorry for typo .. it is 1.2.5.* > >> > >>>>and observing that my SMI requests are serving very fast > >>>>and response time is very less on the contrary my GSI requests were served in > >>>>more time and response time sometime goes more than 2 sec. Any light on this > >>>>different behavior. > >>> > >>> > >>>What are the specific GS requests which are slow in response ? Are they > >>>compute intensive ? > >> > >>I am sending only request for > >> > >> rpc.mgtclass = IB_PERFORMANCE_CLASS; > >> rpc.method = IB_MAD_METHOD_GET; > >> > >>at every one second. Does perfquery work reliably with the same node(s) you are having trouble with ? Does your app follow what perfquery does ? > >>>In general, there are a few possibilities (which can cause this). SM > >>>traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most > >>>subnets). > >>> > >>>Some possibilities are: > >>>1. Timeout/retry being hit for some GS traffic (GS request or response > >>>lost/corrupted) > >> > >>Yes, this is also happening, Sometimes I am getting corrupt data back, > > > > > > Is there an error indicated ? > For such packets I am getting umad_status as 110. That's ETIMEDOUT. You need to handle the errors (and not treat the receive as a valid packet). Are you doing that ? The underlying question is why are you getting the timeout relatively frequently so I recommend checking all the error counters along the path. Are you sure the request gets to the responder ? Does the responder respond and it doesn't make it back ? -- Hal > >>and if I retry to send same request again it fails or send corrupted data back again. > > > > > >>>2. Data VL busy (is there anything else utilizing VL0 ?) > >> > >>Not sure about it. Is there anything to verify it? > > > > > > There's an optional counter not commonly implemented so maybe starting > > by verifying all the PortCounters along the path from requester to > > responder to see whether there are any low level issues with your > > subnet. > > From vlad at mellanox.co.il Wed Jun 25 09:05:46 2008 From: vlad at mellanox.co.il (Vladimir Sokolovsky) Date: Wed, 25 Jun 2008 19:05:46 +0300 Subject: [ofa-general] [PATCH v3 for-2.6.27] IPOIB: add LRO support. Message-ID: <20080625160546.GC31769@mellanox.co.il> >From 018f224b65cdde5280d59756090e78d3e1a0bb9c Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Mon, 23 Jun 2008 15:38:27 +0300 Subject: [PATCH] IPOIB: add LRO support. add "ipoib_use_lro" module parameter to enable LRO. add "ipoib_lro_max_aggr" module parameter to set the Max number of packets to be aggregated. LRO statistics accessible through ethtool. Signed-off-by: Eli Cohen Signed-off-by: Vladimir Sokolovsky --- Changes from v2: Set/unset LRO using ethtool instead of sysfs. drivers/infiniband/ulp/ipoib/Kconfig | 1 + drivers/infiniband/ulp/ipoib/ipoib.h | 13 ++++++ drivers/infiniband/ulp/ipoib/ipoib_ethtool.c | 52 +++++++++++++++++++++++ drivers/infiniband/ulp/ipoib/ipoib_ib.c | 8 +++- drivers/infiniband/ulp/ipoib/ipoib_main.c | 58 ++++++++++++++++++++++++++ 5 files changed, 131 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/Kconfig b/drivers/infiniband/ulp/ipoib/Kconfig index 1f76bad..691525c 100644 --- a/drivers/infiniband/ulp/ipoib/Kconfig +++ b/drivers/infiniband/ulp/ipoib/Kconfig @@ -1,6 +1,7 @@ config INFINIBAND_IPOIB tristate "IP-over-InfiniBand" depends on NETDEVICES && INET && (IPV6 || IPV6=n) + select INET_LRO ---help--- Support for the IP-over-InfiniBand protocol (IPoIB). This transports IP packets over InfiniBand so you can use your IB diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 8754b36..985ee87 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -50,6 +50,7 @@ #include #include #include +#include /* constants */ @@ -94,6 +95,9 @@ enum { IPOIB_MCAST_FLAG_BUSY = 2, /* joining or already joined */ IPOIB_MCAST_FLAG_ATTACHED = 3, + IPOIB_MAX_LRO_DESCRIPTORS = 8, + IPOIB_LRO_MAX_AGGR = 64, + MAX_SEND_CQE = 16, IPOIB_CM_COPYBREAK = 256, }; @@ -248,6 +252,11 @@ struct ipoib_ethtool_st { u16 max_coalesced_frames; }; +struct ipoib_lro { + struct net_lro_mgr lro_mgr; + struct net_lro_desc lro_desc[IPOIB_MAX_LRO_DESCRIPTORS]; +}; + /* * Device private locking: tx_lock protects members used in TX fast * path (and we use LLTX so upper layers don't do extra locking). @@ -334,6 +343,8 @@ struct ipoib_dev_priv { int hca_caps; struct ipoib_ethtool_st ethtool; struct timer_list poll_timer; + + struct ipoib_lro lro; }; struct ipoib_ah { @@ -679,6 +690,8 @@ extern struct ib_sa_client ipoib_sa_client; #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG extern int ipoib_debug_level; +extern int ipoib_use_lro; +extern int ipoib_lro_max_aggr; #define ipoib_dbg(priv, format, arg...) \ do { \ diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c index 10279b7..86cf3e7 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c @@ -86,11 +86,63 @@ static int ipoib_set_coalesce(struct net_device *dev, return 0; } +#define IPOIB_STATS_LEN ARRAY_SIZE(ipoib_gstrings_stats) + +static const char ipoib_gstrings_stats[][ETH_GSTRING_LEN] = { + "LRO aggregated", "LRO flushed", + "LRO avg aggr", "LRO no_desc" +}; + +static void +ipoib_get_strings(struct net_device *netdev, u32 stringset, u8 *data) +{ + switch (stringset) { + case ETH_SS_STATS: + memcpy(data, *ipoib_gstrings_stats, + sizeof(ipoib_gstrings_stats)); + data += sizeof(ipoib_gstrings_stats); + break; + } +} + +static int ipoib_get_sset_count(struct net_device *dev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return IPOIB_STATS_LEN; + default: + return -EOPNOTSUPP; + } +} + +static void ipoib_get_ethtool_stats(struct net_device *dev, + struct ethtool_stats *stats, uint64_t *data) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + int index = 0; + + /* Get LRO statistics */ + data[index++] = priv->lro.lro_mgr.stats.aggregated; + data[index++] = priv->lro.lro_mgr.stats.flushed; + if (priv->lro.lro_mgr.stats.flushed) + data[index++] = priv->lro.lro_mgr.stats.aggregated / + priv->lro.lro_mgr.stats.flushed; + else + data[index++] = 0; + data[index++] = priv->lro.lro_mgr.stats.no_desc; + +} + static const struct ethtool_ops ipoib_ethtool_ops = { .get_drvinfo = ipoib_get_drvinfo, .get_tso = ethtool_op_get_tso, .get_coalesce = ipoib_get_coalesce, .set_coalesce = ipoib_set_coalesce, + .get_flags = ethtool_op_get_flags, + .set_flags = ethtool_op_set_flags, + .get_strings = ipoib_get_strings, + .get_sset_count = ipoib_get_sset_count, + .get_ethtool_stats = ipoib_get_ethtool_stats, }; void ipoib_set_ethtool_ops(struct net_device *dev) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index eca8518..8bd7db8 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -288,7 +288,10 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) if (test_bit(IPOIB_FLAG_CSUM, &priv->flags) && likely(wc->csum_ok)) skb->ip_summed = CHECKSUM_UNNECESSARY; - netif_receive_skb(skb); + if (dev->features & NETIF_F_LRO) + lro_receive_skb(&priv->lro.lro_mgr, skb, 0); + else + netif_receive_skb(skb); repost: if (unlikely(ipoib_ib_post_receive(dev, wr_id))) @@ -448,6 +451,9 @@ poll_more: goto poll_more; } + if (dev->features & NETIF_F_LRO) + lro_flush_all(&priv->lro.lro_mgr); + return done; } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index bfe1dbf..ff8608b 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -60,6 +60,17 @@ MODULE_PARM_DESC(send_queue_size, "Number of descriptors in send queue"); module_param_named(recv_queue_size, ipoib_recvq_size, int, 0444); MODULE_PARM_DESC(recv_queue_size, "Number of descriptors in receive queue"); +int ipoib_use_lro __read_mostly; +module_param_named(ipoib_use_lro, ipoib_use_lro, int, 0444); +MODULE_PARM_DESC(ipoib_use_lro, " Large Receive Offload, 1: enable, " + "0: disable, Default = 0"); + +int ipoib_lro_max_aggr __read_mostly = IPOIB_LRO_MAX_AGGR; +module_param_named(ipoib_lro_max_aggr, ipoib_lro_max_aggr, int, 0644); +MODULE_PARM_DESC(ipoib_lro_max_aggr, " LRO: Max packets to be aggregated. " + "Default = 64"); + + #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG int ipoib_debug_level; @@ -936,6 +947,49 @@ static const struct header_ops ipoib_header_ops = { .create = ipoib_hard_header, }; +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, + void **tcph, u64 *hdr_flags, void *priv) +{ + unsigned int ip_len; + struct iphdr *iph; + + if (unlikely(skb->protocol != htons(ETH_P_IP))) + return -1; + + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) + return -1; + + /* non tcp packet */ + skb_reset_network_header(skb); + iph = ip_hdr(skb); + if (iph->protocol != IPPROTO_TCP) + return -1; + + ip_len = ip_hdrlen(skb); + skb_set_transport_header(skb, ip_len); + *tcph = tcp_hdr(skb); + + /* check if ip header and tcp header are complete */ + if (iph->tot_len < ip_len + tcp_hdrlen(skb)) + return -1; + + *hdr_flags = LRO_IPV4 | LRO_TCP; + *iphdr = iph; + + return 0; +} + +static void ipoib_lro_setup(struct ipoib_dev_priv *priv) +{ + priv->lro.lro_mgr.max_aggr = ipoib_lro_max_aggr; + priv->lro.lro_mgr.max_desc = IPOIB_MAX_LRO_DESCRIPTORS; + priv->lro.lro_mgr.lro_arr = priv->lro.lro_desc; + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; + priv->lro.lro_mgr.features = LRO_F_NAPI; + priv->lro.lro_mgr.dev = priv->dev; + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; +} + static void ipoib_setup(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -975,6 +1029,8 @@ static void ipoib_setup(struct net_device *dev) priv->dev = dev; + ipoib_lro_setup(priv); + spin_lock_init(&priv->lock); spin_lock_init(&priv->tx_lock); @@ -1152,6 +1208,8 @@ static struct net_device *ipoib_add_port(const char *format, priv->dev->features |= NETIF_F_SG | NETIF_F_IP_CSUM; } + if (ipoib_use_lro) + priv->dev->features |= NETIF_F_LRO; /* * Set the full membership bit, so that we join the right * broadcast group, etc. -- 1.5.4.3 From sashak at voltaire.com Wed Jun 25 09:55:34 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 19:55:34 +0300 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <4861E87C.3000007@dev.mellanox.co.il> References: <4861E87C.3000007@dev.mellanox.co.il> Message-ID: <20080625165534.GS22159@sashak.voltaire.com> Hi Yevgeny, On 09:41 Wed 25 Jun , Yevgeny Kliteynik wrote: > > Fixing some data types and corresponding log messages, > thus solving compiler warnings. > > Signed-off-by: Yevgeny Kliteynik > --- > opensm/opensm/osm_state_mgr.c | 9 +++++---- > 1 files changed, 5 insertions(+), 4 deletions(-) > > diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c > index f15a4af..6ca276b 100644 > --- a/opensm/opensm/osm_state_mgr.c > +++ b/opensm/opensm/osm_state_mgr.c > @@ -863,7 +863,8 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) > osm_port_t *p_port; > osm_port_t *p_next_port; > cl_ptr_vector_t *p_port_lid_tbl; > - size_t max_lid, ref_size, curr_size, lid; > + size_t ref_size, curr_size; > + uint16_t max_lid, lid; Finally it stores ref_size or cur_size values in max_lid, so it can overflow. Could you be more specific about compiler warnings? > osm_port_t *p_port_ref, *p_port_stored; > cl_ptr_vector_t ref_port_lid_tbl; > uint16_t min_lid_ho; > @@ -921,7 +922,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) > * with the new lid we wanted to give it in our > * port_lid_tbl. */ > OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3322: " > - "lid %u is wrongly assigned to port 0x%016" > + "lid 0x%04x is wrongly assigned to port 0x%016" It was discussed on the list (and even marked as a bug in bugzilla) that unicast LIDs should be represented in decimal form in logs. Sasha From sashak at voltaire.com Wed Jun 25 09:58:46 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 19:58:46 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes In-Reply-To: <1214407019.13056.415.camel@hrosenstock-ws.xsigo.com> References: <1214407019.13056.415.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080625165846.GT22159@sashak.voltaire.com> On 08:16 Wed 25 Jun , Hal Rosenstock wrote: > opensm/include/opensm/osm_port_profile.h: Cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hrosenstock at xsigo.com Wed Jun 25 10:09:38 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:09:38 -0700 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <20080625165534.GS22159@sashak.voltaire.com> References: <4861E87C.3000007@dev.mellanox.co.il> <20080625165534.GS22159@sashak.voltaire.com> Message-ID: <1214413778.13056.448.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 19:55 +0300, Sasha Khapyorsky wrote: > It was discussed on the list (and even marked as a bug in bugzilla) > that unicast LIDs should be represented in decimal form in logs. Are all the other unicast LID print instances in OpenSM being changed too ? I think hex consistency is better than a piecemeal approach. -- Hal From parhizj at ufl.edu Wed Jun 25 10:12:09 2008 From: parhizj at ufl.edu (John Parhizgari) Date: Wed, 25 Jun 2008 13:12:09 -0400 Subject: [ofa-general] PM MAD tools Message-ID: <48627C69.5000606@ufl.edu> Are there any other tools like IBIS but that have more commands for the PM MADS, specifically for setting/getting attributes like the PortSamplesControl and PortRcvErrorDetails mainly for the purpose of getting the information from the counters that make up the PortRcvError counter, such as PortLocalPhysicalErrors (this would allow a substantial amount of more useful error information) From sashak at voltaire.com Wed Jun 25 10:12:01 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 20:12:01 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <1214406458.13056.411.camel@hrosenstock-ws.xsigo.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> <20080624003321.GC7341@sashak.voltaire.com> <1214406458.13056.411.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080625171201.GU22159@sashak.voltaire.com> On 08:07 Wed 25 Jun , Hal Rosenstock wrote: > > Right now, these are nice to have but maybe are needed for future > changes and trying to judge the waters in terms of inline use. I was > also experimenting with some header inclusion issues I ran into. Like what? > > I likely fine about the first case (especially about > > osm_port_prof_set_ignored_port()), but the second function really looks > > as "one-line stuff" for me. > > By second function, I presume you are referring to > osm_port_prof_is_ignored_port. I'm not sure what you mean by "one line > stuff" but maybe that also applies to the other patch relating to inline > (opensm/osm_switch: Don't inline osm_switch_sp0_is_lmc_capable > function). I was about second patch, but osm_port_prof_is_ignored_port() is pretty short too. I would not bother and leave it to compiler to decide. Sasha From hrosenstock at xsigo.com Wed Jun 25 10:25:23 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:25:23 -0700 Subject: [ofa-general] PM MAD tools In-Reply-To: <48627C69.5000606@ufl.edu> References: <48627C69.5000606@ufl.edu> Message-ID: <1214414723.13056.450.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 13:12 -0400, John Parhizgari wrote: > Are there any other tools like IBIS but that have more commands for the > PM MADS, > specifically for setting/getting attributes like the PortSamplesControl > and PortRcvErrorDetails > > mainly for the purpose of getting the information from the counters that > make up the PortRcvError counter, such as PortLocalPhysicalErrors (this > would allow a substantial amount of more useful error information) libibmad has partial support for these attributes/operations. It needs work in those areas but could be added there. Another approach would be to add support for this via perfmgr with console support. -- Hal > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Wed Jun 25 10:30:10 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:30:10 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <20080625171201.GU22159@sashak.voltaire.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> <20080624003321.GC7341@sashak.voltaire.com> <1214406458.13056.411.camel@hrosenstock-ws.xsigo.com> <20080625171201.GU22159@sashak.voltaire.com> Message-ID: <1214415010.13056.456.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 20:12 +0300, Sasha Khapyorsky wrote: > On 08:07 Wed 25 Jun , Hal Rosenstock wrote: > > > > Right now, these are nice to have but maybe are needed for future > > changes and trying to judge the waters in terms of inline use. I was > > also experimenting with some header inclusion issues I ran into. > > Like what? In an early approach to adding the MLID array, I had tried including osm_multicast.h in osm_subnet.h and started dealing with the compile issues. -- Hal > > > I likely fine about the first case (especially about > > > osm_port_prof_set_ignored_port()), but the second function really looks > > > as "one-line stuff" for me. > > > > By second function, I presume you are referring to > > osm_port_prof_is_ignored_port. I'm not sure what you mean by "one line > > stuff" but maybe that also applies to the other patch relating to inline > > (opensm/osm_switch: Don't inline osm_switch_sp0_is_lmc_capable > > function). > > I was about second patch, but osm_port_prof_is_ignored_port() is pretty > short too. I would not bother and leave it to compiler to decide. > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From parhizj at ufl.edu Wed Jun 25 10:30:51 2008 From: parhizj at ufl.edu (John Parhizgari) Date: Wed, 25 Jun 2008 13:30:51 -0400 Subject: [ofa-general] PM MAD tools In-Reply-To: <1214414723.13056.450.camel@hrosenstock-ws.xsigo.com> References: <48627C69.5000606@ufl.edu> <1214414723.13056.450.camel@hrosenstock-ws.xsigo.com> Message-ID: <486280CB.1070408@ufl.edu> Hal Rosenstock wrote: > Another approach would be to add support for this via perfmgr with > console support. > Is there a way to guarantee opensm will enter Standby state? I don't want it to mess up the fabric by accident since another SM is already running (I don't have the socket option enabled). From hrosenstock at xsigo.com Wed Jun 25 10:43:41 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:43:41 -0700 Subject: [ofa-general] PM MAD tools In-Reply-To: <486280CB.1070408@ufl.edu> References: <48627C69.5000606@ufl.edu> <1214414723.13056.450.camel@hrosenstock-ws.xsigo.com> <486280CB.1070408@ufl.edu> Message-ID: <1214415821.13056.472.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 13:30 -0400, John Parhizgari wrote: > Hal Rosenstock wrote: > > Another approach would be to add support for this via perfmgr with > > console support. > > > Is there a way to guarantee opensm will enter Standby state? I don't > want it to mess up the fabric by accident since another SM is already > running (I don't have the socket option enabled). I think you can just run perfmgr but not sure you can run the console currently. That might need changing so there's a perfmgr only console. -- Hal > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Wed Jun 25 10:47:33 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 20:47:33 +0300 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <1214413778.13056.448.camel@hrosenstock-ws.xsigo.com> References: <4861E87C.3000007@dev.mellanox.co.il> <20080625165534.GS22159@sashak.voltaire.com> <1214413778.13056.448.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080625174733.GV22159@sashak.voltaire.com> On 10:09 Wed 25 Jun , Hal Rosenstock wrote: > On Wed, 2008-06-25 at 19:55 +0300, Sasha Khapyorsky wrote: > > It was discussed on the list (and even marked as a bug in bugzilla) > > that unicast LIDs should be represented in decimal form in logs. > > Are all the other unicast LID print instances in OpenSM being changed > too ? No, and now it is good time to do. > I think hex consistency is better than a piecemeal approach. I'm fine with both approaches - do all at once or step by step. In any case convert to "deprecated" format doesn't seem reasonable for me. Sasha From hrosenstock at xsigo.com Wed Jun 25 10:49:28 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:49:28 -0700 Subject: [ofa-general] PM MAD tools In-Reply-To: <1214415821.13056.472.camel@hrosenstock-ws.xsigo.com> References: <48627C69.5000606@ufl.edu> <1214414723.13056.450.camel@hrosenstock-ws.xsigo.com> <486280CB.1070408@ufl.edu> <1214415821.13056.472.camel@hrosenstock-ws.xsigo.com> Message-ID: <1214416168.13056.476.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 10:43 -0700, Hal Rosenstock wrote: > > Is there a way to guarantee opensm will enter Standby state? > I don't > > want it to mess up the fabric by accident since another SM is > already > > running (I don't have the socket option enabled). > > I think you can just run perfmgr but not sure you can run the console > currently. That might need changing so there's a perfmgr only console. If you're also not interested in perfmgr (as well as not OpenSM), then the libibmad/diag tools approach might be the way to go. Are you just looking for more directed queries under admin control or something that automatically checks the subnet ? -- Hal From hrosenstock at xsigo.com Wed Jun 25 10:51:43 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:51:43 -0700 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <20080625174733.GV22159@sashak.voltaire.com> References: <4861E87C.3000007@dev.mellanox.co.il> <20080625165534.GS22159@sashak.voltaire.com> <1214413778.13056.448.camel@hrosenstock-ws.xsigo.com> <20080625174733.GV22159@sashak.voltaire.com> Message-ID: <1214416303.13056.480.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 20:47 +0300, Sasha Khapyorsky wrote: > On 10:09 Wed 25 Jun , Hal Rosenstock wrote: > > On Wed, 2008-06-25 at 19:55 +0300, Sasha Khapyorsky wrote: > > > It was discussed on the list (and even marked as a bug in bugzilla) > > > that unicast LIDs should be represented in decimal form in logs. > > > > Are all the other unicast LID print instances in OpenSM being changed > > too ? > > No, and now it is good time to do. Sure but is anyone ? > > I think hex consistency is better than a piecemeal approach. > > I'm fine with both approaches - do all at once or step by step. In any > case convert to "deprecated" format doesn't seem reasonable for me. IMO it's more confusing to the log reader to mix and match unicast LID formats. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Wed Jun 25 10:52:16 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 10:52:16 -0700 Subject: [ofa-general] Re: [PATCH v2 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <1e336d250806250711l320ec086i8457b4a19cd7102e@mail.gmail.com> (Vladimir Sokolovsky's message of "Wed, 25 Jun 2008 17:11:53 +0300") References: <48611FEB.1030709@dev.mellanox.co.il> <1e336d250806250711l320ec086i8457b4a19cd7102e@mail.gmail.com> Message-ID: > I don't see that there is LRO support in the userspace ethtool. I would guess upstream ethtool will be getting support if it doesn't have it already. Clearly commit 3ae7c0b2 ("[ETHTOOL]: Add ETHTOOL_[GS]FLAGS sub-ioctls") in the kernel shows that ethtool is the right way to control LRO per interface. From sashak at voltaire.com Wed Jun 25 10:50:55 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 20:50:55 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <1214415010.13056.456.camel@hrosenstock-ws.xsigo.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> <20080624003321.GC7341@sashak.voltaire.com> <1214406458.13056.411.camel@hrosenstock-ws.xsigo.com> <20080625171201.GU22159@sashak.voltaire.com> <1214415010.13056.456.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080625175055.GW22159@sashak.voltaire.com> On 10:30 Wed 25 Jun , Hal Rosenstock wrote: > > In an early approach to adding the MLID array, I had tried including > osm_multicast.h in osm_subnet.h and started dealing with the compile > issues. Or yeah, know this very well. But it is due to stupid typedef cross-definitions not inlines. Sasha From hrosenstock at xsigo.com Wed Jun 25 10:54:06 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Wed, 25 Jun 2008 10:54:06 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port_profile: No need to inline some functions In-Reply-To: <20080625175055.GW22159@sashak.voltaire.com> References: <1213802285.13056.127.camel@hrosenstock-ws.xsigo.com> <20080620133953.GE3158@sashak.voltaire.com> <1214229690.13056.236.camel@hrosenstock-ws.xsigo.com> <20080624003321.GC7341@sashak.voltaire.com> <1214406458.13056.411.camel@hrosenstock-ws.xsigo.com> <20080625171201.GU22159@sashak.voltaire.com> <1214415010.13056.456.camel@hrosenstock-ws.xsigo.com> <20080625175055.GW22159@sashak.voltaire.com> Message-ID: <1214416447.13056.482.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 20:50 +0300, Sasha Khapyorsky wrote: > On 10:30 Wed 25 Jun , Hal Rosenstock wrote: > > > > In an early approach to adding the MLID array, I had tried including > > osm_multicast.h in osm_subnet.h and started dealing with the compile > > issues. > > Or yeah, know this very well. But it is due to stupid typedef > cross-definitions not inlines. AFAIR the inlines were in the way too. -- Hal > Sasha From sashak at voltaire.com Wed Jun 25 10:58:49 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 20:58:49 +0300 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <1214416303.13056.480.camel@hrosenstock-ws.xsigo.com> References: <4861E87C.3000007@dev.mellanox.co.il> <20080625165534.GS22159@sashak.voltaire.com> <1214413778.13056.448.camel@hrosenstock-ws.xsigo.com> <20080625174733.GV22159@sashak.voltaire.com> <1214416303.13056.480.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080625175849.GX22159@sashak.voltaire.com> On 10:51 Wed 25 Jun , Hal Rosenstock wrote: > > > > > > Are all the other unicast LID print instances in OpenSM being changed > > > too ? > > > > No, and now it is good time to do. > > Sure but is anyone ? Why to ask, just do it :) Sasha From rdreier at cisco.com Wed Jun 25 11:17:22 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 11:17:22 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 23:16:16 -0700") References: Message-ID: > 1. QP100 last WQE reached event, QP100 context is added into flush_list, > and then it is put into drain_list, and does post_send of a drain WR. > 2. QP200 last WQE reached event, QP200 context is added into flush_list, > but not drain_list since only one drain WR will be posted > 3. QP300 ...., QP300 context is added into flush_list, but not drain_list > > So QP100 is on drain_list, QP200, QP300 are on flush_list > > In rcq poll_cq, > 1. QP 100 drain WR cqe is polled, it will put QP100 into reap_list then > call ipoib_cm_start_rx_drain(), post_send of QP200 drain WR, and QP200, > QP300 are both moved from flush_list to drain_list > 2. QP 200 drain WR cqe is polled, it will move both QP200 and QP300 from > drain_list to reap_list > 3. QP300 cqe comes, but QP300 context has been freed, ---> panic. > > Does that make sense? This is a really good explanation (exactly what I would hope to see in a changelog). I'm not positive I understand though: is the issue that when the QP300 last WQE reached event is seen, we are not guaranteed that we have handled all CQEs for QP300 yet? - R. From rdreier at cisco.com Wed Jun 25 11:18:38 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 11:18:38 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: <1214382352.23583.54.camel@mtls03> (Eli Cohen's message of "Wed, 25 Jun 2008 11:25:52 +0300") References: <1214382352.23583.54.camel@mtls03> Message-ID: > What you describe makes sense to me but it looks like your fix opens > another hole. Now you removed the limit on the number of drain WRs that > may be posted and you risk a CQ overflow. Wouldn't it be simpler to just > move one entry from the flush list to the drain list instead of moving > appending the whole flush list to the drain list? > > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c > index 6223fc3..1fcbf7b 100644 > --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c > +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c > @@ -222,7 +222,7 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) > if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) > ipoib_warn(priv, "failed to post drain wr\n"); > > - list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_drain_list); > + list_move(priv->cm.rx_flush_list.next, &priv->cm.rx_drain_list); > } > > static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) Yes, this makes a lot of sense to me. The issue with the current code is that getting a last WQE reached event doesn't imply that all CQEs have been processed yet ... so we can't free a whole list of QPs at once as the current code does. - R. From rdreier at cisco.com Wed Jun 25 11:19:11 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 11:19:11 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 22:53:31 -0700") References: Message-ID: > Also there is a possible race between timeout reap and last wqe reached > reap, so check the status of that QP context is necessary. What is the race here? - R. From sashak at voltaire.com Wed Jun 25 11:37:05 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 21:37:05 +0300 Subject: [ofa-general] PM MAD tools In-Reply-To: <486280CB.1070408@ufl.edu> References: <48627C69.5000606@ufl.edu> <1214414723.13056.450.camel@hrosenstock-ws.xsigo.com> <486280CB.1070408@ufl.edu> Message-ID: <20080625183705.GZ22159@sashak.voltaire.com> On 13:30 Wed 25 Jun , John Parhizgari wrote: >> > Is there a way to guarantee opensm will enter Standby state? You can run start it in inactive state. Or have lower priority than other SMs. Sasha From rdreier at cisco.com Wed Jun 25 11:45:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 11:45:21 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: <1214382352.23583.54.camel@mtls03> (Eli Cohen's message of "Wed, 25 Jun 2008 11:25:52 +0300") References: <1214382352.23583.54.camel@mtls03> Message-ID: So I think I'll queue this up -- does this make sense to everyone? commit af806651d0042ca014eef62832c7e7083013b954 Author: Roland Dreier Date: Wed Jun 25 11:44:47 2008 -0700 IPoIB/cm: Drain receive QPs one at a time The current handling of last WQE reached events is broken when multiple QPs are drained at once -- for example: 1. Last WQE reached event for QP100: QP100 context is added into flush_list, and then it is put into drain_list, and does post_send of a drain WR. 2. Last WQE reached event for QP200: QP200 context is added into flush_list, but not drain_list since only one drain WR will be posted. 3. Last WQE reached event for QP300: QP300 context is added into flush_list, but not drain_list since only one drain WR will be posted. So at this point, QP100 is on drain_list and QP200 and QP300 are on flush_list. 4. QP100 drain WR CQE is handled: put QP100 into reap_list then call ipoib_cm_start_rx_drain(), which does post_send of QP200 drain WR, and moves both QP200 and QP300 from flush_list to drain_list. 5. QP200 drain WR CQE is polled, which will move both QP200 and QP300 from drain_list to reap_list. 6. QP200 and QP300 are both freed from reap_list. 7. A CQE for QP300 is seen, but QP300 context has been freed ---> crash. The fundamental problem is that a last WQE reached event does not mean all pending completions have been queued. Fix this by moving QPs from the flush_list to the drain_list one at a time, so that we never free QPs until we really know that we are done with them. Debugged-by: Shirley Ma Fix-suggested-by: Eli Cohen Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 6223fc3..793f2bd 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -222,7 +222,7 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); - list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_drain_list); + list_move(&p->list, &priv->cm.rx_drain_list); } static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) From rdreier at cisco.com Wed Jun 25 11:47:50 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 11:47:50 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 18:34:43 -0600") References: Message-ID: > @@ -237,9 +235,11 @@ static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) > return; > > spin_lock_irqsave(&priv->lock, flags); > - list_move(&p->list, &priv->cm.rx_flush_list); > - p->state = IPOIB_CM_RX_FLUSH; > - ipoib_cm_start_rx_drain(priv); > + if (p->state == IPOIB_CM_RX_LIVE) { > + list_move(&p->list, &priv->cm.rx_flush_list); > + p->state = IPOIB_CM_RX_FLUSH; > + ipoib_cm_start_rx_drain(priv); > + } > spin_unlock_irqrestore(&priv->lock, flags); Can you explain this change a little more? It seems quite likely that we would get last WQE reached events for other states, such as IPOIB_CM_RX_ERROR coming from ipoib_cm_dev_stop(), and I don't see how things work if we make this change. - R. From rdreier at cisco.com Wed Jun 25 11:51:36 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 11:51:36 -0700 Subject: [ofa-general] [RFC][PATCH] IPoIB-cm: release connection if post_send drain WR failure In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 23:22:06 -0700") References: Message-ID: > I saw it happened from below ipoib debug output on testing ehca2: > > static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) > { > .... > /* > * QPs on flush list are error state. This way, a "flush > * error" WC will be immediately generated for each WR we post. > */ > p = list_entry(priv->cm.rx_flush_list.next, typeof(*p), list); > if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) > ipoib_warn(priv, "failed to post drain wr\n"); Was this with a kernel that had 863fb09f ("IB/ehca: Prevent posting of SQ WQEs if QP not in RTS") but not 088af154 ("IB/ehca: Reject send WRs only for RESET, INIT and RTR state")? For example something between 2.6.25 and 2.6.26-rc4? - R. From rdreier at cisco.com Wed Jun 25 12:29:04 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 12:29:04 -0700 Subject: [ofa-general] [PATCH] RDMA/cma: Add missing newlines to printk()s Message-ID: Signed-off-by: Roland Dreier --- drivers/infiniband/core/cma.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index e5bd617..44d190f 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -974,7 +974,7 @@ static int cma_ib_handler(struct ib_cm_id *cm_id, struct ib_cm_event *ib_event) event.param.conn.private_data_len = IB_CM_REJ_PRIVATE_DATA_SIZE; break; default: - printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d", + printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d\n", ib_event->event); goto out; } @@ -1450,7 +1450,7 @@ static void cma_listen_on_dev(struct rdma_id_private *id_priv, ret = rdma_listen(id, id_priv->backlog); if (ret) printk(KERN_WARNING "RDMA CMA: cma_listen_on_dev, error %d, " - "listening on device %s", ret, cma_dev->device->name); + "listening on device %s\n", ret, cma_dev->device->name); } static void cma_listen_on_all(struct rdma_id_private *id_priv) @@ -2155,7 +2155,7 @@ static int cma_sidr_rep_handler(struct ib_cm_id *cm_id, event.status = 0; break; default: - printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d", + printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d\n", ib_event->event); goto out; } From xma at us.ibm.com Wed Jun 25 12:57:08 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 12:57:08 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 11:17:22 AM: > > 1. QP100 last WQE reached event, QP100 context is added into flush_list, > > and then it is put into drain_list, and does post_send of a drain WR. > > 2. QP200 last WQE reached event, QP200 context is added into flush_list, > > but not drain_list since only one drain WR will be posted > > 3. QP300 ...., QP300 context is added into flush_list, but not drain_list > > > > So QP100 is on drain_list, QP200, QP300 are on flush_list > > > > In rcq poll_cq, > > 1. QP 100 drain WR cqe is polled, it will put QP100 into reap_list then > > call ipoib_cm_start_rx_drain(), post_send of QP200 drain WR, and QP200, > > QP300 are both moved from flush_list to drain_list > > 2. QP 200 drain WR cqe is polled, it will move both QP200 and QP300 from > > drain_list to reap_list > > 3. QP300 cqe comes, but QP300 context has been freed, ---> panic. > > > > Does that make sense? > > This is a really good explanation (exactly what I would hope to see in a > changelog). I'm not positive I understand though: is the issue that > when the QP300 last WQE reached event is seen, we are not guaranteed > that we have handled all CQEs for QP300 yet? > > - R. Right. We see QP300 last WQE reached event from async event handler, all remaining cqes for QP300 not being processed are in recv completion queue. When QP300 QP is destroyed within QP200 drain WRs context, not in QP300 drain WR context. There is a window QP300 has been destoryed but some QP300 cqes might still be in completion queue and not being processed yet. Thanks Shirley From xma at us.ibm.com Wed Jun 25 13:01:50 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 13:01:50 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 11:47:50 AM: > > @@ -237,9 +235,11 @@ static void ipoib_cm_rx_event_handler(struct > ib_event *event, void *ctx) > > return; > > > > spin_lock_irqsave(&priv->lock, flags); > > - list_move(&p->list, &priv->cm.rx_flush_list); > > - p->state = IPOIB_CM_RX_FLUSH; > > - ipoib_cm_start_rx_drain(priv); > > + if (p->state == IPOIB_CM_RX_LIVE) { > > + list_move(&p->list, &priv->cm.rx_flush_list); > > + p->state = IPOIB_CM_RX_FLUSH; > > + ipoib_cm_start_rx_drain(priv); > > + } > > spin_unlock_irqrestore(&priv->lock, flags); > > Can you explain this change a little more? It seems quite likely that > we would get last WQE reached events for other states, such as > IPOIB_CM_RX_ERROR coming from ipoib_cm_dev_stop(), and I don't see how > things work if we make this change. > > - R. Hello Roland, If it's already in ERROR status, it will be processed through rx_error_list. In the case of ipoib_cm_dev_stop(), it will wait for 5 * HZ to be drained and then put into reap_list. In the case of IPoIB running status, I put a 60 * HZ timer for drain in the stale connection release patch. Thanks Shirley From rdreier at cisco.com Wed Jun 25 13:04:14 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 13:04:14 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 12:57:08 -0700") References: Message-ID: > > This is a really good explanation (exactly what I would hope to see in a > > changelog). I'm not positive I understand though: is the issue that > > when the QP300 last WQE reached event is seen, we are not guaranteed > > that we have handled all CQEs for QP300 yet? > Right. We see QP300 last WQE reached event from async event handler, all > remaining cqes for QP300 not being processed are in recv completion queue. Is there a missing "not" somewhere here? If all CQEs for QP300 are in the recv CQ at the time that the last WQE reached event is handled, then since the drain WQE is posted to QP200 after that event, all the QP300 CQEs will be handled before the completion for the drain WQE completion is handled. It seems that the problem is the following happening (HCA: entries are done by HCA hw, CPU entry is what our code does): HCA: QP200 last WQE reached event generated HCA: QP300 last WQE reached event generated CPU: post QP200 drain WQE HCA: drain WQE completes, add CQE to CQ for it HCA: add CQE to CQ for QP300 receive completion (probably flush err) Is that right? What HCA is this happening with? > When QP300 QP is destroyed within QP200 drain WRs context, not in QP300 > drain WR context. There is a window QP300 has been destoryed but some > QP300 cqes might still be in completion queue and not being processed yet. - R. From rdreier at cisco.com Wed Jun 25 13:07:04 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 13:07:04 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 13:01:50 -0700") References: Message-ID: > > Can you explain this change a little more? It seems quite likely that > > we would get last WQE reached events for other states, such as > > IPOIB_CM_RX_ERROR coming from ipoib_cm_dev_stop(), and I don't see how > > things work if we make this change. > > > > - R. > > Hello Roland, > > If it's already in ERROR status, it will be processed through > rx_error_list. In the case of ipoib_cm_dev_stop(), it will wait for 5 * HZ > to be drained and then put into reap_list. In the case of IPoIB running > status, I put a 60 * HZ timer for drain in the stale connection release > patch. But the 5 second timeout in ipoib_cm_dev_stop() is supposed to be an exception when something gets wedged, just to avoid waiting forever. We want to handle the last WQE reached events normally in most cases. Would a better fix to add locking around the "assume HW is wedged" code in ipoib_cm_dev_stop() to avoid problems if the 5 second timeout is too short? - R. From rdreier at cisco.com Wed Jun 25 13:18:02 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 13:18:02 -0700 Subject: [ofa-general] Re: [PATCH 2/3 v2] mlx4: multicast cast loopback block In-Reply-To: <48622DB0.6090604@voltaire.com> (Or Gerlitz's message of "Wed, 25 Jun 2008 14:36:16 +0300") References: <4860C216.3010400@voltaire.com> <48622DB0.6090604@voltaire.com> Message-ID: > Understood. Will act to have myself and people here run it before > sending next times ... Don't bother to fix it yourself, there's still > time, Ron will fix and resend. OK, thanks. > > not a big deal either way. I prefer my way because if I have > > if (something) > > func(lots, of, parameters, all, the, same, except, 1); > > else > > func(lots, of, parameters, all, the, same, except, 0); > > then it's not always so clear what the difference between the two > > function calls really is without reading carefully. By putting > > !!something into the call: > > > > func(lots, of, parameters, all, the, same, except, !!something); > > > > it becomes obvious how the value of something is used. > yes, I see the point, once you get used to it, !! code is clearer than > func(lots, of, parameters, all, the, same, except, X). So Ron will fix > this as well. By the way, if you prefer I think func(lots, of, parameters, all, the, same, except, something ? 1 : 0); is OK too but I personally find the !! idiom a little nicer. - R. From sashak at voltaire.com Wed Jun 25 13:48:29 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 23:48:29 +0300 Subject: [ofa-general] [PATCH] opensm/configure.in: fix HAVE_DEFAULT_OPENSM_CONFIG_FILE define generation Message-ID: <20080625204829.GB22159@sashak.voltaire.com> Signed-off-by: Sasha Khapyorsky --- opensm/configure.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/configure.in b/opensm/configure.in index 409e7c6..688ba80 100644 --- a/opensm/configure.in +++ b/opensm/configure.in @@ -122,7 +122,7 @@ AC_ARG_WITH(opensm-conf-file, ) AC_MSG_RESULT(${OPENSM_CONFIG_FILE}) AC_DEFINE_UNQUOTED(HAVE_DEFAULT_OPENSM_CONFIG_FILE, - ["$CONF_DIR/$OPENSM_CONFIG_FILE"], + ["$OPENSM_CONFIG_DIR/$OPENSM_CONFIG_FILE"], [Define a default OpenSM config file]) AC_SUBST(OPENSM_CONFIG_FILE) -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Wed Jun 25 13:49:48 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 23:49:48 +0300 Subject: [ofa-general] [PATCH] opensm: --create-config command line option Message-ID: <20080625204948.GC22159@sashak.voltaire.com> Yevgeny pointed out that now when we are using conventional configuration file with OpenSM there still be confused situation with write-only option cache file in predefined directory. Instead he suggested to have --create-config option where file name is mandatory, so OpenSM will dump configuration template there. We agreed and this is the patch. It adds -c, --create-config command line option. When used OpenSM will dump its current configuration (including another specified options) to the specified file and exit. Signed-off-by: Sasha Khapyorsky --- opensm/man/opensm.8.in | 15 +++++---------- opensm/opensm/main.c | 45 +++++++++++++++++++-------------------------- 2 files changed, 24 insertions(+), 36 deletions(-) diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index 964fbd3..0fb1af5 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -6,7 +6,8 @@ opensm \- InfiniBand subnet manager and administration (SM/SA) .SH SYNOPSIS .B opensm [\-\-version]] -[\-c(ache-options)] [\-g(uid)[=]] [\-l(mc) ] +[\-c(reate-config) ] +[\-g(uid)[=]] [\-l(mc) ] [\-p(riority) ] [\-smkey ] [\-r(eassign_lids)] [\-R | \-\-routing_engine ] [\-z | \-\-connect_roots] @@ -67,15 +68,9 @@ setup the subnet correctly. \fB\-\-version\fR Prints OpenSM version and exits. .TP -\fB\-c\fR, \fB\-\-cache-options\fR -Write out a list of all tunable OpenSM parameters, -including their current values from the command line -as well as defaults for others, into the file -OSM_CACHE_DIR/opensm.opts (OSM_CACHE_DIR defaults to -/var/cache/opensm if the corresponding environment -variable is not set). The options file is then -used for subsequent OpenSM invocations but any -command line options take precedence. +\fB\-c\fR, \fB\-\-create-config\fR +OpenSM will dump its configuration to the specified file and exit. +This is a way to generate OpenSM configuration file template. .TP \fB\-g\fR, \fB\-\-guid\fR This option specifies the local port GUID value diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 248171a..77c4b19 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -129,15 +129,11 @@ static void show_usage(void) printf("--version\n" " Prints OpenSM version and exits.\n\n"); printf("-F , --config \n" - " The name of the OpenSM config file. It has a same format\n" - " as opensm.opts option cache file. When not specified\n" + " The name of the OpenSM config file. When not specified\n" " " OSM_DEFAULT_CONFIG_FILE " will be used (if exists).\n\n"); - printf("-c\n" - "--cache-options\n" - " Cache the given command line options into the file\n" - " /var/cache/opensm/opensm.opts for use on next invocation.\n" - " The cache directory can be changed by the environment\n" - " variable OSM_CACHE_DIR\n\n"); + printf("-c , --create-config \n" + " OpenSM will dump its configuration to the specified file and exit.\n" + " This is a way to generate OpenSM configuration file template.\n\n"); printf("-g[=]\n" "--guid[=]\n" " This option specifies the local port GUID value\n" @@ -608,11 +604,11 @@ int main(int argc, char *argv[]) boolean_t run_once_flag = FALSE; int32_t vendor_debug = 0; uint32_t next_option; - boolean_t cache_options = FALSE; + char *conf_template = NULL; uint32_t val; unsigned config_file_done = 0; const char *const short_option = - "F:i:f:ed:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:NBIQvVhorcyxp:n:q:k:C:"; + "F:c:i:f:ed:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:NBIQvVhoryxp:n:q:k:C:"; /* In the array below, the 2nd parameter specifies the number @@ -624,6 +620,7 @@ int main(int argc, char *argv[]) const struct option long_option[] = { {"version", 0, NULL, 12}, {"config", 1, NULL, 'F'}, + {"create-config", 1, NULL, 'c'}, {"debug", 1, NULL, 'd'}, {"guid", 1, NULL, 'g'}, {"ignore_guids", 1, NULL, 'i'}, @@ -656,7 +653,6 @@ int main(int argc, char *argv[]) {"cn_guid_file", 1, NULL, 'u'}, {"ids_guid_file", 1, NULL, 'm'}, {"guid_routing_order_file", 1, NULL, 'X'}, - {"cache-options", 0, NULL, 'c'}, {"stay_on_fatal", 0, NULL, 'y'}, {"honor_guid2lid", 0, NULL, 'x'}, #ifdef ENABLE_OSM_CONSOLE_SOCKET @@ -714,6 +710,11 @@ int main(int argc, char *argv[]) config_file_done = 1; optind = 0; break; + case 'c': + conf_template = optarg; + printf(" Creating config file template \'%s\'.\n", + conf_template); + break; case 'o': /* Run once option. @@ -967,11 +968,6 @@ int main(int argc, char *argv[]) printf(" GUID Routing Order File: %s\n", opt.guid_routing_order_file); break; - case 'c': - cache_options = TRUE; - printf(" Caching command line options\n"); - break; - case 'x': opt.honor_guid2lid_file = TRUE; printf(" Honor guid2lid file, if possible\n"); @@ -1021,6 +1017,13 @@ int main(int argc, char *argv[]) /* Done with options description */ printf("-------------------------------------------------\n"); + if (conf_template) { + status = osm_subn_write_conf_file(conf_template, &opt); + if (status) + printf("\nosm_subn_write_conf_file failed!\n"); + exit(status); + } + if (vendor_debug) osm_vendor_set_debug(osm.p_vendor, vendor_debug); @@ -1050,16 +1053,6 @@ int main(int argc, char *argv[]) if (opt.guid == 0 || cl_hton64(opt.guid) == CL_HTON64(INVALID_GUID)) opt.guid = get_port_guid(&osm, opt.guid); - if (cache_options == TRUE) { - char conf_file[256]; - char *cache_dir = getenv("OSM_CACHE_DIR"); - if (!cache_dir || !(*cache_dir)) - cache_dir = OSM_DEFAULT_CACHE_DIR; - snprintf(conf_file, sizeof(conf_file), "%s/opensm.opts", cache_dir); - if (osm_subn_write_conf_file(conf_file, &opt)) - printf("\nosm_subn_write_conf_file failed!\n"); - } - status = osm_opensm_bind(&osm, opt.guid); if (status != IB_SUCCESS) { printf("\nError from osm_opensm_bind (0x%X)\n", status); -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Wed Jun 25 13:50:32 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 25 Jun 2008 23:50:32 +0300 Subject: [ofa-general] [PATCH] opensm/man: add -F. --config option to OpenSM man page In-Reply-To: <20080625204948.GC22159@sashak.voltaire.com> References: <20080625204948.GC22159@sashak.voltaire.com> Message-ID: <20080625205032.GD22159@sashak.voltaire.com> Signed-off-by: Sasha Khapyorsky --- opensm/man/opensm.8.in | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index 0fb1af5..d5cd820 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -6,7 +6,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA) .SH SYNOPSIS .B opensm [\-\-version]] -[\-c(reate-config) ] +[\-F | \-\-config ] [\-c(reate-config) ] [\-g(uid)[=]] [\-l(mc) ] [\-p(riority) ] [\-smkey ] [\-r(eassign_lids)] [\-R | \-\-routing_engine ] @@ -68,6 +68,10 @@ setup the subnet correctly. \fB\-\-version\fR Prints OpenSM version and exits. .TP +\fB\-F\fR, \fB\-\-config\fR +The name of the OpenSM config file. When not specified +\fB\% @OPENSM_CONFIG_DIR@/@OPENSM_CONFIG_FILE@\fP will be used (if exists). +.TP \fB\-c\fR, \fB\-\-create-config\fR OpenSM will dump its configuration to the specified file and exit. This is a way to generate OpenSM configuration file template. @@ -941,6 +945,10 @@ Both or one of options -U and -M can be specified together with \'-R file\'. .SH FILES .TP +.B @OPENSM_CONFIG_DIR@/@OPENSM_CONFIG_FILE@ +default OpenSM config file. + +.TP .B @OPENSM_CONFIG_DIR@/@NODENAMEMAPFILE@ default node name map file. See ibnetdiscover for more information on format. -- 1.5.5.1.178.g1f811 From rdreier at cisco.com Wed Jun 25 14:05:59 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 14:05:59 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 13:45:31 -0700") References: Message-ID: > I prefer the original patch, the reason is: drain_list only have one > element, which the first of of the flush list, whenever its drain WR > processed, it will be put in reap_list, why we need drain_list here? Maybe I'm missing something. If we get rid of drain_list, how does ipoib_cm_start_rx_drain() know if a drain WQE is pending or not? I guess we can look at how many entries the flush_list has, something like this? (compile tested only) commit 17120d826ec4c12f71d075b69a836f0d4f5ac499 Author: Roland Dreier Date: Wed Jun 25 14:05:17 2008 -0700 IPoIB/cm: Drain receive QPs one at a time The current handling of last WQE reached events is broken when multiple QPs are drained at once -- for example: 1. Last WQE reached event for QP100: QP100 context is added into flush_list, and then it is put into drain_list, and does post_send of a drain WR. 2. Last WQE reached event for QP200: QP200 context is added into flush_list, but not drain_list since only one drain WR will be posted. 3. Last WQE reached event for QP300: QP300 context is added into flush_list, but not drain_list since only one drain WR will be posted. So at this point, QP100 is on drain_list and QP200 and QP300 are on flush_list. 4. QP100 drain WR CQE is handled: put QP100 into reap_list then call ipoib_cm_start_rx_drain(), which does post_send of QP200 drain WR, and moves both QP200 and QP300 from flush_list to drain_list. 5. QP200 drain WR CQE is polled, which will move both QP200 and QP300 from drain_list to reap_list. 6. QP200 and QP300 are both freed from reap_list. 7. A CQE for QP300 is seen, but QP300 context has been freed ---> crash. The fundamental problem is that a last WQE reached event does not mean all pending completions have been queued. Fix this by flushing QPs from the flush_list one at a time, so that we never free QPs until we really know that we are done with them. This allows us to simplify the code by removing the drain_list completely (since we flush one at a time, the context to move to the reap_list when the drain WQE completes is always just the first entry in flush_list). Debugged-by: Shirley Ma Fix-suggested-by: Eli Cohen Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 8754b36..c02e3b0 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -224,8 +224,7 @@ struct ipoib_cm_dev_priv { struct ib_cm_id *id; struct list_head passive_ids; /* state: LIVE */ struct list_head rx_error_list; /* state: ERROR */ - struct list_head rx_flush_list; /* state: FLUSH, drain not started */ - struct list_head rx_drain_list; /* state: FLUSH, drain started */ + struct list_head rx_flush_list; /* state: FLUSHING or waiting for FLUSH */ struct list_head rx_reap_list; /* state: FLUSH, drain done */ struct work_struct start_task; struct work_struct reap_task; diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 6223fc3..07939ed 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -208,21 +208,13 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) struct ib_send_wr *bad_wr; struct ipoib_cm_rx *p; - /* We only reserved 1 extra slot in CQ for drain WRs, so - * make sure we have at most 1 outstanding WR. */ - if (list_empty(&priv->cm.rx_flush_list) || - !list_empty(&priv->cm.rx_drain_list)) - return; - /* * QPs on flush list are error state. This way, a "flush * error" WC will be immediately generated for each WR we post. */ - p = list_entry(priv->cm.rx_flush_list.next, typeof(*p), list); + p = list_first_entry(&priv->cm.rx_flush_list, typeof(*p), list); if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); - - list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_drain_list); } static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) @@ -235,9 +227,15 @@ static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) return; spin_lock_irqsave(&priv->lock, flags); - list_move(&p->list, &priv->cm.rx_flush_list); + + list_move_tail(&p->list, &priv->cm.rx_flush_list); p->state = IPOIB_CM_RX_FLUSH; - ipoib_cm_start_rx_drain(priv); + + /* We only reserved 1 extra slot in CQ for drain WRs, so + * make sure we have at most 1 outstanding WR. */ + if (list_is_singular(&priv->cm.rx_flush_list)) + ipoib_cm_start_rx_drain(priv); + spin_unlock_irqrestore(&priv->lock, flags); } @@ -531,8 +529,9 @@ void ipoib_cm_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) if (unlikely(wr_id >= ipoib_recvq_size)) { if (wr_id == (IPOIB_CM_RX_DRAIN_WRID & ~(IPOIB_OP_CM | IPOIB_OP_RECV))) { spin_lock_irqsave(&priv->lock, flags); - list_splice_init(&priv->cm.rx_drain_list, &priv->cm.rx_reap_list); - ipoib_cm_start_rx_drain(priv); + list_move(&priv->cm.rx_flush_list, &priv->cm.rx_reap_list); + if (!list_empty(&priv->cm.rx_flush_list)) + ipoib_cm_start_rx_drain(priv); queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); spin_unlock_irqrestore(&priv->lock, flags); } else @@ -867,8 +866,7 @@ void ipoib_cm_dev_stop(struct net_device *dev) begin = jiffies; while (!list_empty(&priv->cm.rx_error_list) || - !list_empty(&priv->cm.rx_flush_list) || - !list_empty(&priv->cm.rx_drain_list)) { + !list_empty(&priv->cm.rx_flush_list)) { if (time_after(jiffies, begin + 5 * HZ)) { ipoib_warn(priv, "RX drain timing out\n"); @@ -879,8 +877,6 @@ void ipoib_cm_dev_stop(struct net_device *dev) &priv->cm.rx_reap_list); list_splice_init(&priv->cm.rx_error_list, &priv->cm.rx_reap_list); - list_splice_init(&priv->cm.rx_drain_list, - &priv->cm.rx_reap_list); break; } spin_unlock_irq(&priv->lock); @@ -1472,7 +1468,6 @@ int ipoib_cm_dev_init(struct net_device *dev) INIT_LIST_HEAD(&priv->cm.start_list); INIT_LIST_HEAD(&priv->cm.rx_error_list); INIT_LIST_HEAD(&priv->cm.rx_flush_list); - INIT_LIST_HEAD(&priv->cm.rx_drain_list); INIT_LIST_HEAD(&priv->cm.rx_reap_list); INIT_WORK(&priv->cm.start_task, ipoib_cm_tx_start); INIT_WORK(&priv->cm.reap_task, ipoib_cm_tx_reap); From parhizj at ufl.edu Wed Jun 25 14:08:28 2008 From: parhizj at ufl.edu (John Parhizgari) Date: Wed, 25 Jun 2008 17:08:28 -0400 Subject: [ofa-general] opensm config In-Reply-To: <20080625204948.GC22159@sashak.voltaire.com> References: <20080625204948.GC22159@sashak.voltaire.com> Message-ID: <4862B3CC.5060702@ufl.edu> I do not know if you have changed this since opensm 3.1.11, but the default behavior for "opensm -c" is to create the opensm.opts with program defaults, unfortunately if it runs in master mode it will also simultaneously change all the current SM settings to these program defaults at this time; this behavior is awkward since most will understand from the man page -c is ONLY supposed to create the config file -- not create and load in program defaults into the subnet. At the least make it clear in the manual that the config file will be created using program defaults, not the current settings of the SM, and these settings will be loaded in at this time if you use -c (to make sure the user still has a copy of what their SM settings were) It might be beneficial for it to be able to get the current SM settings and use those to create the .opts file (for those that aren't using OpenSM now). I think the default behavior should really be that unless the user specifies to load in the configuration file opensm.opts file either in the opensm.conf or on the command line it shouldn't modify any of the SM settings, since they may inadvertently mess with the current SM settings if the cluster is up and running. From xma at us.ibm.com Wed Jun 25 13:45:31 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 13:45:31 -0700 Subject: ***SPAM*** Re: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: I prefer the original patch, the reason is: drain_list only have one element, which the first of of the flush list, whenever its drain WR processed, it will be put in reap_list, why we need drain_list here? Thanks Shirley Roland Dreier > cc Shirley Ma/Beaverton/IBM at IBMUS, 06/25/08 , 11:45 AM "Olga Stern" , "OpenFabrics General" Subject Re: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch So I think I'll queue this up -- does this make sense to everyone? commit af806651d0042ca014eef62832c7e7083013b954 Author: Roland Dreier Date: Wed Jun 25 11:44:47 2008 -0700 IPoIB/cm: Drain receive QPs one at a time The current handling of last WQE reached events is broken when multiple QPs are drained at once -- for example: 1. Last WQE reached event for QP100: QP100 context is added into flush_list, and then it is put into drain_list, and does post_send of a drain WR. 2. Last WQE reached event for QP200: QP200 context is added into flush_list, but not drain_list since only one drain WR will be posted. 3. Last WQE reached event for QP300: QP300 context is added into flush_list, but not drain_list since only one drain WR will be posted. So at this point, QP100 is on drain_list and QP200 and QP300 are on flush_list. 4. QP100 drain WR CQE is handled: put QP100 into reap_list then call ipoib_cm_start_rx_drain(), which does post_send of QP200 drain WR, and moves both QP200 and QP300 from flush_list to drain_list. 5. QP200 drain WR CQE is polled, which will move both QP200 and QP300 from drain_list to reap_list. 6. QP200 and QP300 are both freed from reap_list. 7. A CQE for QP300 is seen, but QP300 context has been freed ---> crash. The fundamental problem is that a last WQE reached event does not mean all pending completions have been queued. Fix this by moving QPs from the flush_list to the drain_list one at a time, so that we never free QPs until we really know that we are done with them. Debugged-by: Shirley Ma Fix-suggested-by: Eli Cohen Signed-off-by: Roland Dreier diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 6223fc3..793f2bd 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -222,7 +222,7 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); - list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_drain_list); + list_move(&p->list, &priv->cm.rx_drain_list); } static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pic21829.gif Type: image/gif Size: 1255 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: From xma at us.ibm.com Wed Jun 25 14:29:56 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 14:29:56 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 01:04:14 PM: > Is there a missing "not" somewhere here? If all CQEs for QP300 are in > the recv CQ at the time that the last WQE reached event is handled, then > since the drain WQE is posted to QP200 after that event, all the QP300 > CQEs will be handled before the completion for the drain WQE completion > is handled. > > It seems that the problem is the following happening (HCA: entries are > done by HCA hw, CPU entry is what our code does): > > HCA: QP200 last WQE reached event generated > HCA: QP300 last WQE reached event generated > CPU: post QP200 drain WQE > HCA: drain WQE completes, add CQE to CQ for it > HCA: add CQE to CQ for QP300 receive completion (probably flush err) > > Is that right? Yes. > What HCA is this happening with? I saw the panic on mthca. I guess that's what happened from the code review only. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Wed Jun 25 14:28:22 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 26 Jun 2008 00:28:22 +0300 Subject: [ofa-general] opensm config In-Reply-To: <4862B3CC.5060702@ufl.edu> References: <20080625204948.GC22159@sashak.voltaire.com> <4862B3CC.5060702@ufl.edu> Message-ID: <20080625212822.GE22159@sashak.voltaire.com> On 17:08 Wed 25 Jun , John Parhizgari wrote: > I do not know if you have changed this since opensm 3.1.11, Yes, we changed this in the master (3.2.1+) already. > but the default > behavior for "opensm -c" is to create the opensm.opts with program > defaults, Not only defaults, if you will specify something in OpenSM command line, or already have opensm.opts file it will be there too. > unfortunately if it runs in master mode it will also > simultaneously change all the current SM settings to these program defaults > at this time; this behavior is awkward since most will understand from the > man page -c is ONLY supposed to create the config file It is how it will work now - generate config file and exit. > -- not create and > load in program defaults into the subnet. At the least make it clear in > the manual that the config file will be created using program defaults, not > the current settings of the SM, and these settings will be loaded in at > this time if you use -c (to make sure the user still has a copy of what > their SM settings were) > > It might be beneficial for it to be able to get the current SM settings and > use those to create the .opts file (for those that aren't using OpenSM > now). I see what you mean - to be aware about another SM configuration. OpenSM will try to preserve many already configured things on a fabric (like LID assignment), but knowing all another SM (which is also proprietary) internal rules is not something easy doable. Sasha From xma at us.ibm.com Wed Jun 25 14:33:43 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 14:33:43 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 01:07:04 PM: > Would a better fix to add locking around the "assume HW is wedged" > code in ipoib_cm_dev_stop() to avoid problems if the 5 second timeout is > too short? > > - R. According to our stress test, 5 second timeout is way too short. What do you mean by add locking there? Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Wed Jun 25 14:39:54 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 26 Jun 2008 00:39:54 +0300 Subject: [ofa-general] Re: [PATCH] allow explicit HCA selection by functions from libibmad In-Reply-To: <18528.58760.868810.615990@kuku.melbourne.sgi.com> References: <18522.29300.545966.523800@kuku.melbourne.sgi.com> <20080620134622.GG3158@sashak.voltaire.com> <18523.47597.85812.745257@kuku.melbourne.sgi.com> <20080620145722.GH3158@sashak.voltaire.com> <18528.58760.868810.615990@kuku.melbourne.sgi.com> Message-ID: <20080625213954.GF22159@sashak.voltaire.com> On 22:16 Tue 24 Jun , Max Matveev wrote: > > Add extra entry points to allow explicit selection of > HCA ports used to send MAD rpcs. > > Signed-off-by: Max Matveev Good work! Thanks. Applied. [snip...] > diff --git a/libibmad/src/smp.c b/libibmad/src/smp.c > index cc4c776..e3795e4 100644 > --- a/libibmad/src/smp.c > +++ b/libibmad/src/smp.c > @@ -49,7 +49,7 @@ > #define DEBUG if (ibdebug) IBWARN > > uint8_t * > -smp_set(void *data, ib_portid_t *portid, unsigned attrid, unsigned mod, unsigned timeout) > +smp_set_via(void *data, ib_portid_t *portid, unsigned attrid, unsigned mod, unsigned timeout, const void *srcport) > { > ib_rpc_t rpc = {0}; > > @@ -71,12 +71,22 @@ smp_set(void *data, ib_portid_t *portid, unsigned attrid, unsigned mod, unsigned > portid->sl = 0; > portid->qp = 0; > > - return madrpc(&rpc, portid, data, data); > + if (srcport) { > + return mad_rpc(srcport, &rpc, portid, data, data); > + } else { > + return madrpc(&rpc, portid, data, data); > + } > +} > + > +uint8_t * > +smp_set(void *data, ib_portid_t *portid, unsigned attrid, unsigned mod, unsigned timeout) > +{ > + return smp_set_via(data, portid, attrid, mod, timeout, NULL); > } > > uint8_t * > -smp_query(void *rcvbuf, ib_portid_t *portid, unsigned attrid, unsigned mod, > - unsigned timeout) > +smp_query_via(void *rcvbuf, ib_portid_t *portid, unsigned attrid, unsigned mod, > + unsigned timeout, const void *srcport) > { > ib_rpc_t rpc = {0}; Somehow you removed smp_query() function. I fixed this in merge time. Sasha > > @@ -98,5 +108,9 @@ smp_query(void *rcvbuf, ib_portid_t *portid, unsigned attrid, unsigned mod, > portid->sl = 0; > portid->qp = 0; > > - return madrpc(&rpc, portid, 0, rcvbuf); > + if (srcport) { > + return mad_rpc(srcport, &rpc, portid, 0, rcvbuf); > + } else { > + return madrpc(&rpc, portid, 0, rcvbuf); > + } > } From rdreier at cisco.com Wed Jun 25 14:43:19 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 14:43:19 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 14:33:43 -0700") References: Message-ID: > According to our stress test, 5 second timeout is way too short. What do > you mean by add locking there? Interesting... I wonder if it really is taking that long for everything to finish draining, or if the system is too busy so it sees a spurious timeout? The intention of all of this is that it should "never happen" unless the hardware really is stuck. Anyway I meant by locking that we take the lock around /* * assume the HW is wedged and just free up everything. */ list_splice_init(&priv->cm.rx_flush_list, &priv->cm.rx_reap_list); list_splice_init(&priv->cm.rx_error_list, &priv->cm.rx_reap_list); so that we don't end up moving stuff out from under other processing. But now I see that the lock is already held so maybe that's nonsense. What exactly is causing the crash here? - R. From xma at us.ibm.com Wed Jun 25 14:46:16 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 14:46:16 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Hello Roland, >I guess we can look at how many entries the flush_list has, something like this? (compile tested only) The approach looks good to me. thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Wed Jun 25 14:45:22 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 26 Jun 2008 00:45:22 +0300 Subject: [ofa-general] [PATCH 1/2 v2] libibmad: add a drpath2str function for easier printing of DR Paths. In-Reply-To: <20080624164557.398354bf.weiny2@llnl.gov> References: <20080624144838.55ec7e3d.weiny2@llnl.gov> <1214346687.13056.352.camel@hrosenstock-ws.xsigo.com> <20080624164557.398354bf.weiny2@llnl.gov> Message-ID: <20080625214522.GG22159@sashak.voltaire.com> On 16:45 Tue 24 Jun , Ira Weiny wrote: > > >From 3e192dabd78f81342106b24ee0407725f8cf78eb Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Tue, 24 Jun 2008 14:42:25 -0700 > Subject: [PATCH] libibmad: add a drpath2str function for easier printing of DR Paths. > > > Signed-off-by: Ira K. Weiny Applied. Thanks. Sasha From teo at qr.ro Wed Jun 25 14:52:10 2008 From: teo at qr.ro (Shawn Luna) Date: Wed, 25 Jun 2008 18:52:10 -0300 Subject: [ofa-general] Re: 244194 Message-ID: <01c8d6f4$929a2900$40b70cbd@teo> Take more from this world! Ultimate effect! http://www.ebletka.cn/v/ From rdreier at cisco.com Wed Jun 25 14:58:39 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 14:58:39 -0700 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: <4860C352.5070603@voltaire.com> (Or Gerlitz's message of "Tue, 24 Jun 2008 12:50:10 +0300") References: <485A7903.4050407@voltaire.com> <4860C352.5070603@voltaire.com> Message-ID: > My patches are only dependent on the three commits to Jeff's tree > which I have stated in the 0/4 email message. They are not dependent > on other patches to the infiniband tree or any other tree. I do want > them to be queued for push into 2.6.27 at some tree. > So one option is that you pull Jeff's davem-next into a branch at > your tree and once Jeff does his push you ask to pull this branch > whose only diff is would be those three patches, I think it > possible, and you did it with libiscsi on the time we pushed > iser. If this becomes a mess, lets just push the four patches to > Jeff. How about if we just wait for these patches to make it all the way to Linus's tree, and then I can just merge your patches and ask Linus to pull in a simple way. - R. From or.gerlitz at gmail.com Wed Jun 25 15:08:53 2008 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Thu, 26 Jun 2008 01:08:53 +0300 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: References: <485A7903.4050407@voltaire.com> <4860C352.5070603@voltaire.com> Message-ID: <15ddcffd0806251508r65ddcd76q709e449a3a960460@mail.gmail.com> On 6/26/08, Roland Dreier wrote: > > > How about if we just wait for these patches to make it all the way to > Linus's tree, and then I can just merge your patches and ask Linus to pull > in a simple way. Generally speaking, if this is your preference, I'm fine with this approach. I wonder why you prefer this way or just pushing them to Jeff such that he asks Linus to pull them. Or -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Wed Jun 25 15:09:12 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 15:09:12 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 02:43:19 PM: > Interesting... I wonder if it really is taking that long for everything > to finish draining, or if the system is too busy so it sees a spurious > timeout? The intention of all of this is that it should "never happen" > unless the hardware really is stuck. I guess the reason might be we have a large cluster, each node has 4 ports, too many RC QPs in this set up. We saw QPs went to dead and 5 secs drain didn't work. > What exactly is causing the crash here? You can ignore this for now, it's related to other patch not current code level. I will explain it in drain WR post_send failure patch. Please review the stale connection resource cleanup patch to see whether it makes sense. thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Wed Jun 25 15:10:18 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 15:10:18 -0700 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: <15ddcffd0806251508r65ddcd76q709e449a3a960460@mail.gmail.com> (Or Gerlitz's message of "Thu, 26 Jun 2008 01:08:53 +0300") References: <485A7903.4050407@voltaire.com> <4860C352.5070603@voltaire.com> <15ddcffd0806251508r65ddcd76q709e449a3a960460@mail.gmail.com> Message-ID: > Generally speaking, if this is your preference, I'm fine with this approach. > I wonder why you prefer this way or just pushing them to Jeff such that he > asks Linus to pull them. It just seems to minimize the hassle. I am fine if you want to send them to Jeff but you seemed to worry about conflicts with other RDMA CM stuff that might be pending. - R. From rdreier at cisco.com Wed Jun 25 15:11:18 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 15:11:18 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last recv race patch In-Reply-To: (Shirley Ma's message of "Tue, 24 Jun 2008 18:28:54 -0600") References: Message-ID: Did I miss it or did you ever explain to me what the race here is? > Address a possible race > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c > index 97e67d3..0886ee7 100644 > --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c > +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c > @@ -559,9 +559,12 @@ void ipoib_cm_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) > else { > if (!--p->recv_count) { > spin_lock_irqsave(&priv->lock, flags); > - list_move(&p->list, &priv->cm.rx_reap_list); > - spin_unlock_irqrestore(&priv->lock, flags); > - queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); > + if (p->state == IPOIB_CM_RX_LIVE) { > + list_move(&p->list, &priv->cm.rx_reap_list); > + spin_unlock_irqrestore(&priv->lock, flags); > + queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); > + } else > + spin_unlock_irqrestore(&priv->lock, flags); From xma at us.ibm.com Wed Jun 25 15:16:16 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 15:16:16 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last recv race patch In-Reply-To: Message-ID: Hello Roland, The race is between the list is taken off from reap_list for destory because of the timeout and readd to reap_list from this condition. So we need a condition check here. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Wed Jun 25 15:21:35 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 26 Jun 2008 01:21:35 +0300 Subject: [ofa-general] [PATCH] opensm: add osm_version field to osm_opensm_t object Message-ID: <20080625222135.GA28318@sashak.voltaire.com> This adds osm_version field to osm_opensm_t object. The value is generated in OSM_VERSION macro in osm_version.h file. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_opensm.h | 4 ++++ opensm/opensm/osm_console.c | 5 ++--- opensm/opensm/osm_opensm.c | 5 +++-- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index 285fd88..a1c255b 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -163,6 +163,7 @@ struct osm_routing_engine { * SYNOPSIS */ typedef struct osm_opensm { + const char *osm_version; osm_subn_t subn; osm_sm_t sm; osm_sa_t sa; @@ -185,6 +186,9 @@ typedef struct osm_opensm { } osm_opensm_t; /* * FIELDS +* osm_version +* OpenSM version (as generated in osm_version.h) +* * subn * Subnet object for this subnet. * diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 7f0b00f..39fb17a 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -50,7 +50,6 @@ #include #include #include -#include #include #include #include @@ -312,7 +311,7 @@ static void print_status(osm_opensm_t * p_osm, FILE * out) { if (out) { cl_plock_acquire(&p_osm->lock); - fprintf(out, " OpenSM Version: %s\n", OSM_VERSION); + fprintf(out, " OpenSM Version: %s\n", p_osm->osm_version); fprintf(out, " SM State : %s\n", sm_state_str(p_osm->subn.sm_state)); fprintf(out, " SA State : %s\n", @@ -1137,7 +1136,7 @@ static void help_version(FILE * out, int detail) static void version_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) { - fprintf(out, "%s build %s %s\n", OSM_VERSION, __DATE__, __TIME__); + fprintf(out, "%s build %s %s\n", p_osm->osm_version, __DATE__, __TIME__); } /* more parse routines go here */ diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index adacd93..0101b97 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -169,6 +169,7 @@ static int osm_ucast_null_setup(osm_opensm_t * p_osm) void osm_opensm_construct(IN osm_opensm_t * const p_osm) { memset(p_osm, 0, sizeof(*p_osm)); + p_osm->osm_version = OSM_VERSION; osm_subn_construct(&p_osm->subn); osm_sm_construct(&p_osm->sm); osm_sa_construct(&p_osm->sa); @@ -283,9 +284,9 @@ osm_opensm_init(IN osm_opensm_t * const p_osm, /* If there is a log level defined - add the OSM_VERSION to it */ osm_log(&p_osm->log, osm_log_get_level(&p_osm->log) & (OSM_LOG_SYS ^ 0xFF), "%s\n", - OSM_VERSION); + p_osm->osm_version); /* Write the OSM_VERSION to the SYS_LOG */ - osm_log(&p_osm->log, OSM_LOG_SYS, "%s\n", OSM_VERSION); /* Format Waived */ + osm_log(&p_osm->log, OSM_LOG_SYS, "%s\n", p_osm->osm_version); /* Format Waived */ OSM_LOG(&p_osm->log, OSM_LOG_FUNCS, "[\n"); /* Format Waived */ -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Wed Jun 25 15:23:31 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 26 Jun 2008 01:23:31 +0300 Subject: [ofa-general] [PATCH RFC] opensm/event_plugin: plugin API version 2 Message-ID: <20080625222331.GB28318@sashak.voltaire.com> Main difference is that construct() method is renamed to create() and gets reference to osm_opensm_t object instead of just osm_log_t. This will provide to event plugin access to all OpenSM data structures and should help to create more generic plugins. For consistency with above destroy() method is renamed to delete(). Event plugin interface version is increased and osmeventplugin example updated accordingly. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_event_plugin.h | 12 ++++++------ opensm/opensm/osm_event_plugin.c | 23 +++++++++++------------ opensm/opensm/osm_opensm.c | 2 +- opensm/osmeventplugin/src/osmeventplugin.c | 18 ++++++++++++------ 4 files changed, 30 insertions(+), 25 deletions(-) diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h index 35d5b7d..953709d 100644 --- a/opensm/include/opensm/osm_event_plugin.h +++ b/opensm/include/opensm/osm_event_plugin.h @@ -37,7 +37,6 @@ #include #include #include -#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -60,6 +59,8 @@ BEGIN_C_DECLS *********/ #define OSM_EPI_NODE_NAME_LEN (128) + +struct osm_opensm; /** ========================================================================= * Event types */ @@ -145,11 +146,11 @@ typedef struct osm_epi_trap_event { * The version should be set to OSM_EVENT_PLUGIN_INTERFACE_VER */ #define OSM_EVENT_PLUGIN_IMPL_NAME "osm_event_plugin" -#define OSM_EVENT_PLUGIN_INTERFACE_VER (1) +#define OSM_EVENT_PLUGIN_INTERFACE_VER 2 typedef struct osm_event_plugin { int interface_version; - void *(*construct) (osm_log_t * osm_log); - void (*destroy) (void *plugin_data); + void *(*create) (struct osm_opensm *osm); + void (*delete) (void *plugin_data); void (*report) (void *plugin_data, osm_epi_event_id_t event_id, void *event_data); } osm_event_plugin_t; @@ -162,14 +163,13 @@ typedef struct osm_epi_plugin { void *handle; osm_event_plugin_t *impl; void *plugin_data; - osm_log_t *p_log; char *plugin_name; } osm_epi_plugin_t; /** * functions */ -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name); +osm_epi_plugin_t *osm_epi_construct(struct osm_opensm *osm, char *plugin_name); void osm_epi_destroy(osm_epi_plugin_t * plugin); /** ========================================================================= diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c index e579fad..98d5302 100644 --- a/opensm/opensm/osm_event_plugin.c +++ b/opensm/opensm/osm_event_plugin.c @@ -44,8 +44,8 @@ #include #include - #include +#include #if defined(PATH_MAX) #define OSM_PATH_MAX (PATH_MAX + 1) @@ -58,7 +58,7 @@ /** * functions */ -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) +osm_epi_plugin_t *osm_epi_construct(osm_opensm_t *osm, char *plugin_name) { char lib_name[OSM_PATH_MAX]; osm_epi_plugin_t *rc = NULL; @@ -75,7 +75,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) rc->handle = dlopen(lib_name, RTLD_LAZY); if (!rc->handle) { - OSM_LOG(p_log, OSM_LOG_ERROR, + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Failed to open event plugin \"%s\" : \"%s\"\n", lib_name, dlerror()); goto DLOPENFAIL; @@ -85,7 +85,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) (osm_event_plugin_t *) dlsym(rc->handle, OSM_EVENT_PLUGIN_IMPL_NAME); if (!rc->impl) { - OSM_LOG(p_log, OSM_LOG_ERROR, + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Failed to find \"%s\" symbol in \"%s\" : \"%s\"\n", OSM_EVENT_PLUGIN_IMPL_NAME, lib_name, dlerror()); goto Exit; @@ -93,7 +93,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) /* Check the version to make sure this module will work with us */ if (rc->impl->interface_version != OSM_EVENT_PLUGIN_INTERFACE_VER) { - OSM_LOG(p_log, OSM_LOG_ERROR, + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Error opening %s: " "%s symbol is the wrong version %d != %d\n", plugin_name, @@ -103,19 +103,18 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) goto Exit; } - if (!rc->impl->construct) { - OSM_LOG(p_log, OSM_LOG_ERROR, - "%s symbol has no construct function\n", + if (!rc->impl->create) { + OSM_LOG(&osm->log, OSM_LOG_ERROR, + "%s symbol has no create() function\n", OSM_EVENT_PLUGIN_IMPL_NAME); goto Exit; } - rc->plugin_data = rc->impl->construct(p_log); + rc->plugin_data = rc->impl->create(osm); if (!rc->plugin_data) goto Exit; - rc->p_log = p_log; rc->plugin_name = strdup(plugin_name); return (rc); @@ -129,8 +128,8 @@ DLOPENFAIL: void osm_epi_destroy(osm_epi_plugin_t * plugin) { if (plugin) { - if (plugin->impl->destroy) - plugin->impl->destroy(plugin->plugin_data); + if (plugin->impl->delete) + plugin->impl->delete(plugin->plugin_data); dlclose(plugin->handle); free(plugin->plugin_name); free(plugin); diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 0101b97..6cf4726 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -251,7 +251,7 @@ static void load_plugins(osm_opensm_t *osm, char *plugin_names) name = strtok_r(plugin_names, " \t\n", &p); while (name && *name) { - epi = osm_epi_construct(&osm->log, name); + epi = osm_epi_construct(osm, name); if (!epi) osm_log(&osm->log, OSM_LOG_ERROR, "cannot load plugin \'%s\'\n", name); diff --git a/opensm/osmeventplugin/src/osmeventplugin.c b/opensm/osmeventplugin/src/osmeventplugin.c index 6cc4c70..a56d3c3 100644 --- a/opensm/osmeventplugin/src/osmeventplugin.c +++ b/opensm/osmeventplugin/src/osmeventplugin.c @@ -38,12 +38,15 @@ #include #include #include +#include #include #include #include -#include #include #include +#include +#include +#include /** ========================================================================= * This is a simple example plugin which logs some of the events the OSM @@ -57,8 +60,11 @@ typedef struct _log_events { /** ========================================================================= */ -static void *construct(osm_log_t * osmlog) +static void *construct(osm_opensm_t *osm) { + if (strcmp(osm->osm_version, OSM_VERSION)) + return NULL; + _log_events_t *log = malloc(sizeof(*log)); if (!log) return (NULL); @@ -66,14 +72,14 @@ static void *construct(osm_log_t * osmlog) log->log_file = fopen(SAMPLE_PLUGIN_OUTPUT_FILE, "a+"); if (!(log->log_file)) { - osm_log(osmlog, OSM_LOG_ERROR, + osm_log(&osm->log, OSM_LOG_ERROR, "Sample Event Plugin: Failed to open output file \"%s\"\n", SAMPLE_PLUGIN_OUTPUT_FILE); free(log); return (NULL); } - log->osmlog = osmlog; + log->osmlog = &osm->log; return ((void *)log); } @@ -173,7 +179,7 @@ static void report(void *_log, osm_epi_event_id_t event_id, void *event_data) */ osm_event_plugin_t osm_event_plugin = { interface_version:OSM_EVENT_PLUGIN_INTERFACE_VER, - construct:construct, - destroy:destroy, + create:construct, + delete:destroy, report:report }; -- 1.5.5.1.178.g1f811 From dwilder at us.ibm.com Wed Jun 25 15:24:47 2008 From: dwilder at us.ibm.com (David J. Wilder) Date: Wed, 25 Jun 2008 15:24:47 -0700 Subject: [ofa-general] [PATCH] Corrects a race in ipoib_cm_post_receive_nonsrq() Message-ID: <1214432687.30933.4.camel@dumpserver> This patch corrects a race condition in ipoib_cm_post_receive_nonsrq() which allows wqes from one QP context to be post_recv to another QP context. Thanks to Hong-Nam Nguyen for finding this bug Signed-off-by: Pradeep Satyanarayana Signed-off-by: David Wilder --- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 18 ++++++++++++++---- 1 files changed, 14 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 97e67d3..8a1f92b 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -116,15 +116,25 @@ static int ipoib_cm_post_receive_nonsrq(struct net_device *dev, struct ipoib_cm_rx *rx, int id) { struct ipoib_dev_priv *priv = netdev_priv(dev); - struct ib_recv_wr *bad_wr; + struct ib_recv_wr *bad_wr, rx_wr; + struct ib_sge rx_sge[IPOIB_CM_RX_SG]; int i, ret; - priv->cm.rx_wr.wr_id = id | IPOIB_OP_CM | IPOIB_OP_RECV; + rx_sge[0].length = IPOIB_CM_HEAD_SIZE; + rx_sge[0].lkey = priv->mr->lkey; + for (i = 1; i < IPOIB_CM_RX_SG; ++i) { + rx_sge[i].length = PAGE_SIZE; + rx_sge[i].lkey = priv->mr->lkey; + } + rx_wr.next = NULL; + rx_wr.sg_list = rx_sge; + rx_wr.num_sge = priv->cm.num_frags; + rx_wr.wr_id = id | IPOIB_OP_CM | IPOIB_OP_RECV; for (i = 0; i < IPOIB_CM_RX_SG; ++i) - priv->cm.rx_sge[i].addr = rx->rx_ring[id].mapping[i]; + rx_sge[i].addr = rx->rx_ring[id].mapping[i]; - ret = ib_post_recv(rx->qp, &priv->cm.rx_wr, &bad_wr); + ret = ib_post_recv(rx->qp, &rx_wr, &bad_wr); if (unlikely(ret)) { ipoib_warn(priv, "post recv failed for buf %d (%d)\n", id, ret); ipoib_cm_dma_unmap_rx(priv, IPOIB_CM_RX_SG - 1, From rdreier at cisco.com Wed Jun 25 15:26:07 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 15:26:07 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: <1214382352.23583.54.camel@mtls03> (Eli Cohen's message of "Wed, 25 Jun 2008 11:25:52 +0300") References: <1214382352.23583.54.camel@mtls03> Message-ID: Actually I'm beginning to wonder what's going on here. The IB spec says a "last WQE reached" event is generated when a CQE is generated for the last WQE. Is this different from the CQE actually being in the CQ? This is probably a question for Eli or other Mellanox guys: is it possible for the driver to see a "last WQE reached" event before all CQEs are actually visible with mthca HW -- ie is the scenario below with two QPs attached to an SRQ: last WQE event for QP A last WQE event for QP B post send to QP that will complete with flush err then poll CQEs and see the send completion from QP A followed by a receive completion for QP B actually possible? From sean.hefty at intel.com Wed Jun 25 15:27:29 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 25 Jun 2008 15:27:29 -0700 Subject: [ofa-general] RE: [PATCH] RDMA/cma: Add missing newlines to printk()s In-Reply-To: References: Message-ID: <000201c8d712$a74ccab0$bb37170a@amr.corp.intel.com> >Signed-off-by: Roland Dreier thanks for catching this Acked-by: Sean Hefty >--- > drivers/infiniband/core/cma.c | 6 +++--- > 1 files changed, 3 insertions(+), 3 deletions(-) > >diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c >index e5bd617..44d190f 100644 >--- a/drivers/infiniband/core/cma.c >+++ b/drivers/infiniband/core/cma.c >@@ -974,7 +974,7 @@ static int cma_ib_handler(struct ib_cm_id *cm_id, struct >ib_cm_event *ib_event) > event.param.conn.private_data_len = IB_CM_REJ_PRIVATE_DATA_SIZE; > break; > default: >- printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d", >+ printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d\n", > ib_event->event); > goto out; > } >@@ -1450,7 +1450,7 @@ static void cma_listen_on_dev(struct rdma_id_private >*id_priv, > ret = rdma_listen(id, id_priv->backlog); > if (ret) > printk(KERN_WARNING "RDMA CMA: cma_listen_on_dev, error %d, " >- "listening on device %s", ret, cma_dev->device->name); >+ "listening on device %s\n", ret, cma_dev->device->name); > } > > static void cma_listen_on_all(struct rdma_id_private *id_priv) >@@ -2155,7 +2155,7 @@ static int cma_sidr_rep_handler(struct ib_cm_id *cm_id, > event.status = 0; > break; > default: >- printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d", >+ printk(KERN_ERR "RDMA CMA: unexpected IB CM event: %d\n", > ib_event->event); > goto out; > } From rdreier at cisco.com Wed Jun 25 15:30:00 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 15:30:00 -0700 Subject: [ofa-general] Re: [PATCH] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: <1214432687.30933.4.camel@dumpserver> (David J. Wilder's message of "Wed, 25 Jun 2008 15:24:47 -0700") References: <1214432687.30933.4.camel@dumpserver> Message-ID: > This patch corrects a race condition in ipoib_cm_post_receive_nonsrq() > which allows wqes from one QP context to be post_recv > to another QP context. Thanks to Hong-Nam Nguyen for finding this bug Please be explicit about what the race is, so that I don't have to repeat your debugging work. > Signed-off-by: Pradeep Satyanarayana How does Pradeep figure into this? Is he the original author of this patch? By default when I import this into git, the From: line of your email will be used as the author, so you (David) will be the author. If you are sending a patch that someone else wrote, please include a line in the body of your email like From: Pradeep Satyanarayana showing the original author. - R. From xma at us.ibm.com Wed Jun 25 15:53:24 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 15:53:24 -0700 Subject: [ofa-general] Re: [PATCH] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: Message-ID: Roland, It's a team work debugging and testing effort. The problem is for each connection, ipoib_cm_post_receive_nonsrq() gets called, it uses a shared structure, which overwrite by each other. So one QP's post_recv will go to another QP context. Thanks Shirley Roland Dreier cc Sent by: general at lists.openfabrics.org general-b Subject ounces at li [ofa-general] Re: [PATCH] Corrects a race sts.openf in ipoib_cm_post_receive_nonsrq() abrics.or g 06/25/08 03:30 PM > This patch corrects a race condition in ipoib_cm_post_receive_nonsrq() > which allows wqes from one QP context to be post_recv > to another QP context. Thanks to Hong-Nam Nguyen for finding this bug Please be explicit about what the race is, so that I don't have to repeat your debugging work. > Signed-off-by: Pradeep Satyanarayana How does Pradeep figure into this? Is he the original author of this patch? By default when I import this into git, the From: line of your email will be used as the author, so you (David) will be the author. If you are sending a patch that someone else wrote, please include a line in the body of your email like From: Pradeep Satyanarayana showing the original author. - R. _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pic16749.gif Type: image/gif Size: 1255 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ecblank.gif Type: image/gif Size: 45 bytes Desc: not available URL: From dwilder at us.ibm.com Wed Jun 25 16:08:19 2008 From: dwilder at us.ibm.com (David J. Wilder) Date: Wed, 25 Jun 2008 16:08:19 -0700 Subject: [ofa-general] [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() Message-ID: <1214435300.30933.14.camel@dumpserver> Corrects a race condition in ipoib_cm_post_receive_nonsrq() which allows wqes from one QP context to be post_recv to another QP context. The ipoib_cm_post_receive_nonsrq() saves the wr_id in the shared structure ipoib_cm_dev_priv making it possible for the saved wr_id to be overwritten by a subsequent event and posting to the incorrect qp context. The patch switches to a local variable to save the wr_id. Thanks to Hong-Nam Nguyen for finding this bug. Signed-off-by: Pradeep Satyanarayana Signed-off-by: David Wilder --- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 18 ++++++++++++++---- 1 files changed, 14 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 97e67d3..8a1f92b 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -116,15 +116,25 @@ static int ipoib_cm_post_receive_nonsrq(struct net_device *dev, struct ipoib_cm_rx *rx, int id) { struct ipoib_dev_priv *priv = netdev_priv(dev); - struct ib_recv_wr *bad_wr; + struct ib_recv_wr *bad_wr, rx_wr; + struct ib_sge rx_sge[IPOIB_CM_RX_SG]; int i, ret; - priv->cm.rx_wr.wr_id = id | IPOIB_OP_CM | IPOIB_OP_RECV; + rx_sge[0].length = IPOIB_CM_HEAD_SIZE; + rx_sge[0].lkey = priv->mr->lkey; + for (i = 1; i < IPOIB_CM_RX_SG; ++i) { + rx_sge[i].length = PAGE_SIZE; + rx_sge[i].lkey = priv->mr->lkey; + } + rx_wr.next = NULL; + rx_wr.sg_list = rx_sge; + rx_wr.num_sge = priv->cm.num_frags; + rx_wr.wr_id = id | IPOIB_OP_CM | IPOIB_OP_RECV; for (i = 0; i < IPOIB_CM_RX_SG; ++i) - priv->cm.rx_sge[i].addr = rx->rx_ring[id].mapping[i]; + rx_sge[i].addr = rx->rx_ring[id].mapping[i]; - ret = ib_post_recv(rx->qp, &priv->cm.rx_wr, &bad_wr); + ret = ib_post_recv(rx->qp, &rx_wr, &bad_wr); if (unlikely(ret)) { ipoib_warn(priv, "post recv failed for buf %d (%d)\n", id, ret); ipoib_cm_dma_unmap_rx(priv, IPOIB_CM_RX_SG - 1, From weiny2 at llnl.gov Wed Jun 25 16:09:34 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 25 Jun 2008 16:09:34 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080624192251.38214cf1.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> Message-ID: <20080625160934.5025c073.weiny2@llnl.gov> On Tue, 24 Jun 2008 19:22:51 -0700 Ira Weiny wrote: > On Wed, 25 Jun 2008 10:33:13 +1000 > Max Matveev wrote: > > > Same as above. In fact, consider factoring that bit out. > > Ah, yes... > > Actually... there is already a portid2str functions... :-( > > I will have to leave this till tomorrow to make sure I don't mess anything else up tonight. > Included are 2 patches which accomplish the same thing "print warning when _do_madrpc fails with destination port information". However, they now use the existing portid2str function and modify that to A) print gids in nice ipv6 format and B) print slid,dlid and drpath. Ira >From ec5c867213d58a2a297a2ca817b50266f55a35bd Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Tue, 24 Jun 2008 14:43:24 -0700 Subject: [PATCH] libibmad: print warning containing destination port when _do_madrpc fails Signed-off-by: Ira K. Weiny --- libibmad/src/rpc.c | 14 ++++++++++---- 1 files changed, 10 insertions(+), 4 deletions(-) diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 15543b2..a2cec95 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -185,13 +185,16 @@ mad_rpc(const void *port_id, ib_rpc_t *rpc, ib_portid_t *dport, void *payload, if ((len = _do_madrpc(p->port_id, sndbuf, rcvbuf, p->class_agents[rpc->mgtclass], - len, rpc->timeout)) < 0) + len, rpc->timeout)) < 0) { + IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); return 0; + } mad = umad_get_mad(rcvbuf); if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { - ERRS("MAD completed with error status 0x%x", status); + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); return 0; } @@ -223,13 +226,16 @@ mad_rpc_rmpp(const void *port_id, ib_rpc_t *rpc, ib_portid_t *dport, if ((len = _do_madrpc(p->port_id, sndbuf, rcvbuf, p->class_agents[rpc->mgtclass], - len, rpc->timeout)) < 0) + len, rpc->timeout)) < 0) { + IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); return 0; + } mad = umad_get_mad(rcvbuf); if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { - ERRS("MAD completed with error status 0x%x", status); + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); return 0; } -- 1.5.1 >From 31c73fc09fab80e599559e5de501708dc0f0899e Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Wed, 25 Jun 2008 15:08:13 -0700 Subject: [PATCH] libibmad: update portid2str to use inet_ntop and drpath2str functions for gid and drpath printing respectively Signed-off-by: Ira K. Weiny --- libibmad/src/portid.c | 18 +++++++----------- 1 files changed, 7 insertions(+), 11 deletions(-) diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c index 7279e14..01f9530 100644 --- a/libibmad/src/portid.c +++ b/libibmad/src/portid.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include @@ -65,29 +66,24 @@ char * portid2str(ib_portid_t *portid) { static char buf[1024] = "local"; + char drpath[512]; char *s = buf; int i; if (portid->lid > 0) { s += sprintf(s, "Lid %d", portid->lid); if (portid->grh_present) { - s += sprintf(s, " Gid 0x%" PRIx64 "%" PRIx64, - ntohll(*(uint64_t *)portid->gid), - ntohll(*(uint64_t *)(portid->gid+8))); + char gid[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; + inet_ntop(AF_INET6, portid->gid, gid, sizeof(gid)); + s += sprintf(s, " Gid %s", gid); } if (portid->drpath.cnt) s += sprintf(s, " "); else return buf; } - s += sprintf(s, "DR path "); - for (i = 0; i < portid->drpath.cnt+1; i++) { - if (i == 0) - s += sprintf( s, "%d", portid->drpath.p[i] ); - else - s += sprintf( s, ",%d", portid->drpath.p[i] ); - } - + s += sprintf(s, "DR path %s", + drpath2str(&(portid->drpath), drpath, sizeof(drpath))); return buf; } -- 1.5.1 From xma at us.ibm.com Wed Jun 25 16:33:25 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 16:33:25 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 03:26:07 PM: > This is probably a question for Eli or other Mellanox guys: is it > possible for the driver to see a "last WQE reached" event before all > CQEs are actually visible with mthca HW -- ie is the scenario below with > two QPs attached to an SRQ: > > last WQE event for QP A > last WQE event for QP B > post send to QP that will complete with flush err > > then poll CQEs and see the send completion from QP A followed by a > receive completion for QP B actually possible? My understand it could happen from the comments under static void ipoib_cm_start_rx_drain() { /* * QPs on flush list are error state. This way, a "flush * error" WC will be immediately generated for each WR we post. */ p = list_entry(priv->cm.rx_flush_list.next, typeof(*p), list); if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) ipoib_warn(priv, "failed to post drain wr\n"); ... } Eli, Please clarify here. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrea at qumranet.com Wed Jun 25 17:26:57 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 26 Jun 2008 02:26:57 +0200 Subject: [ofa-general] [PATCH 1 of 3] list_del_init_rcu In-Reply-To: Message-ID: <5e8c41d283ccef7c739b.1214440017@duo.random> From: Andrea Arcangeli Introduces list_del_init_rcu and documents it (fixes a comment for list_del_rcu too). Signed-off-by: Andrea Arcangeli Acked-by: Linus Torvalds --- diff -r 98f755616212 -r 5e8c41d283cc include/linux/list.h --- a/include/linux/list.h Tue Jun 24 11:23:35 2008 -0700 +++ b/include/linux/list.h Wed Jun 25 03:34:11 2008 +0200 @@ -747,7 +747,7 @@ static inline void hlist_del(struct hlis * or hlist_del_rcu(), running on this same list. * However, it is perfectly legal to run concurrently with * the _rcu list-traversal primitives, such as - * hlist_for_each_entry(). + * hlist_for_each_entry_rcu(). */ static inline void hlist_del_rcu(struct hlist_node *n) { @@ -760,6 +760,34 @@ static inline void hlist_del_init(struct if (!hlist_unhashed(n)) { __hlist_del(n); INIT_HLIST_NODE(n); + } +} + +/** + * hlist_del_init_rcu - deletes entry from hash list with re-initialization + * @n: the element to delete from the hash list. + * + * Note: list_unhashed() on the node return true after this. It is + * useful for RCU based read lockfree traversal if the writer side + * must know if the list entry is still hashed or already unhashed. + * + * In particular, it means that we can not poison the forward pointers + * that may still be used for walking the hash list and we can only + * zero the pprev pointer so list_unhashed() will return true after + * this. + * + * The caller must take whatever precautions are necessary (such as + * holding appropriate locks) to avoid racing with another + * list-mutation primitive, such as hlist_add_head_rcu() or + * hlist_del_rcu(), running on this same list. However, it is + * perfectly legal to run concurrently with the _rcu list-traversal + * primitives, such as hlist_for_each_entry_rcu(). + */ +static inline void hlist_del_init_rcu(struct hlist_node *n) +{ + if (!hlist_unhashed(n)) { + __hlist_del(n); + n->pprev = NULL; } } From andrea at qumranet.com Wed Jun 25 17:26:56 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 26 Jun 2008 02:26:56 +0200 Subject: [ofa-general] [PATCH 0 of 3] mmu notifier v18 for -mm Message-ID: Hello, Christoph suggested me to repost v18 for merging in -mm, to give it more exposure before the .27 merge window opens. There's no code change compared to the previous v18 submission (the only change is the correction in the comment in the mm_take_all_locks patch rightfully pointed out by Linus). Full patchset including other XPMEM support patches can be found here: http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.6/2.6.26-rc7/mmu-notifier-v18 Only the three patches of the patchset I'm submitting here by email are ready for merging, the rest you can find in the website is not ready for merging yet for various performance degradations, lots of the XPMEM patches needs to be elaborated to avoid any slowdown for the non-XPMEM case, but I keep maintaining them to make life easier to XPMEM current development and later we can keep work on them to make them suitable for inclusion to avoid any performance degradation risk. (the fourth patch in the series of the above url, is not strictly relealted to mmu notifiers but it's good at least for me to keep it in the same tree to test pci-passthrough capable guest running on reserved-ram at the same time of two regular guests swapping heavily with mmu notifiers which tends to exercises both spte models at the same time, if you find this confusing I'll remove it from any later upload, but xpmem users can totally ignore it, it only touches x86-64 code) Thanks a lot. Andrea From andrea at qumranet.com Wed Jun 25 17:26:58 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 26 Jun 2008 02:26:58 +0200 Subject: [ofa-general] [PATCH 2 of 3] mm_take_all_locks In-Reply-To: Message-ID: <167f154fa536c2c70c9d.1214440018@duo.random> From: Andrea Arcangeli mm_take_all_locks holds off reclaim from an entire mm_struct. This allows mmu notifiers to register into the mm at any time with the guarantee that no mmu operation is in progress on the mm. This operation locks against the VM for all pte/vma/mm related operations that could ever happen on a certain mm. This includes vmtruncate, try_to_unmap, and all page faults. The caller must take the mmap_sem in write mode before calling mm_take_all_locks(). The caller isn't allowed to release the mmap_sem until mm_drop_all_locks() returns. mmap_sem in write mode is required in order to block all operations that could modify pagetables and free pages without need of altering the vma layout (for example populate_range() with nonlinear vmas). It's also needed in write mode to avoid new anon_vmas to be associated with existing vmas. A single task can't take more than one mm_take_all_locks() in a row or it would deadlock. mm_take_all_locks() and mm_drop_all_locks are expensive operations that may have to take thousand of locks. mm_take_all_locks() can fail if it's interrupted by signals. When mmu_notifier_register returns, we must be sure that the driver is notified if some task is in the middle of a vmtruncate for the 'mm' where the mmu notifier was registered (mmu_notifier_invalidate_range_start/end is run around the vmtruncation but mmu_notifier_register can run after mmu_notifier_invalidate_range_start and before mmu_notifier_invalidate_range_end). Same problem for rmap paths. And we've to remove page pinning to avoid replicating the tlb_gather logic inside KVM (and GRU doesn't work well with page pinning regardless of needing tlb_gather), so without mm_take_all_locks when vmtruncate frees the page, kvm would have no way to notice that it mapped into sptes a page that is going into the freelist without a chance of any further mmu_notifier notification. Signed-off-by: Andrea Arcangeli Acked-by: Linus Torvalds --- diff -r 5e8c41d283cc -r 167f154fa536 include/linux/mm.h --- a/include/linux/mm.h Wed Jun 25 03:34:11 2008 +0200 +++ b/include/linux/mm.h Wed Jun 25 03:34:14 2008 +0200 @@ -1068,6 +1068,9 @@ extern struct vm_area_struct *copy_vma(s unsigned long addr, unsigned long len, pgoff_t pgoff); extern void exit_mmap(struct mm_struct *); +extern int mm_take_all_locks(struct mm_struct *mm); +extern void mm_drop_all_locks(struct mm_struct *mm); + #ifdef CONFIG_PROC_FS /* From fs/proc/base.c. callers must _not_ hold the mm's exe_file_lock */ extern void added_exe_file_vma(struct mm_struct *mm); diff -r 5e8c41d283cc -r 167f154fa536 include/linux/pagemap.h --- a/include/linux/pagemap.h Wed Jun 25 03:34:11 2008 +0200 +++ b/include/linux/pagemap.h Wed Jun 25 03:34:14 2008 +0200 @@ -19,6 +19,7 @@ */ #define AS_EIO (__GFP_BITS_SHIFT + 0) /* IO error on async write */ #define AS_ENOSPC (__GFP_BITS_SHIFT + 1) /* ENOSPC on async write */ +#define AS_MM_ALL_LOCKS (__GFP_BITS_SHIFT + 2) /* under mm_take_all_locks() */ static inline void mapping_set_error(struct address_space *mapping, int error) { diff -r 5e8c41d283cc -r 167f154fa536 include/linux/rmap.h --- a/include/linux/rmap.h Wed Jun 25 03:34:11 2008 +0200 +++ b/include/linux/rmap.h Wed Jun 25 03:34:14 2008 +0200 @@ -26,6 +26,14 @@ */ struct anon_vma { spinlock_t lock; /* Serialize access to vma list */ + /* + * NOTE: the LSB of the head.next is set by + * mm_take_all_locks() _after_ taking the above lock. So the + * head must only be read/written after taking the above lock + * to be sure to see a valid next pointer. The LSB bit itself + * is serialized by a system wide lock only visible to + * mm_take_all_locks() (mm_all_locks_mutex). + */ struct list_head head; /* List of private "related" vmas */ }; diff -r 5e8c41d283cc -r 167f154fa536 mm/mmap.c --- a/mm/mmap.c Wed Jun 25 03:34:11 2008 +0200 +++ b/mm/mmap.c Wed Jun 25 03:34:14 2008 +0200 @@ -2261,3 +2261,161 @@ int install_special_mapping(struct mm_st return 0; } + +static DEFINE_MUTEX(mm_all_locks_mutex); + +static void vm_lock_anon_vma(struct anon_vma *anon_vma) +{ + if (!test_bit(0, (unsigned long *) &anon_vma->head.next)) { + /* + * The LSB of head.next can't change from under us + * because we hold the mm_all_locks_mutex. + */ + spin_lock(&anon_vma->lock); + /* + * We can safely modify head.next after taking the + * anon_vma->lock. If some other vma in this mm shares + * the same anon_vma we won't take it again. + * + * No need of atomic instructions here, head.next + * can't change from under us thanks to the + * anon_vma->lock. + */ + if (__test_and_set_bit(0, (unsigned long *) + &anon_vma->head.next)) + BUG(); + } +} + +static void vm_lock_mapping(struct address_space *mapping) +{ + if (!test_bit(AS_MM_ALL_LOCKS, &mapping->flags)) { + /* + * AS_MM_ALL_LOCKS can't change from under us because + * we hold the mm_all_locks_mutex. + * + * Operations on ->flags have to be atomic because + * even if AS_MM_ALL_LOCKS is stable thanks to the + * mm_all_locks_mutex, there may be other cpus + * changing other bitflags in parallel to us. + */ + if (test_and_set_bit(AS_MM_ALL_LOCKS, &mapping->flags)) + BUG(); + spin_lock(&mapping->i_mmap_lock); + } +} + +/* + * This operation locks against the VM for all pte/vma/mm related + * operations that could ever happen on a certain mm. This includes + * vmtruncate, try_to_unmap, and all page faults. + * + * The caller must take the mmap_sem in write mode before calling + * mm_take_all_locks(). The caller isn't allowed to release the + * mmap_sem until mm_drop_all_locks() returns. + * + * mmap_sem in write mode is required in order to block all operations + * that could modify pagetables and free pages without need of + * altering the vma layout (for example populate_range() with + * nonlinear vmas). It's also needed in write mode to avoid new + * anon_vmas to be associated with existing vmas. + * + * A single task can't take more than one mm_take_all_locks() in a row + * or it would deadlock. + * + * The LSB in anon_vma->head.next and the AS_MM_ALL_LOCKS bitflag in + * mapping->flags avoid to take the same lock twice, if more than one + * vma in this mm is backed by the same anon_vma or address_space. + * + * We can take all the locks in random order because the VM code + * taking i_mmap_lock or anon_vma->lock outside the mmap_sem never + * takes more than one of them in a row. Secondly we're protected + * against a concurrent mm_take_all_locks() by the mm_all_locks_mutex. + * + * mm_take_all_locks() and mm_drop_all_locks are expensive operations + * that may have to take thousand of locks. + * + * mm_take_all_locks() can fail if it's interrupted by signals. + */ +int mm_take_all_locks(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + int ret = -EINTR; + + BUG_ON(down_read_trylock(&mm->mmap_sem)); + + mutex_lock(&mm_all_locks_mutex); + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (signal_pending(current)) + goto out_unlock; + if (vma->anon_vma) + vm_lock_anon_vma(vma->anon_vma); + if (vma->vm_file && vma->vm_file->f_mapping) + vm_lock_mapping(vma->vm_file->f_mapping); + } + ret = 0; + +out_unlock: + if (ret) + mm_drop_all_locks(mm); + + return ret; +} + +static void vm_unlock_anon_vma(struct anon_vma *anon_vma) +{ + if (test_bit(0, (unsigned long *) &anon_vma->head.next)) { + /* + * The LSB of head.next can't change to 0 from under + * us because we hold the mm_all_locks_mutex. + * + * We must however clear the bitflag before unlocking + * the vma so the users using the anon_vma->head will + * never see our bitflag. + * + * No need of atomic instructions here, head.next + * can't change from under us until we release the + * anon_vma->lock. + */ + if (!__test_and_clear_bit(0, (unsigned long *) + &anon_vma->head.next)) + BUG(); + spin_unlock(&anon_vma->lock); + } +} + +static void vm_unlock_mapping(struct address_space *mapping) +{ + if (test_bit(AS_MM_ALL_LOCKS, &mapping->flags)) { + /* + * AS_MM_ALL_LOCKS can't change to 0 from under us + * because we hold the mm_all_locks_mutex. + */ + spin_unlock(&mapping->i_mmap_lock); + if (!test_and_clear_bit(AS_MM_ALL_LOCKS, + &mapping->flags)) + BUG(); + } +} + +/* + * The mmap_sem cannot be released by the caller until + * mm_drop_all_locks() returns. + */ +void mm_drop_all_locks(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + + BUG_ON(down_read_trylock(&mm->mmap_sem)); + BUG_ON(!mutex_is_locked(&mm_all_locks_mutex)); + + for (vma = mm->mmap; vma; vma = vma->vm_next) { + if (vma->anon_vma) + vm_unlock_anon_vma(vma->anon_vma); + if (vma->vm_file && vma->vm_file->f_mapping) + vm_unlock_mapping(vma->vm_file->f_mapping); + } + + mutex_unlock(&mm_all_locks_mutex); +} From andrea at qumranet.com Wed Jun 25 17:26:59 2008 From: andrea at qumranet.com (Andrea Arcangeli) Date: Thu, 26 Jun 2008 02:26:59 +0200 Subject: [ofa-general] [PATCH 3 of 3] mmu-notifier-core In-Reply-To: Message-ID: <6fd7f58efb21ee00473f.1214440019@duo.random> From: Andrea Arcangeli With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages. There are secondary MMUs (with secondary sptes and secondary tlbs) too. sptes in the kvm case are shadow pagetables, but when I say spte in mmu-notifier context, I mean "secondary pte". In GRU case there's no actual secondary pte and there's only a secondary tlb because the GRU secondary MMU has no knowledge about sptes and every secondary tlb miss event in the MMU always generates a page fault that has to be resolved by the CPU (this is not the case of KVM where the a secondary tlb miss will walk sptes in hardware and it will refill the secondary tlb transparently to software if the corresponding spte is present). The same way zap_page_range has to invalidate the pte before freeing the page, the spte (and secondary tlb) must also be invalidated before any page is freed and reused. Currently we take a page_count pin on every page mapped by sptes, but that means the pages can't be swapped whenever they're mapped by any spte because they're part of the guest working set. Furthermore a spte unmap event can immediately lead to a page to be freed when the pin is released (so requiring the same complex and relatively slow tlb_gather smp safe logic we have in zap_page_range and that can be avoided completely if the spte unmap event doesn't require an unpin of the page previously mapped in the secondary MMU). The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know when the VM is swapping or freeing or doing anything on the primary MMU so that the secondary MMU code can drop sptes before the pages are freed, avoiding all page pinning and allowing 100% reliable swapping of guest physical address space. Furthermore it avoids the code that teardown the mappings of the secondary MMU, to implement a logic like tlb_gather in zap_page_range that would require many IPI to flush other cpu tlbs, for each fixed number of spte unmapped. To make an example: if what happens on the primary MMU is a protection downgrade (from writeable to wrprotect) the secondary MMU mappings will be invalidated, and the next secondary-mmu-page-fault will call get_user_pages and trigger a do_wp_page through get_user_pages if it called get_user_pages with write=1, and it'll re-establishing an updated spte or secondary-tlb-mapping on the copied page. Or it will setup a readonly spte or readonly tlb mapping if it's a guest-read, if it calls get_user_pages with write=0. This is just an example. This allows to map any page pointed by any pte (and in turn visible in the primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an full MMU with both sptes and secondary-tlb like the shadow-pagetable layer with kvm), or a remote DMA in software like XPMEM (hence needing of schedule in XPMEM code to send the invalidate to the remote node, while no need to schedule in kvm/gru as it's an immediate event like invalidating primary-mmu pte). At least for KVM without this patch it's impossible to swap guests reliably. And having this feature and removing the page pin allows several other optimizations that simplify life considerably. Dependencies: 1) mm_take_all_locks() to register the mmu notifier when the whole VM isn't doing anything with "mm". This allows mmu notifier users to keep track if the VM is in the middle of the invalidate_range_begin/end critical section with an atomic counter incraese in range_begin and decreased in range_end. No secondary MMU page fault is allowed to map any spte or secondary tlb reference, while the VM is in the middle of range_begin/end as any page returned by get_user_pages in that critical section could later immediately be freed without any further ->invalidate_page notification (invalidate_range_begin/end works on ranges and ->invalidate_page isn't called immediately before freeing the page). To stop all page freeing and pagetable overwrites the mmap_sem must be taken in write mode and all other anon_vma/i_mmap locks must be taken too. 2) It'd be a waste to add branches in the VM if nobody could possibly run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of mmu notifiers, but this already allows to compile a KVM external module against a kernel with mmu notifiers enabled and from the next pull from kvm.git we'll start using them. And GRU/XPMEM will also be able to continue the development by enabling KVM=m in their config, until they submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n). This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM are all =n. The mmu_notifier_register call can fail because mm_take_all_locks may be interrupted by a signal and return -EINTR. Because mmu_notifier_reigster is used when a driver startup, a failure can be gracefully handled. Here an example of the change applied to kvm to register the mmu notifiers. Usually when a driver startups other allocations are required anyway and -ENOMEM failure paths exists already. struct kvm *kvm_arch_create_vm(void) { struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL); + int err; if (!kvm) return ERR_PTR(-ENOMEM); INIT_LIST_HEAD(&kvm->arch.active_mmu_pages); + kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops; + err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm); + if (err) { + kfree(kvm); + return ERR_PTR(err); + } + return kvm; } mmu_notifier_unregister returns void and it's reliable. The patch also adds a few needed but missing includes that would prevent kernel to compile after these changes on non-x86 archs (x86 didn't need them by luck). Signed-off-by: Andrea Arcangeli Signed-off-by: Nick Piggin Signed-off-by: Christoph Lameter --- diff -r 167f154fa536 -r 6fd7f58efb21 arch/x86/kvm/Kconfig --- a/arch/x86/kvm/Kconfig Wed Jun 25 03:34:14 2008 +0200 +++ b/arch/x86/kvm/Kconfig Wed Jun 25 03:34:17 2008 +0200 @@ -21,6 +21,7 @@ config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM select PREEMPT_NOTIFIERS + select MMU_NOTIFIER select ANON_INODES ---help--- Support hosting fully virtualized guest machines using hardware diff -r 167f154fa536 -r 6fd7f58efb21 include/linux/mm_types.h --- a/include/linux/mm_types.h Wed Jun 25 03:34:14 2008 +0200 +++ b/include/linux/mm_types.h Wed Jun 25 03:34:17 2008 +0200 @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -244,6 +245,9 @@ struct mm_struct { struct file *exe_file; unsigned long num_exe_file_vmas; #endif +#ifdef CONFIG_MMU_NOTIFIER + struct mmu_notifier_mm *mmu_notifier_mm; +#endif }; #endif /* _LINUX_MM_TYPES_H */ diff -r 167f154fa536 -r 6fd7f58efb21 include/linux/mmu_notifier.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/include/linux/mmu_notifier.h Wed Jun 25 03:34:17 2008 +0200 @@ -0,0 +1,279 @@ +#ifndef _LINUX_MMU_NOTIFIER_H +#define _LINUX_MMU_NOTIFIER_H + +#include +#include +#include + +struct mmu_notifier; +struct mmu_notifier_ops; + +#ifdef CONFIG_MMU_NOTIFIER + +/* + * The mmu notifier_mm structure is allocated and installed in + * mm->mmu_notifier_mm inside the mm_take_all_locks() protected + * critical section and it's released only when mm_count reaches zero + * in mmdrop(). + */ +struct mmu_notifier_mm { + /* all mmu notifiers registerd in this mm are queued in this list */ + struct hlist_head list; + /* to serialize the list modifications and hlist_unhashed */ + spinlock_t lock; +}; + +struct mmu_notifier_ops { + /* + * Called either by mmu_notifier_unregister or when the mm is + * being destroyed by exit_mmap, always before all pages are + * freed. This can run concurrently with other mmu notifier + * methods (the ones invoked outside the mm context) and it + * should tear down all secondary mmu mappings and freeze the + * secondary mmu. If this method isn't implemented you've to + * be sure that nothing could possibly write to the pages + * through the secondary mmu by the time the last thread with + * tsk->mm == mm exits. + * + * As side note: the pages freed after ->release returns could + * be immediately reallocated by the gart at an alias physical + * address with a different cache model, so if ->release isn't + * implemented because all _software_ driven memory accesses + * through the secondary mmu are terminated by the time the + * last thread of this mm quits, you've also to be sure that + * speculative _hardware_ operations can't allocate dirty + * cachelines in the cpu that could not be snooped and made + * coherent with the other read and write operations happening + * through the gart alias address, so leading to memory + * corruption. + */ + void (*release)(struct mmu_notifier *mn, + struct mm_struct *mm); + + /* + * clear_flush_young is called after the VM is + * test-and-clearing the young/accessed bitflag in the + * pte. This way the VM will provide proper aging to the + * accesses to the page through the secondary MMUs and not + * only to the ones through the Linux pte. + */ + int (*clear_flush_young)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long address); + + /* + * Before this is invoked any secondary MMU is still ok to + * read/write to the page previously pointed to by the Linux + * pte because the page hasn't been freed yet and it won't be + * freed until this returns. If required set_page_dirty has to + * be called internally to this method. + */ + void (*invalidate_page)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long address); + + /* + * invalidate_range_start() and invalidate_range_end() must be + * paired and are called only when the mmap_sem and/or the + * locks protecting the reverse maps are held. The subsystem + * must guarantee that no additional references are taken to + * the pages in the range established between the call to + * invalidate_range_start() and the matching call to + * invalidate_range_end(). + * + * Invalidation of multiple concurrent ranges may be + * optionally permitted by the driver. Either way the + * establishment of sptes is forbidden in the range passed to + * invalidate_range_begin/end for the whole duration of the + * invalidate_range_begin/end critical section. + * + * invalidate_range_start() is called when all pages in the + * range are still mapped and have at least a refcount of one. + * + * invalidate_range_end() is called when all pages in the + * range have been unmapped and the pages have been freed by + * the VM. + * + * The VM will remove the page table entries and potentially + * the page between invalidate_range_start() and + * invalidate_range_end(). If the page must not be freed + * because of pending I/O or other circumstances then the + * invalidate_range_start() callback (or the initial mapping + * by the driver) must make sure that the refcount is kept + * elevated. + * + * If the driver increases the refcount when the pages are + * initially mapped into an address space then either + * invalidate_range_start() or invalidate_range_end() may + * decrease the refcount. If the refcount is decreased on + * invalidate_range_start() then the VM can free pages as page + * table entries are removed. If the refcount is only + * droppped on invalidate_range_end() then the driver itself + * will drop the last refcount but it must take care to flush + * any secondary tlb before doing the final free on the + * page. Pages will no longer be referenced by the linux + * address space but may still be referenced by sptes until + * the last refcount is dropped. + */ + void (*invalidate_range_start)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end); + void (*invalidate_range_end)(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end); +}; + +/* + * The notifier chains are protected by mmap_sem and/or the reverse map + * semaphores. Notifier chains are only changed when all reverse maps and + * the mmap_sem locks are taken. + * + * Therefore notifier chains can only be traversed when either + * + * 1. mmap_sem is held. + * 2. One of the reverse map locks is held (i_mmap_lock or anon_vma->lock). + * 3. No other concurrent thread can access the list (release) + */ +struct mmu_notifier { + struct hlist_node hlist; + const struct mmu_notifier_ops *ops; +}; + +static inline int mm_has_notifiers(struct mm_struct *mm) +{ + return unlikely(mm->mmu_notifier_mm); +} + +extern int mmu_notifier_register(struct mmu_notifier *mn, + struct mm_struct *mm); +extern int __mmu_notifier_register(struct mmu_notifier *mn, + struct mm_struct *mm); +extern void mmu_notifier_unregister(struct mmu_notifier *mn, + struct mm_struct *mm); +extern void __mmu_notifier_mm_destroy(struct mm_struct *mm); +extern void __mmu_notifier_release(struct mm_struct *mm); +extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address); +extern void __mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address); +extern void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end); +extern void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end); + +static inline void mmu_notifier_release(struct mm_struct *mm) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_release(mm); +} + +static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_flush_young(mm, address); + return 0; +} + +static inline void mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_page(mm, address); +} + +static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_range_start(mm, start, end); +} + +static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_invalidate_range_end(mm, start, end); +} + +static inline void mmu_notifier_mm_init(struct mm_struct *mm) +{ + mm->mmu_notifier_mm = NULL; +} + +static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) +{ + if (mm_has_notifiers(mm)) + __mmu_notifier_mm_destroy(mm); +} + +/* + * These two macros will sometime replace ptep_clear_flush. + * ptep_clear_flush is impleemnted as macro itself, so this also is + * implemented as a macro until ptep_clear_flush will converted to an + * inline function, to diminish the risk of compilation failure. The + * invalidate_page method over time can be moved outside the PT lock + * and these two macros can be later removed. + */ +#define ptep_clear_flush_notify(__vma, __address, __ptep) \ +({ \ + pte_t __pte; \ + struct vm_area_struct *___vma = __vma; \ + unsigned long ___address = __address; \ + __pte = ptep_clear_flush(___vma, ___address, __ptep); \ + mmu_notifier_invalidate_page(___vma->vm_mm, ___address); \ + __pte; \ +}) + +#define ptep_clear_flush_young_notify(__vma, __address, __ptep) \ +({ \ + int __young; \ + struct vm_area_struct *___vma = __vma; \ + unsigned long ___address = __address; \ + __young = ptep_clear_flush_young(___vma, ___address, __ptep); \ + __young |= mmu_notifier_clear_flush_young(___vma->vm_mm, \ + ___address); \ + __young; \ +}) + +#else /* CONFIG_MMU_NOTIFIER */ + +static inline void mmu_notifier_release(struct mm_struct *mm) +{ +} + +static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address) +{ + return 0; +} + +static inline void mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address) +{ +} + +static inline void mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline void mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline void mmu_notifier_mm_init(struct mm_struct *mm) +{ +} + +static inline void mmu_notifier_mm_destroy(struct mm_struct *mm) +{ +} + +#define ptep_clear_flush_young_notify ptep_clear_flush_young +#define ptep_clear_flush_notify ptep_clear_flush + +#endif /* CONFIG_MMU_NOTIFIER */ + +#endif /* _LINUX_MMU_NOTIFIER_H */ diff -r 167f154fa536 -r 6fd7f58efb21 kernel/fork.c --- a/kernel/fork.c Wed Jun 25 03:34:14 2008 +0200 +++ b/kernel/fork.c Wed Jun 25 03:34:17 2008 +0200 @@ -54,6 +54,7 @@ #include #include #include +#include #include #include @@ -386,6 +387,7 @@ static struct mm_struct * mm_init(struct if (likely(!mm_alloc_pgd(mm))) { mm->def_flags = 0; + mmu_notifier_mm_init(mm); return mm; } @@ -418,6 +420,7 @@ void __mmdrop(struct mm_struct *mm) BUG_ON(mm == &init_mm); mm_free_pgd(mm); destroy_context(mm); + mmu_notifier_mm_destroy(mm); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); diff -r 167f154fa536 -r 6fd7f58efb21 mm/Kconfig --- a/mm/Kconfig Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/Kconfig Wed Jun 25 03:34:17 2008 +0200 @@ -205,3 +205,6 @@ config VIRT_TO_BUS config VIRT_TO_BUS def_bool y depends on !ARCH_NO_VIRT_TO_BUS + +config MMU_NOTIFIER + bool diff -r 167f154fa536 -r 6fd7f58efb21 mm/Makefile --- a/mm/Makefile Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/Makefile Wed Jun 25 03:34:17 2008 +0200 @@ -33,4 +33,5 @@ obj-$(CONFIG_SMP) += allocpercpu.o obj-$(CONFIG_SMP) += allocpercpu.o obj-$(CONFIG_QUICKLIST) += quicklist.o obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o +obj-$(CONFIG_MMU_NOTIFIER) += mmu_notifier.o diff -r 167f154fa536 -r 6fd7f58efb21 mm/filemap_xip.c --- a/mm/filemap_xip.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/filemap_xip.c Wed Jun 25 03:34:17 2008 +0200 @@ -188,7 +188,7 @@ __xip_unmap (struct address_space * mapp if (pte) { /* Nuke the page table entry. */ flush_cache_page(vma, address, pte_pfn(*pte)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); page_remove_rmap(page, vma); dec_mm_counter(mm, file_rss); BUG_ON(pte_dirty(pteval)); diff -r 167f154fa536 -r 6fd7f58efb21 mm/fremap.c --- a/mm/fremap.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/fremap.c Wed Jun 25 03:34:17 2008 +0200 @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -214,7 +215,9 @@ asmlinkage long sys_remap_file_pages(uns spin_unlock(&mapping->i_mmap_lock); } + mmu_notifier_invalidate_range_start(mm, start, start + size); err = populate_range(mm, vma, start, size, pgoff); + mmu_notifier_invalidate_range_end(mm, start, start + size); if (!err && !(flags & MAP_NONBLOCK)) { if (unlikely(has_write_lock)) { downgrade_write(&mm->mmap_sem); diff -r 167f154fa536 -r 6fd7f58efb21 mm/hugetlb.c --- a/mm/hugetlb.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/hugetlb.c Wed Jun 25 03:34:17 2008 +0200 @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -823,6 +824,7 @@ void __unmap_hugepage_range(struct vm_ar BUG_ON(start & ~HPAGE_MASK); BUG_ON(end & ~HPAGE_MASK); + mmu_notifier_invalidate_range_start(mm, start, end); spin_lock(&mm->page_table_lock); for (address = start; address < end; address += HPAGE_SIZE) { ptep = huge_pte_offset(mm, address); @@ -843,6 +845,7 @@ void __unmap_hugepage_range(struct vm_ar } spin_unlock(&mm->page_table_lock); flush_tlb_range(vma, start, end); + mmu_notifier_invalidate_range_end(mm, start, end); list_for_each_entry_safe(page, tmp, &page_list, lru) { list_del(&page->lru); put_page(page); diff -r 167f154fa536 -r 6fd7f58efb21 mm/memory.c --- a/mm/memory.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/memory.c Wed Jun 25 03:34:17 2008 +0200 @@ -51,6 +51,7 @@ #include #include #include +#include #include #include @@ -649,6 +650,7 @@ int copy_page_range(struct mm_struct *ds unsigned long next; unsigned long addr = vma->vm_start; unsigned long end = vma->vm_end; + int ret; /* * Don't copy ptes where a page fault will fill them correctly. @@ -664,17 +666,33 @@ int copy_page_range(struct mm_struct *ds if (is_vm_hugetlb_page(vma)) return copy_hugetlb_page_range(dst_mm, src_mm, vma); + /* + * We need to invalidate the secondary MMU mappings only when + * there could be a permission downgrade on the ptes of the + * parent mm. And a permission downgrade will only happen if + * is_cow_mapping() returns true. + */ + if (is_cow_mapping(vma->vm_flags)) + mmu_notifier_invalidate_range_start(src_mm, addr, end); + + ret = 0; dst_pgd = pgd_offset(dst_mm, addr); src_pgd = pgd_offset(src_mm, addr); do { next = pgd_addr_end(addr, end); if (pgd_none_or_clear_bad(src_pgd)) continue; - if (copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd, - vma, addr, next)) - return -ENOMEM; + if (unlikely(copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd, + vma, addr, next))) { + ret = -ENOMEM; + break; + } } while (dst_pgd++, src_pgd++, addr = next, addr != end); - return 0; + + if (is_cow_mapping(vma->vm_flags)) + mmu_notifier_invalidate_range_end(src_mm, + vma->vm_start, end); + return ret; } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -878,7 +896,9 @@ unsigned long unmap_vmas(struct mmu_gath unsigned long start = start_addr; spinlock_t *i_mmap_lock = details? details->i_mmap_lock: NULL; int fullmm = (*tlbp)->fullmm; + struct mm_struct *mm = vma->vm_mm; + mmu_notifier_invalidate_range_start(mm, start_addr, end_addr); for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) { unsigned long end; @@ -929,6 +949,7 @@ unsigned long unmap_vmas(struct mmu_gath } } out: + mmu_notifier_invalidate_range_end(mm, start_addr, end_addr); return start; /* which is now the end (or restart) address */ } @@ -1589,10 +1610,11 @@ int apply_to_page_range(struct mm_struct { pgd_t *pgd; unsigned long next; - unsigned long end = addr + size; + unsigned long start = addr, end = addr + size; int err; BUG_ON(addr >= end); + mmu_notifier_invalidate_range_start(mm, start, end); pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); @@ -1600,6 +1622,7 @@ int apply_to_page_range(struct mm_struct if (err) break; } while (pgd++, addr = next, addr != end); + mmu_notifier_invalidate_range_end(mm, start, end); return err; } EXPORT_SYMBOL_GPL(apply_to_page_range); @@ -1800,7 +1823,7 @@ gotten: * seen in the presence of one thread doing SMC and another * thread doing COW. */ - ptep_clear_flush(vma, address, page_table); + ptep_clear_flush_notify(vma, address, page_table); set_pte_at(mm, address, page_table, entry); update_mmu_cache(vma, address, entry); lru_cache_add_active(new_page); diff -r 167f154fa536 -r 6fd7f58efb21 mm/mmap.c --- a/mm/mmap.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/mmap.c Wed Jun 25 03:34:17 2008 +0200 @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -2054,6 +2055,7 @@ void exit_mmap(struct mm_struct *mm) /* mm's last user has gone, and its about to be pulled down */ arch_exit_mmap(mm); + mmu_notifier_release(mm); lru_add_drain(); flush_cache_mm(mm); diff -r 167f154fa536 -r 6fd7f58efb21 mm/mmu_notifier.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/mm/mmu_notifier.c Wed Jun 25 03:34:17 2008 +0200 @@ -0,0 +1,276 @@ +/* + * linux/mm/mmu_notifier.c + * + * Copyright (C) 2008 Qumranet, Inc. + * Copyright (C) 2008 SGI + * Christoph Lameter + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include +#include +#include +#include +#include +#include + +/* + * This function can't run concurrently against mmu_notifier_register + * because mm->mm_users > 0 during mmu_notifier_register and exit_mmap + * runs with mm_users == 0. Other tasks may still invoke mmu notifiers + * in parallel despite there being no task using this mm any more, + * through the vmas outside of the exit_mmap context, such as with + * vmtruncate. This serializes against mmu_notifier_unregister with + * the mmu_notifier_mm->lock in addition to RCU and it serializes + * against the other mmu notifiers with RCU. struct mmu_notifier_mm + * can't go away from under us as exit_mmap holds an mm_count pin + * itself. + */ +void __mmu_notifier_release(struct mm_struct *mm) +{ + struct mmu_notifier *mn; + + spin_lock(&mm->mmu_notifier_mm->lock); + while (unlikely(!hlist_empty(&mm->mmu_notifier_mm->list))) { + mn = hlist_entry(mm->mmu_notifier_mm->list.first, + struct mmu_notifier, + hlist); + /* + * We arrived before mmu_notifier_unregister so + * mmu_notifier_unregister will do nothing other than + * to wait ->release to finish and + * mmu_notifier_unregister to return. + */ + hlist_del_init_rcu(&mn->hlist); + /* + * RCU here will block mmu_notifier_unregister until + * ->release returns. + */ + rcu_read_lock(); + spin_unlock(&mm->mmu_notifier_mm->lock); + /* + * if ->release runs before mmu_notifier_unregister it + * must be handled as it's the only way for the driver + * to flush all existing sptes and stop the driver + * from establishing any more sptes before all the + * pages in the mm are freed. + */ + if (mn->ops->release) + mn->ops->release(mn, mm); + rcu_read_unlock(); + spin_lock(&mm->mmu_notifier_mm->lock); + } + spin_unlock(&mm->mmu_notifier_mm->lock); + + /* + * synchronize_rcu here prevents mmu_notifier_release to + * return to exit_mmap (which would proceed freeing all pages + * in the mm) until the ->release method returns, if it was + * invoked by mmu_notifier_unregister. + * + * The mmu_notifier_mm can't go away from under us because one + * mm_count is hold by exit_mmap. + */ + synchronize_rcu(); +} + +/* + * If no young bitflag is supported by the hardware, ->clear_flush_young can + * unmap the address and return 1 or 0 depending if the mapping previously + * existed or not. + */ +int __mmu_notifier_clear_flush_young(struct mm_struct *mm, + unsigned long address) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + int young = 0; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->clear_flush_young) + young |= mn->ops->clear_flush_young(mn, mm, address); + } + rcu_read_unlock(); + + return young; +} + +void __mmu_notifier_invalidate_page(struct mm_struct *mm, + unsigned long address) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_page) + mn->ops->invalidate_page(mn, mm, address); + } + rcu_read_unlock(); +} + +void __mmu_notifier_invalidate_range_start(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_range_start) + mn->ops->invalidate_range_start(mn, mm, start, end); + } + rcu_read_unlock(); +} + +void __mmu_notifier_invalidate_range_end(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + struct mmu_notifier *mn; + struct hlist_node *n; + + rcu_read_lock(); + hlist_for_each_entry_rcu(mn, n, &mm->mmu_notifier_mm->list, hlist) { + if (mn->ops->invalidate_range_end) + mn->ops->invalidate_range_end(mn, mm, start, end); + } + rcu_read_unlock(); +} + +static int do_mmu_notifier_register(struct mmu_notifier *mn, + struct mm_struct *mm, + int take_mmap_sem) +{ + struct mmu_notifier_mm * mmu_notifier_mm; + int ret; + + BUG_ON(atomic_read(&mm->mm_users) <= 0); + + ret = -ENOMEM; + mmu_notifier_mm = kmalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL); + if (unlikely(!mmu_notifier_mm)) + goto out; + + if (take_mmap_sem) + down_write(&mm->mmap_sem); + ret = mm_take_all_locks(mm); + if (unlikely(ret)) + goto out_cleanup; + + if (!mm_has_notifiers(mm)) { + INIT_HLIST_HEAD(&mmu_notifier_mm->list); + spin_lock_init(&mmu_notifier_mm->lock); + mm->mmu_notifier_mm = mmu_notifier_mm; + mmu_notifier_mm = NULL; + } + atomic_inc(&mm->mm_count); + + /* + * Serialize the update against mmu_notifier_unregister. A + * side note: mmu_notifier_release can't run concurrently with + * us because we hold the mm_users pin (either implicitly as + * current->mm or explicitly with get_task_mm() or similar). + * We can't race against any other mmu notifier method either + * thanks to mm_take_all_locks(). + */ + spin_lock(&mm->mmu_notifier_mm->lock); + hlist_add_head(&mn->hlist, &mm->mmu_notifier_mm->list); + spin_unlock(&mm->mmu_notifier_mm->lock); + + mm_drop_all_locks(mm); +out_cleanup: + if (take_mmap_sem) + up_write(&mm->mmap_sem); + /* kfree() does nothing if mmu_notifier_mm is NULL */ + kfree(mmu_notifier_mm); +out: + BUG_ON(atomic_read(&mm->mm_users) <= 0); + return ret; +} + +/* + * Must not hold mmap_sem nor any other VM related lock when calling + * this registration function. Must also ensure mm_users can't go down + * to zero while this runs to avoid races with mmu_notifier_release, + * so mm has to be current->mm or the mm should be pinned safely such + * as with get_task_mm(). If the mm is not current->mm, the mm_users + * pin should be released by calling mmput after mmu_notifier_register + * returns. mmu_notifier_unregister must be always called to + * unregister the notifier. mm_count is automatically pinned to allow + * mmu_notifier_unregister to safely run at any time later, before or + * after exit_mmap. ->release will always be called before exit_mmap + * frees the pages. + */ +int mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) +{ + return do_mmu_notifier_register(mn, mm, 1); +} +EXPORT_SYMBOL_GPL(mmu_notifier_register); + +/* + * Same as mmu_notifier_register but here the caller must hold the + * mmap_sem in write mode. + */ +int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm) +{ + return do_mmu_notifier_register(mn, mm, 0); +} +EXPORT_SYMBOL_GPL(__mmu_notifier_register); + +/* this is called after the last mmu_notifier_unregister() returned */ +void __mmu_notifier_mm_destroy(struct mm_struct *mm) +{ + BUG_ON(!hlist_empty(&mm->mmu_notifier_mm->list)); + kfree(mm->mmu_notifier_mm); + mm->mmu_notifier_mm = LIST_POISON1; /* debug */ +} + +/* + * This releases the mm_count pin automatically and frees the mm + * structure if it was the last user of it. It serializes against + * running mmu notifiers with RCU and against mmu_notifier_unregister + * with the unregister lock + RCU. All sptes must be dropped before + * calling mmu_notifier_unregister. ->release or any other notifier + * method may be invoked concurrently with mmu_notifier_unregister, + * and only after mmu_notifier_unregister returned we're guaranteed + * that ->release or any other method can't run anymore. + */ +void mmu_notifier_unregister(struct mmu_notifier *mn, struct mm_struct *mm) +{ + BUG_ON(atomic_read(&mm->mm_count) <= 0); + + spin_lock(&mm->mmu_notifier_mm->lock); + if (!hlist_unhashed(&mn->hlist)) { + hlist_del_rcu(&mn->hlist); + + /* + * RCU here will force exit_mmap to wait ->release to finish + * before freeing the pages. + */ + rcu_read_lock(); + spin_unlock(&mm->mmu_notifier_mm->lock); + /* + * exit_mmap will block in mmu_notifier_release to + * guarantee ->release is called before freeing the + * pages. + */ + if (mn->ops->release) + mn->ops->release(mn, mm); + rcu_read_unlock(); + } else + spin_unlock(&mm->mmu_notifier_mm->lock); + + /* + * Wait any running method to finish, of course including + * ->release if it was run by mmu_notifier_relase instead of us. + */ + synchronize_rcu(); + + BUG_ON(atomic_read(&mm->mm_count) <= 0); + + mmdrop(mm); +} +EXPORT_SYMBOL_GPL(mmu_notifier_unregister); diff -r 167f154fa536 -r 6fd7f58efb21 mm/mprotect.c --- a/mm/mprotect.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/mprotect.c Wed Jun 25 03:34:17 2008 +0200 @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -207,10 +208,12 @@ success: dirty_accountable = 1; } + mmu_notifier_invalidate_range_start(mm, start, end); if (is_vm_hugetlb_page(vma)) hugetlb_change_protection(vma, start, end, vma->vm_page_prot); else change_protection(vma, start, end, vma->vm_page_prot, dirty_accountable); + mmu_notifier_invalidate_range_end(mm, start, end); vm_stat_account(mm, oldflags, vma->vm_file, -nrpages); vm_stat_account(mm, newflags, vma->vm_file, nrpages); return 0; diff -r 167f154fa536 -r 6fd7f58efb21 mm/mremap.c --- a/mm/mremap.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/mremap.c Wed Jun 25 03:34:17 2008 +0200 @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -74,7 +75,11 @@ static void move_ptes(struct vm_area_str struct mm_struct *mm = vma->vm_mm; pte_t *old_pte, *new_pte, pte; spinlock_t *old_ptl, *new_ptl; + unsigned long old_start; + old_start = old_addr; + mmu_notifier_invalidate_range_start(vma->vm_mm, + old_start, old_end); if (vma->vm_file) { /* * Subtle point from Rajesh Venkatasubramanian: before @@ -116,6 +121,7 @@ static void move_ptes(struct vm_area_str pte_unmap_unlock(old_pte - 1, old_ptl); if (mapping) spin_unlock(&mapping->i_mmap_lock); + mmu_notifier_invalidate_range_end(vma->vm_mm, old_start, old_end); } #define LATENCY_LIMIT (64 * PAGE_SIZE) diff -r 167f154fa536 -r 6fd7f58efb21 mm/rmap.c --- a/mm/rmap.c Wed Jun 25 03:34:14 2008 +0200 +++ b/mm/rmap.c Wed Jun 25 03:34:17 2008 +0200 @@ -49,6 +49,7 @@ #include #include #include +#include #include @@ -287,7 +288,7 @@ static int page_referenced_one(struct pa if (vma->vm_flags & VM_LOCKED) { referenced++; *mapcount = 1; /* break early from loop */ - } else if (ptep_clear_flush_young(vma, address, pte)) + } else if (ptep_clear_flush_young_notify(vma, address, pte)) referenced++; /* Pretend the page is referenced if the task has the @@ -457,7 +458,7 @@ static int page_mkclean_one(struct page pte_t entry; flush_cache_page(vma, address, pte_pfn(*pte)); - entry = ptep_clear_flush(vma, address, pte); + entry = ptep_clear_flush_notify(vma, address, pte); entry = pte_wrprotect(entry); entry = pte_mkclean(entry); set_pte_at(mm, address, pte, entry); @@ -717,14 +718,14 @@ static int try_to_unmap_one(struct page * skipped over this mm) then we should reactivate it. */ if (!migration && ((vma->vm_flags & VM_LOCKED) || - (ptep_clear_flush_young(vma, address, pte)))) { + (ptep_clear_flush_young_notify(vma, address, pte)))) { ret = SWAP_FAIL; goto out_unmap; } /* Nuke the page table entry. */ flush_cache_page(vma, address, page_to_pfn(page)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); /* Move the dirty bit to the physical page now the pte is gone. */ if (pte_dirty(pteval)) @@ -849,12 +850,12 @@ static void try_to_unmap_cluster(unsigne page = vm_normal_page(vma, address, *pte); BUG_ON(!page || PageAnon(page)); - if (ptep_clear_flush_young(vma, address, pte)) + if (ptep_clear_flush_young_notify(vma, address, pte)) continue; /* Nuke the page table entry. */ flush_cache_page(vma, address, pte_pfn(*pte)); - pteval = ptep_clear_flush(vma, address, pte); + pteval = ptep_clear_flush_notify(vma, address, pte); /* If nonlinear, store the file page offset in the pte. */ if (page->index != linear_page_index(vma, address)) From rdreier at cisco.com Wed Jun 25 18:30:37 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 18:30:37 -0700 Subject: [ofa-general] Re: [PATCH] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 15:53:24 -0700") References: Message-ID: > It's a team work debugging and testing effort. The problem is for > each connection, ipoib_cm_post_receive_nonsrq() gets called, it uses a > shared structure, which overwrite by each other. So one QP's post_recv will > go to another QP context. I don't follow what the first sentence is talking about. Anyway are the connection requests not single-threaded? - R. From rdreier at cisco.com Wed Jun 25 18:33:00 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 18:33:00 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 16:33:25 -0700") References: Message-ID: > My understand it could happen from the comments under > static void ipoib_cm_start_rx_drain() > { > /* > * QPs on flush list are error state. This way, a "flush > * error" WC will be immediately generated for each WR we post. > */ I don't see what this comment has to do with anything... the QP is in the error state so posting a send request will generate a completion with flush error status. But that doesn't say anything about whether earlier completions may become visible in a CQ after a last WQE reached event is reported to the consumer. From rdreier at cisco.com Wed Jun 25 18:44:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Jun 2008 18:44:21 -0700 Subject: [ofa-general] Re: [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: <1214435300.30933.14.camel@dumpserver> (David J. Wilder's message of "Wed, 25 Jun 2008 16:08:19 -0700") References: <1214435300.30933.14.camel@dumpserver> Message-ID: > Corrects a race condition in ipoib_cm_post_receive_nonsrq() > which allows wqes from one QP context to be post_recv > to another QP context. The ipoib_cm_post_receive_nonsrq() > saves the wr_id in the shared structure ipoib_cm_dev_priv > making it possible for the saved wr_id to be overwritten by > a subsequent event and posting to the incorrect qp context. > The patch switches to a local variable to save the wr_id. What subsequent event? Is this due to connection requests coming in and colliding with each other? Or a connection request colliding with posting receives from the completion processing context? > Signed-off-by: Pradeep Satyanarayana I still don't understand what Pradeep's sign-off is doing here... as this email stands, basically what it is saying is that David wrote the patch, then Pradeep sent it to David, and David is sending it to me. Which is nonsensical. Do you just mean that Pradeep was also involved in writing the patch? > - struct ib_recv_wr *bad_wr; > + struct ib_recv_wr *bad_wr, rx_wr; > + struct ib_sge rx_sge[IPOIB_CM_RX_SG]; I worry about this putting an extra 300 bytes or so on the stack... I think it would be nicer if there were a way to just make sure receive posting was single-threaded, but I still don't know which contexts are racing against each other... From xma at us.ibm.com Wed Jun 25 19:21:45 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 19:21:45 -0700 Subject: [ofa-general] Re: [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 06:44:21 PM: > > Corrects a race condition in ipoib_cm_post_receive_nonsrq() > > which allows wqes from one QP context to be post_recv > > to another QP context. The ipoib_cm_post_receive_nonsrq() > > saves the wr_id in the shared structure ipoib_cm_dev_priv > > making it possible for the saved wr_id to be overwritten by > > a subsequent event and posting to the incorrect qp context. > > The patch switches to a local variable to save the wr_id. > > What subsequent event? Is this due to connection requests coming in and > colliding with each other? Or a connection request colliding with > posting receives from the completion processing context? > > > Signed-off-by: Pradeep Satyanarayana > > I still don't understand what Pradeep's sign-off is doing here... as > this email stands, basically what it is saying is that David wrote the > patch, then Pradeep sent it to David, and David is sending it to me. > Which is nonsensical. Do you just mean that Pradeep was also involved > in writing the patch? Actually the debug was from me, wr_id was messed up for each QP for nonSRQ IPoIB-CM. Nam pointed out the code problem in OFED-1.3, and Pradeep wrote the patch for OFED, Dave wrote the patch for mainline kernel. So I suggested them to be the co-author. Is that OK? > > - struct ib_recv_wr *bad_wr; > > + struct ib_recv_wr *bad_wr, rx_wr; > > + struct ib_sge rx_sge[IPOIB_CM_RX_SG]; > > I worry about this putting an extra 300 bytes or so on the stack... I > think it would be nicer if there were a way to just make sure receive > posting was single-threaded, but I still don't know which contexts are > racing against each other... The race can happend is this condition, supposed we have 2 nodes, each node has one port, so we will have 1 active IPoIB-CM nonSRQ connections on each node. Each connection has its own receive queue, it posts its own receive buffer, then some how another node join the same IPoIB subnet, it will build another IPoIB-CM nonSRQ connection, and does post_recv for its own receiving queue, so ipoib_cm_post_receive_nonsrq() will be called between the new connection ipoib_cm_nonsrq_init_rx() and ipoib_cm_handle_rc_wc(). That's the race. It will cause the one QP's post_recv went to another queue pair then caused kernel random panic because the skb buffer messed up. It also happens if any active connection becomes stale and a new connection is requested. In general it's a race between new connection build up and handle_rx_wc() from poll_cq(). Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Wed Jun 25 19:29:51 2008 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 25 Jun 2008 19:29:51 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: Message-ID: Roland Dreier wrote on 06/25/2008 06:33:00 PM: > > My understand it could happen from the comments under > > static void ipoib_cm_start_rx_drain() > > { > > /* > > * QPs on flush list are error state. This way, a "flush > > * error" WC will be immediately generated for each WR we post. > > */ > > I don't see what this comment has to do with anything... the QP is in > the error state so posting a send request will generate a completion > with flush error status. But that doesn't say anything about whether > earlier completions may become visible in a CQ after a last WQE reached > event is reported to the consumer. I thought handling last wqe event drain WRs one by one is more safe. But if there is no such cqes left in this case, then what I hit could be a different bug. More debug is needed. Thanks for helping this. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Wed Jun 25 20:28:46 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 25 Jun 2008 20:28:46 -0700 Subject: [ofa-general] Re: [PATCH] Corrects a racein ipoib_cm_post_receive_nonsrq() In-Reply-To: References: Message-ID: <000001c8d73c$bdf8fc00$78d5180a@amr.corp.intel.com> >Anyway are the connection requests not single-threaded? The ib_cm uses multiple threads, and only serializes the callbacks wrt a single cm_id. - Sean From vlad at dev.mellanox.co.il Wed Jun 25 21:50:50 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Thu, 26 Jun 2008 07:50:50 +0300 Subject: [ofa-general] Re: [PATCH v2 for-2.6.27] IPOIB: add LRO support. In-Reply-To: References: <48611FEB.1030709@dev.mellanox.co.il> <1e336d250806250711l320ec086i8457b4a19cd7102e@mail.gmail.com> Message-ID: <4863202A.2080709@dev.mellanox.co.il> Roland Dreier wrote: > > I don't see that there is LRO support in the userspace ethtool. > > I would guess upstream ethtool will be getting support if it doesn't > have it already. Clearly commit 3ae7c0b2 ("[ETHTOOL]: Add > ETHTOOL_[GS]FLAGS sub-ioctls") in the kernel shows that ethtool is the > right way to control LRO per interface. > The patch I sent yesterday implements ethtool support for switching LRO on/off per device: http://lists.openfabrics.org/pipermail/general/2008-June/052087.html Regards, Vladimir From HNGUYEN at de.ibm.com Wed Jun 25 22:57:17 2008 From: HNGUYEN at de.ibm.com (Hoang-Nam Nguyen) Date: Thu, 26 Jun 2008 07:57:17 +0200 Subject: [ofa-general] [RFC][PATCH] IPoIB-cm: release connection if post_send drain WR failure In-Reply-To: Message-ID: > > I saw it happened from below ipoib debug output on testing ehca2: > > > > static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv) > > { > > .... > > /* > > * QPs on flush list are error state. This way, a "flush > > * error" WC will be immediately generated for each WR we post. > > */ > > p = list_entry(priv->cm.rx_flush_list.next, typeof(*p), list); > > if (ib_post_send(p->qp, &ipoib_cm_rx_drain_wr, &bad_wr)) > > ipoib_warn(priv, "failed to post drain wr\n"); > > Was this with a kernel that had 863fb09f ("IB/ehca: Prevent posting of > SQ WQEs if QP not in RTS") but not 088af154 ("IB/ehca: Reject send WRs > only for RESET, INIT and RTR state")? For example something between > 2.6.25 and 2.6.26-rc4? Right. BTW: It also occurs in ofed-1.3.1. Nam From Sagir at mellanox.co.il Wed Jun 25 23:31:28 2008 From: Sagir at mellanox.co.il (Sagi Rotem) Date: Thu, 26 Jun 2008 09:31:28 +0300 Subject: [ofa-general] RE: msi-x seems to cause large performance variation for connectx / datagram mode In-Reply-To: Message-ID: <40FA0A8088E8A441973D37502F00933E2581C4@mtlexch01.mtl.com> Or, Did you take a look at the cpu utilization while running ? Did u see higher cpu utilization while getting the 850MB/sec ? How many cpu's were actually working ? I will try and reproduce your results. Sagi -----Original Message----- From: Or Gerlitz [mailto:ogerlitz at voltaire.com] Sent: Wednesday, June 25, 2008 6:34 PM To: Sagi Rotem Cc: general at lists.openfabrics.org Subject: Re: msi-x seems to cause large performance variation for connectx / datagram mode > digging a little more, I see now that if I disable msi-x, the > performance gets to a fixed value of about 620MB/s with connectx which > is the lower range that msi-x gives me... setting the irq daemon off, I see that if I set the cpu affinity of the netperf process to be as the one of the mlx4 irq I get fixed performance of 650 MB/s If I set it to different CPU, then eventually I get 480MB/s constantly and 620MB/s constantly if is a different core on the same CPU... HCA MB/s msi_x irqdaemon netperf affinity ======================================================================= ConnectX-ConnectX 620 0 on none ConnectX-ConnectX 620-850 1 on none ConnectX-ConnectX 650 1 off same as irq ConnectX-ConnectX 620 1 off different core ConnectX-ConnectX 480 1 off different cpu Or. From Sumit.Gaur at Sun.COM Wed Jun 25 23:37:58 2008 From: Sumit.Gaur at Sun.COM (Sumit Gaur - Sun Microsystem) Date: Thu, 26 Jun 2008 12:07:58 +0530 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> Message-ID: <48633946.3070706@Sun.COM> Hi Hal, Hal Rosenstock wrote: > >>>> >>>>I am sending only request for >>>> >>>> rpc.mgtclass = IB_PERFORMANCE_CLASS; >>>> rpc.method = IB_MAD_METHOD_GET; >>>> >>>>at every one second. > > > Does perfquery work reliably with the same node(s) you are having > trouble with ? > > Does your app follow what perfquery does ? Yes, perfquery works fine and I am following similar way of implementation. Here is the output. I think difference is there in Load. I am sending 4 GS request per second basis and some got passed and some got timeout(110) or recv failed. # perfquery # Port counters: Lid 393 port 1 PortSelect:......................1 CounterSelect:...................0x0000 SymbolErrors:....................0 LinkRecovers:....................0 LinkDowned:......................0 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 VL15Dropped:.....................0 XmtData:.........................65899728 RcvData:.........................65899656 XmtPkts:.........................915274 RcvPkts:.........................915273 > >>>>>In general, there are a few possibilities (which can cause this). SM >>>>>traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most >>>>>subnets). >>>>> >>>>>Some possibilities are: >>>>>1. Timeout/retry being hit for some GS traffic (GS request or response >>>>>lost/corrupted) >>>> >>>>Yes, this is also happening, Sometimes I am getting corrupt data back, >>> >>> >>>Is there an error indicated ? >> >>For such packets I am getting umad_status as 110. > > > That's ETIMEDOUT. You need to handle the errors (and not treat the > receive as a valid packet). Are you doing that ? yes, I am catching this error. > > The underlying question is why are you getting the timeout relatively > frequently so I recommend checking all the error counters along the > path. # Checking Ca: nodeguid 0x00144fa5e9ce001c Node check lid 392: OK Error check on lid 392 (HCA-1) port all: OK > > Are you sure the request gets to the responder ? Does the responder > respond and it doesn't make it back ? yes As I told It is not 100% failure, It is 30% to 40% failure. But Why ? > > -- Hal > > From ogerlitz at voltaire.com Thu Jun 26 01:27:20 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 26 Jun 2008 11:27:20 +0300 Subject: [ofa-general] Re: msi-x seems to cause large performance variation for connectx / datagram mode In-Reply-To: <40FA0A8088E8A441973D37502F00933E2581C4@mtlexch01.mtl.com> References: <40FA0A8088E8A441973D37502F00933E2581C4@mtlexch01.mtl.com> Message-ID: <486352E8.3080008@voltaire.com> Sagi Rotem wrote: > Did you take a look at the cpu utilization while running ? Did u see higher cpu utilization while getting the 850MB/sec ? How many cpu's were actually working ? The windows of time which the 850 MB/s performance, are kind of hard to track so I can't give a confident answer, from some looking it seems that yes, the CPU utilization gets higher when the bandwidth is high. Also when running two netperf sessions and irq daemon off, I tend to get constant 700 MB/s which is better then the constant 620 MB/s I see with one session. > I will try and reproduce your results. Very much appreciated. Or. From vlad at lists.openfabrics.org Thu Jun 26 02:37:09 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 26 Jun 2008 02:37:09 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080626-0200 daily build status Message-ID: <20080626093709.BB715E60B38@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080626-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From HNGUYEN at de.ibm.com Thu Jun 26 03:24:16 2008 From: HNGUYEN at de.ibm.com (Hoang-Nam Nguyen) Date: Thu, 26 Jun 2008 12:24:16 +0200 Subject: [ofa-general] Re: [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: Message-ID: > > - struct ib_recv_wr *bad_wr; > > + struct ib_recv_wr *bad_wr, rx_wr; > > + struct ib_sge rx_sge[IPOIB_CM_RX_SG]; > > I worry about this putting an extra 300 bytes or so on the stack... I > think it would be nicer if there were a way to just make sure receive > posting was single-threaded, but I still don't know which contexts are > racing against each other... The race basically occurs between ipoib_poll() and ipoib_cm_nonsrq_init_rx() as Shirley mentioned. Below is sort of seq diagram for a possible race: ipoib_poll() ipoib_cm_req_handler() ipoib_cm_handle_rx_wc() ipoib_cm_nonsrq_init_rx() ipoib_cm_post_receive_nonsrq() ipoib_cm_post_receive_nonsrq() priv->cm.rx_wr.wr_id = id <<<< >>>> priv->cm.rx_wr.wr_id = id ib_post_recv() ib_post_recv() Makes sense? Yes, we should avoid allocating those vars on the stack. Either we use same pattern as done for SRQ, ie rx_wr_arr[conn] or something like this: - enhance ipoib_cm_post_receive_nonsrq() with wr and sge passed to by the caller, ie ipoib_poll() -> ipoib_cm_handle_rx_wc() resp. ipoib_cm_nonsrq_init_rx() - ipoib_cm_handle_rx_wc() passes rx_wr and rx_sge as already defined in struct ipoib_cm_dev_priv to ipoib_cm_post_receive_nonsrq() since we use one cq, there should not be a race in this path - ipoib_cm_nonsrq_init_rx() allocates wr and sge on the heap or slabcache, passes them to ipoib_cm_post_receive_nonsrq() and frees them thereafter If you want me to make a patch, glad to do. Nam From eli at mellanox.co.il Thu Jun 26 04:05:50 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Thu, 26 Jun 2008 14:05:50 +0300 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: References: <1214382352.23583.54.camel@mtls03> Message-ID: <1214478350.23583.104.camel@mtls03> On Wed, 2008-06-25 at 15:26 -0700, Roland Dreier wrote: > Actually I'm beginning to wonder what's going on here. > > The IB spec says a "last WQE reached" event is generated when a CQE is > generated for the last WQE. Is this different from the CQE actually > being in the CQ? For all Mellanox HW, first the last CQE is pushed to the CQ and then a last WQE reached event is generated. This means that if you receive such an event, you're guaranteed to see the CQE in the CQ (that is of of course if there is such a CQE; if there where no WQEs related to the QP at the time it entered an error state then the HW just generates the event). > > This is probably a question for Eli or other Mellanox guys: is it > possible for the driver to see a "last WQE reached" event before all > CQEs are actually visible with mthca HW -- ie is the scenario below with > two QPs attached to an SRQ: > > last WQE event for QP A > last WQE event for QP B > post send to QP that will complete with flush err > > then poll CQEs and see the send completion from QP A followed by a > receive completion for QP B actually possible? This is not possible since when you post the send WQE, any CQE related to QP A or QP B are already in the CQ so the send CQE is the last one with respect to those two QPs. So what you're saying is that the scenario Shirley described is not possible... Shirley, did you see this on Mellanox HCAs too or just ehca? From ronli at voltaire.com Thu Jun 26 07:34:46 2008 From: ronli at voltaire.com (Ron Livne) Date: Thu, 26 Jun 2008 14:34:46 +0000 (UTC) Subject: [ofa-general] [PATCH 2/3 v3] mlx4: enable discarding/passing multicast loopback packets by FW/HW. Message-ID: When attaching a QP to a multicast group, checks if mcast loopback packets should be blocked (receives it as a parameter). If so, it sets the mcast loopback block bit to 1. When querying a QP (which is mlx4_ib_qp) using ib_qp_query, the field create_flags of the struct ib_qp_init_attr that is given as a parameter, will now be set with the flags that were used when this QP was created. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Changes in v3: The if statement that decided whether to block multicast loopback packets when attaching the QP, was replace with: (to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK) ? 1 : 0); Whitespaces deleted. --- linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c | 6 ++++-- linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h | 3 ++- linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c | 15 ++++++++++++++- linux-2.6.26-rc2/drivers/net/mlx4/mcg.c | 15 +++++++++++---- linux-2.6.26-rc2/include/linux/mlx4/device.h | 2 +- 5 files changed, 32 insertions(+), 9 deletions(-) Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/main.c 2008-06-26 14:10:19.000000000 +0300 @@ -90,7 +90,8 @@ static int mlx4_ib_query_device(struct i props->device_cap_flags = IB_DEVICE_CHANGE_PHY_PORT | IB_DEVICE_PORT_ACTIVE_EVENT | IB_DEVICE_SYS_IMAGE_GUID | - IB_DEVICE_RC_RNR_NAK_GEN; + IB_DEVICE_RC_RNR_NAK_GEN | + IB_DEVICE_BLOCK_LOOPBACK; if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_BAD_PKEY_CNTR) props->device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR; if (dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_BAD_QKEY_CNTR) @@ -437,7 +438,8 @@ static int mlx4_ib_dealloc_pd(struct ib_ static int mlx4_ib_mcg_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) { return mlx4_multicast_attach(to_mdev(ibqp->device)->dev, - &to_mqp(ibqp)->mqp, gid->raw); + &to_mqp(ibqp)->mqp, gid->raw, + (to_mqp(ibqp)->flags & MLX4_IB_QP_BLOCK_LOOPBACK) ? 1 : 0); } static int mlx4_ib_mcg_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid) Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/mlx4_ib.h 2008-06-26 14:10:19.000000000 +0300 @@ -101,7 +101,8 @@ struct mlx4_ib_wq { }; enum mlx4_ib_qp_flags { - MLX4_IB_QP_LSO = 1 << 0 + MLX4_IB_QP_LSO = 1 << 0, + MLX4_IB_QP_BLOCK_LOOPBACK = 1 << 1, }; struct mlx4_ib_qp { Index: kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/hw/mlx4/qp.c 2008-06-26 14:10:19.000000000 +0300 @@ -506,6 +506,9 @@ static int create_qp_common(struct mlx4_ } else { qp->sq_no_prefetch = 0; + if (init_attr->create_flags & IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK) + qp->flags |= MLX4_IB_QP_BLOCK_LOOPBACK; + if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO) qp->flags |= MLX4_IB_QP_LSO; @@ -680,7 +683,9 @@ struct ib_qp *mlx4_ib_create_qp(struct i int err; /* We only support LSO, and only for kernel UD QPs. */ - if (init_attr->create_flags & ~IB_QP_CREATE_IPOIB_UD_LSO) + if (init_attr->create_flags & + ~(IB_QP_CREATE_IPOIB_UD_LSO + | IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK)) return ERR_PTR(-EINVAL); if (init_attr->create_flags & IB_QP_CREATE_IPOIB_UD_LSO && (pd->uobject || init_attr->qp_type != IB_QPT_UD)) @@ -1862,6 +1867,14 @@ done: qp_init_attr->cap = qp_attr->cap; + qp_init_attr->create_flags = 0; + if (qp->flags & MLX4_IB_QP_BLOCK_LOOPBACK) + qp_init_attr->create_flags + |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; + + if (qp->flags & MLX4_IB_QP_LSO) + qp_init_attr->create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + out: mutex_unlock(&qp->mutex); return err; Index: kernels/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/net/mlx4/mcg.c 2008-06-26 14:10:19.000000000 +0300 @@ -38,6 +38,9 @@ #include "mlx4.h" +#define MGM_QPN_MASK 0x00FFFFFF +#define MGM_BLCK_LB_BIT 30 + struct mlx4_mgm { __be32 next_gid_index; __be32 members_count; @@ -153,7 +156,7 @@ static int find_mgm(struct mlx4_dev *dev return err; } -int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]) +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_cmd_mailbox *mailbox; @@ -202,13 +205,17 @@ int mlx4_multicast_attach(struct mlx4_de } for (i = 0; i < members_count; ++i) - if (mgm->qp[i] == cpu_to_be32(qp->qpn)) { + if ((be32_to_cpu(mgm->qp[i]) & MGM_QPN_MASK) == qp->qpn) { mlx4_dbg(dev, "QP %06x already a member of MGM\n", qp->qpn); err = 0; goto out; } - mgm->qp[members_count++] = cpu_to_be32(qp->qpn); + if (block_mcast_lb) + mgm->qp[members_count++] = cpu_to_be32((qp->qpn & MGM_QPN_MASK) | (1 << MGM_BLCK_LB_BIT)); + else + mgm->qp[members_count++] = cpu_to_be32(qp->qpn & MGM_QPN_MASK); + mgm->members_count = cpu_to_be32(members_count); err = mlx4_WRITE_MCG(dev, index, mailbox); @@ -283,7 +290,7 @@ int mlx4_multicast_detach(struct mlx4_de members_count = be32_to_cpu(mgm->members_count); for (loc = -1, i = 0; i < members_count; ++i) - if (mgm->qp[i] == cpu_to_be32(qp->qpn)) + if ((be32_to_cpu(mgm->qp[i]) & MGM_QPN_MASK) == qp->qpn) loc = i; if (loc == -1) { Index: kernels/linux-2.6.26-rc2/include/linux/mlx4/device.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/include/linux/mlx4/device.h 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/include/linux/mlx4/device.h 2008-06-26 14:10:19.000000000 +0300 @@ -398,7 +398,7 @@ int mlx4_srq_query(struct mlx4_dev *dev, int mlx4_INIT_PORT(struct mlx4_dev *dev, int port); int mlx4_CLOSE_PORT(struct mlx4_dev *dev, int port); -int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]); +int mlx4_multicast_attach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16], int block_mcast_lb); int mlx4_multicast_detach(struct mlx4_dev *dev, struct mlx4_qp *qp, u8 gid[16]); int mlx4_map_phys_fmr(struct mlx4_dev *dev, struct mlx4_fmr *fmr, u64 *page_list, From hrosenstock at xsigo.com Thu Jun 26 06:00:29 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 06:00:29 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080625160934.5025c073.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> Message-ID: <1214485229.13056.581.camel@hrosenstock-ws.xsigo.com> On Wed, 2008-06-25 at 16:09 -0700, Ira Weiny wrote: > diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c > index 7279e14..01f9530 100644 > --- a/libibmad/src/portid.c > +++ b/libibmad/src/portid.c ... > @@ -65,29 +66,24 @@ char * > portid2str(ib_portid_t *portid) > { > static char buf[1024] = "local"; > + char drpath[512]; > char *s = buf; > int i; > > if (portid->lid > 0) { > s += sprintf(s, "Lid %d", portid->lid); > if (portid->grh_present) { > - s += sprintf(s, " Gid 0x%" PRIx64 "%" PRIx64, > - ntohll(*(uint64_t *)portid- > >gid), > - ntohll(*(uint64_t *)(portid- > >gid+8))); > + char gid[sizeof > "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; > + inet_ntop(AF_INET6, portid->gid, gid, sizeof > (gid)); > + s += sprintf(s, " Gid %s", gid); Should this be: if (inet_ntop(AF_INET6, portid->gid, gid, sizeof (gid)) s += sprintf(s, " Gid %s", gid); > } Also, I'm all for this change but I'm also for consistency (I may be a minority of one on this) so I have to ask whether you plan on updating diags and/or OpenSM for GID format. Note also that if this is ported to Windows a wrapper for inet_ntop will be needed. -- Hal From hrosenstock at xsigo.com Thu Jun 26 06:09:54 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 06:09:54 -0700 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <48633946.3070706@Sun.COM> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> <48633946.3070706@Sun.COM> Message-ID: <1214485794.13056.592.camel@hrosenstock-ws.xsigo.com> Hi Sumit, On Thu, 2008-06-26 at 12:07 +0530, Sumit Gaur - Sun Microsystem wrote: > Hi Hal, > > Hal Rosenstock wrote: > > > >>>> > >>>>I am sending only request for > >>>> > >>>> rpc.mgtclass = IB_PERFORMANCE_CLASS; > >>>> rpc.method = IB_MAD_METHOD_GET; > >>>> > >>>>at every one second. > > > > > > Does perfquery work reliably with the same node(s) you are having > > trouble with ? > > > > Does your app follow what perfquery does ? > > Yes, perfquery works fine and I am following similar way of implementation. Here > is the output. I think difference is there in Load. I am sending 4 GS request > per second basis and some got passed and some got timeout(110) or recv failed. Can you elaborate on the multiple sends ? Are they outstanding concurrently ? Are they to the same destination or different ones ? Are they from a single or multiple threads ? > # perfquery > # Port counters: Lid 393 port 1 > PortSelect:......................1 > CounterSelect:...................0x0000 > SymbolErrors:....................0 > LinkRecovers:....................0 > LinkDowned:......................0 > RcvErrors:.......................0 > RcvRemotePhysErrors:.............0 > RcvSwRelayErrors:................0 > XmtDiscards:.....................0 > XmtConstraintErrors:.............0 > RcvConstraintErrors:.............0 > LinkIntegrityErrors:.............0 > ExcBufOverrunErrors:.............0 > VL15Dropped:.....................0 > XmtData:.........................65899728 > RcvData:.........................65899656 > XmtPkts:.........................915274 > RcvPkts:.........................915273 > > > > >>>>>In general, there are a few possibilities (which can cause this). SM > >>>>>traffic is VL15 whereas GS traffic is on a data VL (usually VL0 in most > >>>>>subnets). > >>>>> > >>>>>Some possibilities are: > >>>>>1. Timeout/retry being hit for some GS traffic (GS request or response > >>>>>lost/corrupted) > >>>> > >>>>Yes, this is also happening, Sometimes I am getting corrupt data back, > >>> > >>> > >>>Is there an error indicated ? > >> > >>For such packets I am getting umad_status as 110. > > > > > > That's ETIMEDOUT. You need to handle the errors (and not treat the > > receive as a valid packet). Are you doing that ? > > yes, I am catching this error. OK but you had said the received packet was corrupted. Maybe a nit, but with timeout and other errors, the receive packet is invalid rather than corrupted (an app shouldn't be looking at the response in the error cases). > > The underlying question is why are you getting the timeout relatively > > frequently so I recommend checking all the error counters along the > > path. > > # Checking Ca: nodeguid 0x00144fa5e9ce001c > Node check lid 392: OK > Error check on lid 392 (HCA-1) port all: OK Is that the requester or responder ? It's not the entire path. Maybe the simplest thing is: what does ibchecknet or ibcheckerrors say ? In any case, based on your comments above about perfquery working reliably, I'm skeptical whether this is the issue but it's best to rule it out. > > Are you sure the request gets to the responder ? Does the responder > > respond and it doesn't make it back ? > > yes As I told It is not 100% failure, It is 30% to 40% failure. But Why ? I don't know enough about what is different about your app yet to say more right now. -- Hal > > -- Hal > > > > From ronli at voltaire.com Thu Jun 26 07:33:59 2008 From: ronli at voltaire.com (Ron Livne) Date: Thu, 26 Jun 2008 14:33:59 +0000 (UTC) Subject: [ofa-general] ***SPAM*** [PATCH 1/3 v3] ib/verbs : multicast loopback blocking Message-ID: This patch adds a capability flag for blocking multicast loopback packets in an ib device. This patch also adds a creation flag for QPs: IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK, which means that when attaching this QP, the multicast packets loopback block bit will be turned on. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Changes in v3: whitespaces deleted. --- linux-2.6.26-rc2/include/rdma/ib_verbs.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) Index: kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h =================================================================== --- kernels.orig/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/include/rdma/ib_verbs.h 2008-06-26 14:10:19.000000000 +0300 @@ -106,6 +106,7 @@ enum ib_device_cap_flags { IB_DEVICE_UD_IP_CSUM = (1<<18), IB_DEVICE_UD_TSO = (1<<19), IB_DEVICE_SEND_W_INV = (1<<21), + IB_DEVICE_BLOCK_LOOPBACK = (1<<22), }; enum ib_atomic_cap { @@ -499,7 +500,8 @@ enum ib_qp_type { }; enum ib_qp_create_flags { - IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + IB_QP_CREATE_IPOIB_UD_LSO = 1 << 0, + IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK = 1 << 1, }; struct ib_qp_init_attr { From ronli at voltaire.com Thu Jun 26 07:35:22 2008 From: ronli at voltaire.com (Ron Livne) Date: Thu, 26 Jun 2008 14:35:22 +0000 (UTC) Subject: [ofa-general] ***SPAM*** [PATCH 3/3 v3] ib/ipoib: blocking multicast loopback ipoib packets Message-ID: This patch makes any mcast packet sent via IPoIB from any host, to be blocked when looped back to the same host it was sent from. The blocking is done by the HCA if possible. This patch creates an improvement of up to 39% in bandwidth when seding mcast packets with IPoIB. There is also an improvment of 12% in cpu usage. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Changes in v3: Whitespaces deleted. --- linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) Index: kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:10:01.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:10:20.000000000 +0300 @@ -153,7 +153,8 @@ int ipoib_transport_dev_init(struct net_ .max_recv_sge = IPOIB_UD_RX_SG }, .sq_sig_type = IB_SIGNAL_ALL_WR, - .qp_type = IB_QPT_UD + .qp_type = IB_QPT_UD, + .create_flags = 0 }; int ret, size; @@ -201,7 +202,10 @@ int ipoib_transport_dev_init(struct net_ init_attr.recv_cq = priv->recv_cq; if (priv->hca_caps & IB_DEVICE_UD_TSO) - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + + if (priv->hca_caps & IB_DEVICE_BLOCK_LOOPBACK) + init_attr.create_flags |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; if (dev->features & NETIF_F_SG) init_attr.cap.max_send_sge = MAX_SKB_FRAGS + 1; From xma at us.ibm.com Thu Jun 26 06:59:58 2008 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 26 Jun 2008 06:59:58 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: <1214478350.23583.104.camel@mtls03> Message-ID: Hello Eli, >This means that if you receive >such an event, you're guaranteed to see the CQE in the CQ (that is of of >course if there is such a CQE; if there where no WQEs related to the QP >at the time it entered an error state then the HW just generates the >event). Thanks for the clarification. Another question, after the QP entered an error state, the HCA can generate a cqe for post_send drain WRs? > Shirley, did you see this on Mellanox HCAs too or just ehca? I saw some panic in Mellanox HCAs, I thought this senarios could cause a panic. ehca doesn't support post_send() drain WRs after such an event. So it can't be a problem for ehca. Since there is no such a post_send() drain WRs for ehca, then we need a timer value to release the resource from rx_flush_list. Otherwise the resouces will remain there forever, for a long run, the node loses lots of memory. We have seen lots of last WQE events when doing interoperability test between MHCAs and ehcas. thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrosenstock at xsigo.com Thu Jun 26 07:07:51 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 07:07:51 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] osm_port_profile: Change some variable names for consistency Message-ID: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> osm_port_profile: Change some variable names for consistency Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index 0376686..e1bdd4a 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -198,10 +198,10 @@ osm_port_prof_path_count_get(IN const osm_port_profile_t * const p_prof) */ static inline boolean_t osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, - IN ib_net64_t port_guid, IN uint8_t port_num) + IN ib_net64_t node_guid, IN uint8_t port_num) { const cl_map_t *p_map = &p_subn->port_prof_ignore_guids; - const void *p_obj = cl_map_get(p_map, port_guid); + const void *p_obj = cl_map_get(p_map, node_guid); size_t res; // HACK: we currently support ignoring ports 0 - 31 @@ -217,8 +217,8 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, * p_subn * [in] Pointer to the OSM Subnet object. * -* port_guid -* [in] The port guid +* node_guid +* [in] The node guid * * RETURN VALUE * None. @@ -233,16 +233,16 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, * osm_port_prof_set_ignored_port * * DESCRIPTION -* Set the ignored property of the port. +* Set the ignored property of a port. * * SYNOPSIS */ static inline void osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, - IN ib_net64_t port_guid, IN uint8_t port_num) + IN ib_net64_t node_guid, IN uint8_t port_num) { cl_map_t *p_map = &p_subn->port_prof_ignore_guids; - const void *p_obj = cl_map_get(p_map, port_guid); + const void *p_obj = cl_map_get(p_map, node_guid); size_t value = 0; // HACK: we currently support ignoring ports 0 - 31 @@ -250,11 +250,11 @@ osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, if (p_obj != NULL) { value = (size_t) p_obj; - cl_map_remove(p_map, port_guid); + cl_map_remove(p_map, node_guid); } value = value | (1 << port_num); - cl_map_insert(p_map, port_guid, (void *)value); + cl_map_insert(p_map, node_guid, (void *)value); } /* @@ -262,8 +262,8 @@ osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, * p_subn * [in] Pointer to the OSM Subnet object. * -* port_guid -* [in] The port guid +* node_guid +* [in] The node guid * * RETURN VALUE * None. diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 77c4b19..7c7525b 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -464,7 +464,7 @@ parse_ignore_guids_file(IN char *guids_file_name, IN osm_opensm_t * p_osm) char line[OSM_MAX_IGNORE_GUID_LINES_LEN]; char *p_c, *p_ec; uint32_t line_num = 0; - uint64_t port_guid; + uint64_t node_guid; ib_api_status_t status = IB_SUCCESS; unsigned int port_num; @@ -487,7 +487,7 @@ parse_ignore_guids_file(IN char *guids_file_name, IN osm_opensm_t * p_osm) p_c = line; while ((*p_c == ' ') && (*p_c != '\0')) p_c++; - port_guid = strtoull(p_c, &p_ec, 16); + node_guid = strtoull(p_c, &p_ec, 16); if (p_ec == p_c) { OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 0602: " "Error in line (%u): %s\n", line_num, line); @@ -506,10 +506,10 @@ parse_ignore_guids_file(IN char *guids_file_name, IN osm_opensm_t * p_osm) /* ok insert it */ osm_port_prof_set_ignored_port(&p_osm->subn, - cl_hton64(port_guid), port_num); + cl_hton64(node_guid), port_num); OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, - "Inserted Port: 0x%" PRIx64 - " PortNum: 0x%X into ignored guids list\n", port_guid, + "Inserted Node: 0x%" PRIx64 + " PortNum: 0x%X into ignored guids list\n", node_guid, port_num); } From hrosenstock at xsigo.com Thu Jun 26 07:08:25 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 07:08:25 -0700 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/include/opensm/osm_port.h: Commentary change Message-ID: <1214489305.13056.602.camel@hrosenstock-ws.xsigo.com> opensm/include/opensm/osm_port.h: Commentary change Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index a07c38e..3521adc 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -163,7 +163,7 @@ typedef struct osm_physp { * * slvl_by_port * A vector of pointers to the sl2vl tables (ordered by input port). -* On switches have an entry for every other input port (inc SMA=0). +* Switches have an entry for every other input port (inc SMA=0). * On CAs only one per port. * * SEE ALSO From eli at mellanox.co.il Thu Jun 26 07:32:36 2008 From: eli at mellanox.co.il (Eli Cohen) Date: Thu, 26 Jun 2008 17:32:36 +0300 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: References: Message-ID: <1214490756.23583.113.camel@mtls03> On Thu, 2008-06-26 at 06:59 -0700, Shirley Ma wrote: > Thanks for the clarification. Another question, after the QP entered > an error state, the HCA can generate a cqe for post_send drain WRs? Yes, it generates a CQE with error (IB_WC_WR_FLUSH_ERR). > I saw some panic in Mellanox HCAs, I thought this senarios could cause > a panic. ehca doesn't support post_send() drain WRs after such an > event. So it can't be a problem for ehca. Please send a dump of the panic you've been seeing along with a description how you got it. From eli at dev.mellanox.co.il Thu Jun 26 07:55:50 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 26 Jun 2008 17:55:50 +0300 Subject: [ofa-general] ***SPAM*** [PATCH 3/3 v3] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: References: Message-ID: <1214492150.23583.116.camel@mtls03> On Thu, 2008-06-26 at 14:35 +0000, Ron Livne wrote: > Index: kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c > =================================================================== > --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:10:01.000000000 +0300 > +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:10:20.000000000 +0300 > @@ -153,7 +153,8 @@ int ipoib_transport_dev_init(struct net_ > .max_recv_sge = IPOIB_UD_RX_SG > }, > .sq_sig_type = IB_SIGNAL_ALL_WR, > - .qp_type = IB_QPT_UD > + .qp_type = IB_QPT_UD, > + .create_flags = 0 > }; Why do you need to set create_flags explicitly? > > int ret, size; > @@ -201,7 +202,10 @@ int ipoib_transport_dev_init(struct net_ > init_attr.recv_cq = priv->recv_cq; > > if (priv->hca_caps & IB_DEVICE_UD_TSO) > - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; > + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; and change this too? > + > + if (priv->hca_caps & IB_DEVICE_BLOCK_LOOPBACK) > + init_attr.create_flags |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; > > if (dev->features & NETIF_F_SG) > init_attr.cap.max_send_sge = MAX_SKB_FRAGS + 1; From swise at opengridcomputing.com Thu Jun 26 07:56:26 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 26 Jun 2008 09:56:26 -0500 Subject: [ofa-general] [ANNOUNCE] MEM_MGT_EXTENSIONS - example/test program Message-ID: <4863AE1A.3040101@opengridcomputing.com> All, I've added logic to krping to use fast_reg_mrs, local_inv, as well as send-with-inv . You can pull the code from git://git.openfabrics.org/~swise/krping.git There is a README that tries to explain the test case. Enjoy. If you have comments/criticisms, please include a patch. :) Steve. From rdreier at cisco.com Thu Jun 26 08:02:25 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 26 Jun 2008 08:02:25 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: (Shirley Ma's message of "Thu, 26 Jun 2008 06:59:58 -0700") References: Message-ID: > Thanks for the clarification. Another question, after the QP entered an > error state, the HCA can generate a cqe for post_send drain WRs? Yes, this is required by the spec. > I saw some panic in Mellanox HCAs, I thought this senarios could cause a > panic. ehca doesn't support post_send() drain WRs after such an event. So > it can't be a problem for ehca. This is just the bug in ehca rejecting send WRs in error state that only was in some 2.6.26-rc kernels, right? From rdreier at cisco.com Thu Jun 26 08:05:12 2008 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 26 Jun 2008 08:05:12 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: <1214478350.23583.104.camel@mtls03> (Eli Cohen's message of "Thu, 26 Jun 2008 14:05:50 +0300") References: <1214382352.23583.54.camel@mtls03> <1214478350.23583.104.camel@mtls03> Message-ID: > So what you're saying is that the scenario Shirley described is not > possible... Seems like it... so the "one at a time" draining bug fix is probably not right. The IB spec says: If the HCA supports SRQ, for RC and UD service, the CI shall generate a Last WQE Reached Affiliated Asynchronous Event on a QP that is in the Error State and is associated with an SRQ when either: . a CQE is generated for the last WQE, or . the QP gets in the Error State and there are no more WQEs on the RQ. It is slightly ambiguous what "generated" means for a CQE -- it is conceivable that someone could interpret that to mean a CQE is "in flight" but has not necessarily become visible. Anyway I think we need to debug this further. - R. From eli at dev.mellanox.co.il Thu Jun 26 08:18:44 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 26 Jun 2008 18:18:44 +0300 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: References: <1214382352.23583.54.camel@mtls03> <1214478350.23583.104.camel@mtls03> Message-ID: <20080626151844.GA9508@mtls03> On Thu, Jun 26, 2008 at 08:05:12AM -0700, Roland Dreier wrote: > > So what you're saying is that the scenario Shirley described is not > > possible... > > Seems like it... so the "one at a time" draining bug fix is probably not > right. The IB spec says: > > If the HCA supports SRQ, for RC and UD service, the CI shall generate a > Last WQE Reached Affiliated Asynchronous Event on a QP that is in the > Error State and is associated with an SRQ when either: > > . a CQE is generated for the last WQE, or > . the QP gets in the Error State and there are no more WQEs on the RQ. > > It is slightly ambiguous what "generated" means for a CQE -- it is > conceivable that someone could interpret that to mean a CQE is "in > flight" but has not necessarily become visible. I believe "generated" means here that the CQE has been pushed into the CQ (posted write) so if you drain the CQ you should see it (if the chipset maintains cache coherency for the CQ buffer). > > Anyway I think we need to debug this further. > > - R. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ogerlitz at voltaire.com Thu Jun 26 08:27:13 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 26 Jun 2008 18:27:13 +0300 Subject: [ofa-general] Re: [PATCH v2 0/4] rdma: addr ADDR_CHANGE rdma-cm event support In-Reply-To: References: <485A7903.4050407@voltaire.com> <4860C352.5070603@voltaire.com> <15ddcffd0806251508r65ddcd76q709e449a3a960460@mail.gmail.com> Message-ID: <4863B551.3010902@voltaire.com> Roland Dreier wrote: > It just seems to minimize the hassle. OK, lets have it this way, thanks Or. From ronli.voltaire at gmail.com Thu Jun 26 08:45:33 2008 From: ronli.voltaire at gmail.com (Ron Livne) Date: Thu, 26 Jun 2008 18:45:33 +0300 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH 3/3 v3] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: <1214492150.23583.116.camel@mtls03> References: <1214492150.23583.116.camel@mtls03> Message-ID: <3b5e77ad0806260845v5b4cc192wd6b1c7940c6fbc0f@mail.gmail.com> On Thu, Jun 26, 2008 at 5:55 PM, Eli Cohen wrote: > On Thu, 2008-06-26 at 14:35 +0000, Ron Livne wrote: >> Index: kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c >> =================================================================== >> --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:10:01.000000000 +0300 >> +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:10:20.000000000 +0300 >> @@ -153,7 +153,8 @@ int ipoib_transport_dev_init(struct net_ >> .max_recv_sge = IPOIB_UD_RX_SG >> }, >> .sq_sig_type = IB_SIGNAL_ALL_WR, >> - .qp_type = IB_QPT_UD >> + .qp_type = IB_QPT_UD, >> + .create_flags = 0 >> }; > Why do you need to set create_flags explicitly? > >> >> int ret, size; >> @@ -201,7 +202,10 @@ int ipoib_transport_dev_init(struct net_ >> init_attr.recv_cq = priv->recv_cq; >> >> if (priv->hca_caps & IB_DEVICE_UD_TSO) >> - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; >> + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; > > and change this too? The reason is that I wanted to return all the creation flags for the QP. So I set it to 0 in case no creation flag was used, not just IB_QP_CREATE_IPOIB_UD_LSO as it was. Notice that I also added: + if (priv->hca_caps & IB_DEVICE_BLOCK_LOOPBACK) + init_attr.create_flags |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; for the creation flag I added. Ron From xma at us.ibm.com Thu Jun 26 09:12:13 2008 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 26 Jun 2008 09:12:13 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last wqe event handler patch In-Reply-To: <1214490756.23583.113.camel@mtls03> Message-ID: Hello Eli, > > I saw some panic in Mellanox HCAs, I thought this senarios could cause > > a panic. ehca doesn't support post_send() drain WRs after such an > > event. So it can't be a problem for ehca. > Please send a dump of the panic you've been seeing along with a > description how you got it. Right now we have several issues on hands for nonSRQ. This issue hardly recreates it, but I will try later after we have addressed nonSRQ problems. thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From suri at baymicrosystems.com Thu Jun 26 09:26:06 2008 From: suri at baymicrosystems.com (Suresh Shelvapille) Date: Thu, 26 Jun 2008 12:26:06 -0400 Subject: [ofa-general] performance drop for datagram mode with the newconnectx FW References: <20080623114236.GA11852@mtls03> Message-ID: <698872956B634A0893C39DC406F8C31E@md.baymicrosystems.com> I get about 520MB with the latest FW 2.5.0, it is definitely worse than 2.3.0 > -----Original Message----- > From: Suresh Shelvapille [mailto:suri at baymicrosystems.com] > Sent: Tuesday, June 24, 2008 12:01 PM > To: 'Eli Cohen'; 'Or Gerlitz' > Cc: 'general at lists.openfabrics.org' > Subject: RE: [ofa-general] performance drop for datagram mode with the newconnectx FW > > > I also get about 650MB for datagram mode with 2.3 FW. > > > -----Original Message----- > > From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On > Behalf > > Of Eli Cohen > > Sent: Monday, June 23, 2008 7:43 AM > > To: Or Gerlitz > > Cc: Eli Cohen; Roland Dreier; general at lists.openfabrics.org > > Subject: Re: [ofa-general] performance drop for datagram mode with the newconnectx FW > > > > On Mon, Jun 23, 2008 at 02:36:39PM +0300, Or Gerlitz wrote: > > > Eli, > > > > > > Using the new connectx FW (2.5), I see performance drop to almost > > > zero with ipoib datagram mode. The code that runs on these systems > > > is ofed 1.3 and not mainline kernel, details below. > > > > > > Running netperf With connected mode (64k MTU) I get about 950MB/s > > > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to > > > see about 650MB/s and above with FW 2.3 and datagram mode. Not that > > > it could explain the drop, but the NIC reports to the OS stateless > > > offload support - /sys/class/net/ib1/features is 0x11423 > > > > > > I have opened the ipoib and mlx4 debug prints, and I don't see anything > > > special other then the dmesg get quite filled with > > > > > > ib1: TX ring full, stopping kernel net queue > > > > > > any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which > > > is about five times what I see with ipoib. > > > > > > > > > Or. > > > > > > git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel > > > commit 564e9e9383272f4311fd87ff4e5447cfcebad73a > > > > > > # uname -a > > > Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux > > > > > > # cat /etc/redhat-release > > > Red Hat Enterprise Linux Server release 5.1 (Tikanga) > > > > Can you tell if changing the FW to 2.3 gives more reasonable results? > > I don't believe such a drop in performance would have passed the QA > > tests but I'll check that. > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From weiny2 at llnl.gov Thu Jun 26 09:39:26 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 26 Jun 2008 09:39:26 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080625222331.GB28318@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> Message-ID: <20080626093926.65e6abcf.weiny2@llnl.gov> One comment, which I mentioned to you off-list, was that passing struct osm_opensm without having that header public currently breaks my plugin and will make it impossible for them to compile against the installed headers. I am really leaning toward including all the headers in the opensm-devel package so plugin writers don't have to have a git tree to build. Ira On Thu, 26 Jun 2008 01:23:31 +0300 Sasha Khapyorsky wrote: > > Main difference is that construct() method is renamed to create() and > gets reference to osm_opensm_t object instead of just osm_log_t. This > will provide to event plugin access to all OpenSM data structures and > should help to create more generic plugins. For consistency with above > destroy() method is renamed to delete(). Event plugin interface version > is increased and osmeventplugin example updated accordingly. > > Signed-off-by: Sasha Khapyorsky > --- > opensm/include/opensm/osm_event_plugin.h | 12 ++++++------ > opensm/opensm/osm_event_plugin.c | 23 +++++++++++------------ > opensm/opensm/osm_opensm.c | 2 +- > opensm/osmeventplugin/src/osmeventplugin.c | 18 ++++++++++++------ > 4 files changed, 30 insertions(+), 25 deletions(-) > > diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h > index 35d5b7d..953709d 100644 > --- a/opensm/include/opensm/osm_event_plugin.h > +++ b/opensm/include/opensm/osm_event_plugin.h > @@ -37,7 +37,6 @@ > #include > #include > #include > -#include > > #ifdef __cplusplus > # define BEGIN_C_DECLS extern "C" { > @@ -60,6 +59,8 @@ BEGIN_C_DECLS > *********/ > > #define OSM_EPI_NODE_NAME_LEN (128) > + > +struct osm_opensm; > /** ========================================================================= > * Event types > */ > @@ -145,11 +146,11 @@ typedef struct osm_epi_trap_event { > * The version should be set to OSM_EVENT_PLUGIN_INTERFACE_VER > */ > #define OSM_EVENT_PLUGIN_IMPL_NAME "osm_event_plugin" > -#define OSM_EVENT_PLUGIN_INTERFACE_VER (1) > +#define OSM_EVENT_PLUGIN_INTERFACE_VER 2 > typedef struct osm_event_plugin { > int interface_version; > - void *(*construct) (osm_log_t * osm_log); > - void (*destroy) (void *plugin_data); > + void *(*create) (struct osm_opensm *osm); > + void (*delete) (void *plugin_data); > void (*report) (void *plugin_data, > osm_epi_event_id_t event_id, void *event_data); > } osm_event_plugin_t; > @@ -162,14 +163,13 @@ typedef struct osm_epi_plugin { > void *handle; > osm_event_plugin_t *impl; > void *plugin_data; > - osm_log_t *p_log; > char *plugin_name; > } osm_epi_plugin_t; > > /** > * functions > */ > -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name); > +osm_epi_plugin_t *osm_epi_construct(struct osm_opensm *osm, char *plugin_name); > void osm_epi_destroy(osm_epi_plugin_t * plugin); > > /** ========================================================================= > diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c > index e579fad..98d5302 100644 > --- a/opensm/opensm/osm_event_plugin.c > +++ b/opensm/opensm/osm_event_plugin.c > @@ -44,8 +44,8 @@ > > #include > #include > - > #include > +#include > > #if defined(PATH_MAX) > #define OSM_PATH_MAX (PATH_MAX + 1) > @@ -58,7 +58,7 @@ > /** > * functions > */ > -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > +osm_epi_plugin_t *osm_epi_construct(osm_opensm_t *osm, char *plugin_name) > { > char lib_name[OSM_PATH_MAX]; > osm_epi_plugin_t *rc = NULL; > @@ -75,7 +75,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > > rc->handle = dlopen(lib_name, RTLD_LAZY); > if (!rc->handle) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > "Failed to open event plugin \"%s\" : \"%s\"\n", > lib_name, dlerror()); > goto DLOPENFAIL; > @@ -85,7 +85,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > (osm_event_plugin_t *) dlsym(rc->handle, > OSM_EVENT_PLUGIN_IMPL_NAME); > if (!rc->impl) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > "Failed to find \"%s\" symbol in \"%s\" : \"%s\"\n", > OSM_EVENT_PLUGIN_IMPL_NAME, lib_name, dlerror()); > goto Exit; > @@ -93,7 +93,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > > /* Check the version to make sure this module will work with us */ > if (rc->impl->interface_version != OSM_EVENT_PLUGIN_INTERFACE_VER) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > "Error opening %s: " > "%s symbol is the wrong version %d != %d\n", > plugin_name, > @@ -103,19 +103,18 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > goto Exit; > } > > - if (!rc->impl->construct) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > - "%s symbol has no construct function\n", > + if (!rc->impl->create) { > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > + "%s symbol has no create() function\n", > OSM_EVENT_PLUGIN_IMPL_NAME); > goto Exit; > } > > - rc->plugin_data = rc->impl->construct(p_log); > + rc->plugin_data = rc->impl->create(osm); > > if (!rc->plugin_data) > goto Exit; > > - rc->p_log = p_log; > rc->plugin_name = strdup(plugin_name); > return (rc); > > @@ -129,8 +128,8 @@ DLOPENFAIL: > void osm_epi_destroy(osm_epi_plugin_t * plugin) > { > if (plugin) { > - if (plugin->impl->destroy) > - plugin->impl->destroy(plugin->plugin_data); > + if (plugin->impl->delete) > + plugin->impl->delete(plugin->plugin_data); > dlclose(plugin->handle); > free(plugin->plugin_name); > free(plugin); > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index 0101b97..6cf4726 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -251,7 +251,7 @@ static void load_plugins(osm_opensm_t *osm, char *plugin_names) > > name = strtok_r(plugin_names, " \t\n", &p); > while (name && *name) { > - epi = osm_epi_construct(&osm->log, name); > + epi = osm_epi_construct(osm, name); > if (!epi) > osm_log(&osm->log, OSM_LOG_ERROR, > "cannot load plugin \'%s\'\n", name); > diff --git a/opensm/osmeventplugin/src/osmeventplugin.c b/opensm/osmeventplugin/src/osmeventplugin.c > index 6cc4c70..a56d3c3 100644 > --- a/opensm/osmeventplugin/src/osmeventplugin.c > +++ b/opensm/osmeventplugin/src/osmeventplugin.c > @@ -38,12 +38,15 @@ > #include > #include > #include > +#include > #include > #include > #include > -#include > #include > #include > +#include > +#include > +#include > > /** ========================================================================= > * This is a simple example plugin which logs some of the events the OSM > @@ -57,8 +60,11 @@ typedef struct _log_events { > > /** ========================================================================= > */ > -static void *construct(osm_log_t * osmlog) > +static void *construct(osm_opensm_t *osm) > { > + if (strcmp(osm->osm_version, OSM_VERSION)) > + return NULL; > + > _log_events_t *log = malloc(sizeof(*log)); > if (!log) > return (NULL); > @@ -66,14 +72,14 @@ static void *construct(osm_log_t * osmlog) > log->log_file = fopen(SAMPLE_PLUGIN_OUTPUT_FILE, "a+"); > > if (!(log->log_file)) { > - osm_log(osmlog, OSM_LOG_ERROR, > + osm_log(&osm->log, OSM_LOG_ERROR, > "Sample Event Plugin: Failed to open output file \"%s\"\n", > SAMPLE_PLUGIN_OUTPUT_FILE); > free(log); > return (NULL); > } > > - log->osmlog = osmlog; > + log->osmlog = &osm->log; > return ((void *)log); > } > > @@ -173,7 +179,7 @@ static void report(void *_log, osm_epi_event_id_t event_id, void *event_data) > */ > osm_event_plugin_t osm_event_plugin = { > interface_version:OSM_EVENT_PLUGIN_INTERFACE_VER, > - construct:construct, > - destroy:destroy, > + create:construct, > + delete:destroy, > report:report > }; > -- > 1.5.5.1.178.g1f811 > From hrosenstock at xsigo.com Thu Jun 26 10:10:56 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 10:10:56 -0700 Subject: [ofa-general] Re: [PATCH] opensm: add osm_version field to osm_opensm_t object In-Reply-To: <20080625222135.GA28318@sashak.voltaire.com> References: <20080625222135.GA28318@sashak.voltaire.com> Message-ID: <1214500256.13056.608.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-26 at 01:21 +0300, Sasha Khapyorsky wrote: > This adds osm_version field to osm_opensm_t object. Is part of the intention here to use this in plugins ? Is this versioning of osm_opensm_t for that (and other uses too) ? If so, that may be a little too granular IMO. Perhaps it should be more like the libraries in terms of versioning. More comments on plugin architecture to follow in a bit... -- Hal > The value is > generated in OSM_VERSION macro in osm_version.h file. > > Signed-off-by: Sasha Khapyorsky > --- > opensm/include/opensm/osm_opensm.h | 4 ++++ > opensm/opensm/osm_console.c | 5 ++--- > opensm/opensm/osm_opensm.c | 5 +++-- > 3 files changed, 9 insertions(+), 5 deletions(-) > > diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h > index 285fd88..a1c255b 100644 > --- a/opensm/include/opensm/osm_opensm.h > +++ b/opensm/include/opensm/osm_opensm.h > @@ -163,6 +163,7 @@ struct osm_routing_engine { > * SYNOPSIS > */ > typedef struct osm_opensm { > + const char *osm_version; > osm_subn_t subn; > osm_sm_t sm; > osm_sa_t sa; > @@ -185,6 +186,9 @@ typedef struct osm_opensm { > } osm_opensm_t; > /* > * FIELDS > +* osm_version > +* OpenSM version (as generated in osm_version.h) > +* > * subn > * Subnet object for this subnet. > * > diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c > index 7f0b00f..39fb17a 100644 > --- a/opensm/opensm/osm_console.c > +++ b/opensm/opensm/osm_console.c > @@ -50,7 +50,6 @@ > #include > #include > #include > -#include > #include > #include > #include > @@ -312,7 +311,7 @@ static void print_status(osm_opensm_t * p_osm, FILE * out) > { > if (out) { > cl_plock_acquire(&p_osm->lock); > - fprintf(out, " OpenSM Version: %s\n", OSM_VERSION); > + fprintf(out, " OpenSM Version: %s\n", p_osm->osm_version); > fprintf(out, " SM State : %s\n", > sm_state_str(p_osm->subn.sm_state)); > fprintf(out, " SA State : %s\n", > @@ -1137,7 +1136,7 @@ static void help_version(FILE * out, int detail) > > static void version_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) > { > - fprintf(out, "%s build %s %s\n", OSM_VERSION, __DATE__, __TIME__); > + fprintf(out, "%s build %s %s\n", p_osm->osm_version, __DATE__, __TIME__); > } > > /* more parse routines go here */ > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index adacd93..0101b97 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -169,6 +169,7 @@ static int osm_ucast_null_setup(osm_opensm_t * p_osm) > void osm_opensm_construct(IN osm_opensm_t * const p_osm) > { > memset(p_osm, 0, sizeof(*p_osm)); > + p_osm->osm_version = OSM_VERSION; > osm_subn_construct(&p_osm->subn); > osm_sm_construct(&p_osm->sm); > osm_sa_construct(&p_osm->sa); > @@ -283,9 +284,9 @@ osm_opensm_init(IN osm_opensm_t * const p_osm, > /* If there is a log level defined - add the OSM_VERSION to it */ > osm_log(&p_osm->log, > osm_log_get_level(&p_osm->log) & (OSM_LOG_SYS ^ 0xFF), "%s\n", > - OSM_VERSION); > + p_osm->osm_version); > /* Write the OSM_VERSION to the SYS_LOG */ > - osm_log(&p_osm->log, OSM_LOG_SYS, "%s\n", OSM_VERSION); /* Format Waived */ > + osm_log(&p_osm->log, OSM_LOG_SYS, "%s\n", p_osm->osm_version); /* Format Waived */ > > OSM_LOG(&p_osm->log, OSM_LOG_FUNCS, "[\n"); /* Format Waived */ > From hrosenstock at xsigo.com Thu Jun 26 10:26:58 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 10:26:58 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080625222331.GB28318@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> Message-ID: <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-26 at 01:23 +0300, Sasha Khapyorsky wrote: > Main difference is that construct() method is renamed to create() and > gets reference to osm_opensm_t object instead of just osm_log_t. This > will provide to event plugin access to all OpenSM data structures and > should help to create more generic plugins. This exposes almost all internal data structures! While that is powerful, it also could be dangerous as plugins can now more readily corrupt OpenSM. It's certainly the most flexible but leaves everything totally open. Is this the best approach ? If this goes ahead, versioning is an issue. Not sure osm_version is the best barometer for everything. It may be for include files for building a plugin but what about a coarser versioning (like library versioning) ? It would be nice if a single plugin can handle as many OpenSM versions as possible (by something simpler than a range of osm_versions) rather than needing a separate plugin per OpenSM version at the other end of the spectrum. -- Hal > For consistency with above > destroy() method is renamed to delete(). Event plugin interface version > is increased and osmeventplugin example updated accordingly. > Signed-off-by: Sasha Khapyorsky > --- > opensm/include/opensm/osm_event_plugin.h | 12 ++++++------ > opensm/opensm/osm_event_plugin.c | 23 +++++++++++------------ > opensm/opensm/osm_opensm.c | 2 +- > opensm/osmeventplugin/src/osmeventplugin.c | 18 ++++++++++++------ > 4 files changed, 30 insertions(+), 25 deletions(-) > > diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h > index 35d5b7d..953709d 100644 > --- a/opensm/include/opensm/osm_event_plugin.h > +++ b/opensm/include/opensm/osm_event_plugin.h > @@ -37,7 +37,6 @@ > #include > #include > #include > -#include > > #ifdef __cplusplus > # define BEGIN_C_DECLS extern "C" { > @@ -60,6 +59,8 @@ BEGIN_C_DECLS > *********/ > > #define OSM_EPI_NODE_NAME_LEN (128) > + > +struct osm_opensm; > /** ========================================================================= > * Event types > */ > @@ -145,11 +146,11 @@ typedef struct osm_epi_trap_event { > * The version should be set to OSM_EVENT_PLUGIN_INTERFACE_VER > */ > #define OSM_EVENT_PLUGIN_IMPL_NAME "osm_event_plugin" > -#define OSM_EVENT_PLUGIN_INTERFACE_VER (1) > +#define OSM_EVENT_PLUGIN_INTERFACE_VER 2 > typedef struct osm_event_plugin { > int interface_version; > - void *(*construct) (osm_log_t * osm_log); > - void (*destroy) (void *plugin_data); > + void *(*create) (struct osm_opensm *osm); > + void (*delete) (void *plugin_data); > void (*report) (void *plugin_data, > osm_epi_event_id_t event_id, void *event_data); > } osm_event_plugin_t; > @@ -162,14 +163,13 @@ typedef struct osm_epi_plugin { > void *handle; > osm_event_plugin_t *impl; > void *plugin_data; > - osm_log_t *p_log; > char *plugin_name; > } osm_epi_plugin_t; > > /** > * functions > */ > -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name); > +osm_epi_plugin_t *osm_epi_construct(struct osm_opensm *osm, char *plugin_name); > void osm_epi_destroy(osm_epi_plugin_t * plugin); > > /** ========================================================================= > diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c > index e579fad..98d5302 100644 > --- a/opensm/opensm/osm_event_plugin.c > +++ b/opensm/opensm/osm_event_plugin.c > @@ -44,8 +44,8 @@ > > #include > #include > - > #include > +#include > > #if defined(PATH_MAX) > #define OSM_PATH_MAX (PATH_MAX + 1) > @@ -58,7 +58,7 @@ > /** > * functions > */ > -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > +osm_epi_plugin_t *osm_epi_construct(osm_opensm_t *osm, char *plugin_name) > { > char lib_name[OSM_PATH_MAX]; > osm_epi_plugin_t *rc = NULL; > @@ -75,7 +75,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > > rc->handle = dlopen(lib_name, RTLD_LAZY); > if (!rc->handle) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > "Failed to open event plugin \"%s\" : \"%s\"\n", > lib_name, dlerror()); > goto DLOPENFAIL; > @@ -85,7 +85,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > (osm_event_plugin_t *) dlsym(rc->handle, > OSM_EVENT_PLUGIN_IMPL_NAME); > if (!rc->impl) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > "Failed to find \"%s\" symbol in \"%s\" : \"%s\"\n", > OSM_EVENT_PLUGIN_IMPL_NAME, lib_name, dlerror()); > goto Exit; > @@ -93,7 +93,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > > /* Check the version to make sure this module will work with us */ > if (rc->impl->interface_version != OSM_EVENT_PLUGIN_INTERFACE_VER) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > "Error opening %s: " > "%s symbol is the wrong version %d != %d\n", > plugin_name, > @@ -103,19 +103,18 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) > goto Exit; > } > > - if (!rc->impl->construct) { > - OSM_LOG(p_log, OSM_LOG_ERROR, > - "%s symbol has no construct function\n", > + if (!rc->impl->create) { > + OSM_LOG(&osm->log, OSM_LOG_ERROR, > + "%s symbol has no create() function\n", > OSM_EVENT_PLUGIN_IMPL_NAME); > goto Exit; > } > > - rc->plugin_data = rc->impl->construct(p_log); > + rc->plugin_data = rc->impl->create(osm); > > if (!rc->plugin_data) > goto Exit; > > - rc->p_log = p_log; > rc->plugin_name = strdup(plugin_name); > return (rc); > > @@ -129,8 +128,8 @@ DLOPENFAIL: > void osm_epi_destroy(osm_epi_plugin_t * plugin) > { > if (plugin) { > - if (plugin->impl->destroy) > - plugin->impl->destroy(plugin->plugin_data); > + if (plugin->impl->delete) > + plugin->impl->delete(plugin->plugin_data); > dlclose(plugin->handle); > free(plugin->plugin_name); > free(plugin); > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index 0101b97..6cf4726 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -251,7 +251,7 @@ static void load_plugins(osm_opensm_t *osm, char *plugin_names) > > name = strtok_r(plugin_names, " \t\n", &p); > while (name && *name) { > - epi = osm_epi_construct(&osm->log, name); > + epi = osm_epi_construct(osm, name); > if (!epi) > osm_log(&osm->log, OSM_LOG_ERROR, > "cannot load plugin \'%s\'\n", name); > diff --git a/opensm/osmeventplugin/src/osmeventplugin.c b/opensm/osmeventplugin/src/osmeventplugin.c > index 6cc4c70..a56d3c3 100644 > --- a/opensm/osmeventplugin/src/osmeventplugin.c > +++ b/opensm/osmeventplugin/src/osmeventplugin.c > @@ -38,12 +38,15 @@ > #include > #include > #include > +#include > #include > #include > #include > -#include > #include > #include > +#include > +#include > +#include > > /** ========================================================================= > * This is a simple example plugin which logs some of the events the OSM > @@ -57,8 +60,11 @@ typedef struct _log_events { > > /** ========================================================================= > */ > -static void *construct(osm_log_t * osmlog) > +static void *construct(osm_opensm_t *osm) > { > + if (strcmp(osm->osm_version, OSM_VERSION)) > + return NULL; > + > _log_events_t *log = malloc(sizeof(*log)); > if (!log) > return (NULL); > @@ -66,14 +72,14 @@ static void *construct(osm_log_t * osmlog) > log->log_file = fopen(SAMPLE_PLUGIN_OUTPUT_FILE, "a+"); > > if (!(log->log_file)) { > - osm_log(osmlog, OSM_LOG_ERROR, > + osm_log(&osm->log, OSM_LOG_ERROR, > "Sample Event Plugin: Failed to open output file \"%s\"\n", > SAMPLE_PLUGIN_OUTPUT_FILE); > free(log); > return (NULL); > } > > - log->osmlog = osmlog; > + log->osmlog = &osm->log; > return ((void *)log); > } > > @@ -173,7 +179,7 @@ static void report(void *_log, osm_epi_event_id_t event_id, void *event_data) > */ > osm_event_plugin_t osm_event_plugin = { > interface_version:OSM_EVENT_PLUGIN_INTERFACE_VER, > - construct:construct, > - destroy:destroy, > + create:construct, > + delete:destroy, > report:report > }; From weiny2 at llnl.gov Thu Jun 26 10:54:25 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 26 Jun 2008 10:54:25 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <1214485229.13056.581.camel@hrosenstock-ws.xsigo.com> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <1214485229.13056.581.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080626105425.1cad6736.weiny2@llnl.gov> On Thu, 26 Jun 2008 06:00:29 -0700 Hal Rosenstock wrote: > On Wed, 2008-06-25 at 16:09 -0700, Ira Weiny wrote: > > diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c > > index 7279e14..01f9530 100644 > > --- a/libibmad/src/portid.c > > +++ b/libibmad/src/portid.c > ... > > @@ -65,29 +66,24 @@ char * > > portid2str(ib_portid_t *portid) > > { > > static char buf[1024] = "local"; > > + char drpath[512]; > > char *s = buf; > > int i; > > > > if (portid->lid > 0) { > > s += sprintf(s, "Lid %d", portid->lid); > > if (portid->grh_present) { > > - s += sprintf(s, " Gid 0x%" PRIx64 "%" PRIx64, > > - ntohll(*(uint64_t *)portid- > > >gid), > > - ntohll(*(uint64_t *)(portid- > > >gid+8))); > > + char gid[sizeof > > "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; > > + inet_ntop(AF_INET6, portid->gid, gid, sizeof > > (gid)); > > + s += sprintf(s, " Gid %s", gid); > > Should this be: > if (inet_ntop(AF_INET6, portid->gid, gid, sizeof (gid)) > s += sprintf(s, " Gid %s", gid); > > > } That is probably safer but the memory should be ok to print from. I will make the change. > > Also, I'm all for this change but I'm also for consistency (I may be a > minority of one on this) so I have to ask whether you plan on updating > diags and/or OpenSM for GID format. Well... I would like to start moving it over. I think it is more standard. > > Note also that if this is ported to Windows a wrapper for inet_ntop will > be needed. > Well... I don't want to break Windows but does Windows really not support this? I am not sure but the man page indicates it is POSIX. http://osdir.com/ml/web.wget.patches/2005-08/msg00008.html Ira From hrosenstock at xsigo.com Thu Jun 26 11:00:14 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 11:00:14 -0700 Subject: [ofa-general] [PATCH] opensm/man/opensm.8.in: Clarify ignore-guids file format Message-ID: <1214503214.13056.625.camel@hrosenstock-ws.xsigo.com> opensm/man/opensm.8.in: Clarify ignore-guids file format Signed-off-by: Hal Rosenstock diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index d5cd820..a14b5b5 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -194,7 +194,7 @@ Note that this option only appears if OpenSM was built with .TP \fB\-i\fR, \fB\-ignore-guids\fR This option provides the means to define a set of ports -(by guid) that will be ignored by the link load +(by node guid and port number) that will be ignored by the link load equalization algorithm. .TP \fB\-x\fR, \fB\-\-honor_guid2lid\fR From hrosenstock at xsigo.com Thu Jun 26 11:06:11 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 11:06:11 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080626105425.1cad6736.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <1214485229.13056.581.camel@hrosenstock-ws.xsigo.com> <20080626105425.1cad6736.weiny2@llnl.gov> Message-ID: <1214503571.13056.631.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-26 at 10:54 -0700, Ira Weiny wrote: > On Thu, 26 Jun 2008 06:00:29 -0700 > Hal Rosenstock wrote: > > > On Wed, 2008-06-25 at 16:09 -0700, Ira Weiny wrote: > > > diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c > > > index 7279e14..01f9530 100644 > > > --- a/libibmad/src/portid.c > > > +++ b/libibmad/src/portid.c > > ... > > > @@ -65,29 +66,24 @@ char * > > > portid2str(ib_portid_t *portid) > > > { > > > static char buf[1024] = "local"; > > > + char drpath[512]; > > > char *s = buf; > > > int i; > > > > > > if (portid->lid > 0) { > > > s += sprintf(s, "Lid %d", portid->lid); > > > if (portid->grh_present) { > > > - s += sprintf(s, " Gid 0x%" PRIx64 "%" PRIx64, > > > - ntohll(*(uint64_t *)portid- > > > >gid), > > > - ntohll(*(uint64_t *)(portid- > > > >gid+8))); > > > + char gid[sizeof > > > "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; > > > + inet_ntop(AF_INET6, portid->gid, gid, sizeof > > > (gid)); > > > + s += sprintf(s, " Gid %s", gid); > > > > Should this be: > > if (inet_ntop(AF_INET6, portid->gid, gid, sizeof (gid)) > > s += sprintf(s, " Gid %s", gid); > > > > > } > > That is probably safer but the memory should be ok to print from. I will make > the change. Per the man page, it doesn't look like it would fail but it is safer as you point out. > > > > > Also, I'm all for this change but I'm also for consistency (I may be a > > minority of one on this) so I have to ask whether you plan on updating > > diags and/or OpenSM for GID format. > > Well... I would like to start moving it over. I think it is more standard. I hope we get this all converted over rapidly. > > > > Note also that if this is ported to Windows a wrapper for inet_ntop will > > be needed. > > > > Well... I don't want to break Windows It's just a future porting issue if this is ported over. > but does Windows really not support this? Not AFAIT although a wrapper looks pretty simple. > I am not sure but the man page indicates it is POSIX. > > http://osdir.com/ml/web.wget.patches/2005-08/msg00008.html Not sure we would want to require POSIX for Windows. -- Hal > Ira > From hrosenstock at xsigo.com Thu Jun 26 12:42:52 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 12:42:52 -0700 Subject: [ofa-general] Multicast traffic generates Bad P_Key trap in SM when working in partial member setup In-Reply-To: References: <39C75744D164D948A170E9792AF8E7CA013E6E2B@exil.voltaire.com> <1213266626.14047.419.camel@hrosenstock-ws.xsigo.com> <1213270788.14047.451.camel@hrosenstock-ws.xsigo.com> <1213278134.14047.498.camel@hrosenstock-ws.xsigo.com> Message-ID: <1214509372.13056.645.camel@hrosenstock-ws.xsigo.com> On Thu, 2008-06-12 at 16:44 +0300, Olga Shern (Voltaire) wrote: > Would you try this with OpenSM (and validate your theory about > getting > switch bad PKey traps v. end port bad PKey traps) or does VSM > have such > a mode (ingress/egress partition filtering) ? > > -- Hal > > Yes, I will test it with OpenSM Any update on this ? Thanks. -- Hal From hrosenstock at xsigo.com Thu Jun 26 14:36:43 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Thu, 26 Jun 2008 14:36:43 -0700 Subject: [ofa-general] [PATCH] opensm/osm_mcast_mgr.c: Only send MCG delete notice if not previously done Message-ID: <1214516203.13056.654.camel@hrosenstock-ws.xsigo.com> opensm/osm_mcast_mgr.c: Only send MCG delete notice if not previously done This eliminates duplicated MCG delete notices when MCG is removed Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index c3167a8..bb78c50 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -1152,9 +1152,13 @@ mcast_mgr_process_mgrp(osm_sm_t * sm, OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Destroying mgrp with lid:0x%X\n", cl_ntoh16(p_mgrp->mlid)); - /* Send a Report to any InformInfo registered for - Trap 67 : MCGroup delete */ - osm_mgrp_send_delete_notice(sm->p_subn, sm->p_log, p_mgrp); + if (p_mgrp->to_be_deleted == FALSE) { + p_mgrp->to_be_deleted = TRUE; + /* Send a Report to any InformInfo registered for + Trap 67 : MCGroup delete */ + osm_mgrp_send_delete_notice(sm->p_subn, sm->p_log, + p_mgrp); + } sm->p_subn->mgroups[cl_ntoh16(p_mgrp->mlid) - IB_LID_MCAST_START_HO] = NULL; osm_mgrp_delete(p_mgrp); } From swise at opengridcomputing.com Thu Jun 26 15:59:45 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 26 Jun 2008 17:59:45 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. Message-ID: <20080626225945.27989.73122.stgit@dell3.ogc.int> From: Steve Wise Handling the zero-stag in recv WRs requires some extra logic: - Only set the QP_PRIV bit if it is a kernel mode qp. - Add a zero-stag build function for recv wrs. The uP needs a PBL allocated and passed down in the recv WR so it can construct a hw pbl for the zero-stag sges. Note: we need to place a few restrictions on zero-stag usage: 1) all SGEs in a RECV WR must either be zero-stag or not. No mixing. 2) an individual SGE length cannot exceed 128MB for a zero-stag SGE. This is ok since its not possible to allocate such a large chunk of pinned contiguous dma-coherent memory. - Add an optimized non zero-stag recv wr format for kernel users. This is needed to optimize both zero and non-zero stag cracking in the recv path for kernel users. - Remove the iwch_ prefix from the static build functions. TODO: Bump the required fw major number since the driver/fw interface has changed. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 10 +- drivers/infiniband/hw/cxgb3/cxio_wr.h | 13 ++- drivers/infiniband/hw/cxgb3/iwch_provider.c | 4 - drivers/infiniband/hw/cxgb3/iwch_qp.c | 128 +++++++++++++++++++++++---- 4 files changed, 129 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 340e418..f0c81d3 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -278,7 +278,7 @@ int cxio_create_qp(struct cxio_rdev *rdev_p, u32 kernel_domain, if (!wq->qpid) return -ENOMEM; - wq->rq = kzalloc(depth * sizeof(u64), GFP_KERNEL); + wq->rq = kzalloc(depth * sizeof(struct t3_swrq), GFP_KERNEL); if (!wq->rq) goto err1; @@ -302,6 +302,7 @@ int cxio_create_qp(struct cxio_rdev *rdev_p, u32 kernel_domain, if (!kernel_domain) wq->udb = (u64)rdev_p->rnic_info.udbell_physbase + (wq->qpid << rdev_p->qpshift); + wq->rdev = rdev_p; PDBG("%s qpid 0x%x doorbell 0x%p udb 0x%llx\n", __func__, wq->qpid, wq->doorbell, (unsigned long long) wq->udb); return 0; @@ -1266,13 +1267,14 @@ proc_cqe: wq->sq_rptr = CQE_WRID_SQ_WPTR(*hw_cqe); PDBG("%s completing sq idx %ld\n", __func__, Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2)); - *cookie = (wq->sq + - Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2))->wr_id; + *cookie = wq->sq[Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2)].wr_id; wq->sq_rptr++; } else { PDBG("%s completing rq idx %ld\n", __func__, Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)); - *cookie = *(wq->rq + Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)); + *cookie = wq->rq[Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)].wr_id; + if (wq->rq[Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)].pbl_addr) + cxio_hal_pblpool_free(wq->rdev, wq->rq[Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)].pbl_addr, T3_STAG0_PBL_SIZE); wq->rq_rptr++; } diff --git a/drivers/infiniband/hw/cxgb3/cxio_wr.h b/drivers/infiniband/hw/cxgb3/cxio_wr.h index 952074f..4a8b7fb 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_wr.h +++ b/drivers/infiniband/hw/cxgb3/cxio_wr.h @@ -39,6 +39,9 @@ #define T3_MAX_SGE 4 #define T3_MAX_INLINE 64 +#define T3_STAG0_PBL_SIZE (2 * T3_MAX_SGE << 3) +#define T3_STAG0_MAX_PBE_LEN (128 * 1024 * 1024) +#define T3_STAG0_PAGE_SHIFT 15 #define Q_EMPTY(rptr,wptr) ((rptr)==(wptr)) #define Q_FULL(rptr,wptr,size_log2) ( (((wptr)-(rptr))>>(size_log2)) && \ @@ -664,6 +667,11 @@ struct t3_swsq { int signaled; }; +struct t3_swrq { + __u64 wr_id; + __u32 pbl_addr; +}; + /* * A T3 WQ implements both the SQ and RQ. */ @@ -680,14 +688,15 @@ struct t3_wq { u32 sq_wptr; /* sq_wptr - sq_rptr == count of */ u32 sq_rptr; /* pending wrs */ u32 sq_size_log2; /* sq size */ - u64 *rq; /* SW RQ (holds consumer wr_ids */ + struct t3_swrq *rq; /* SW RQ (holds consumer wr_ids */ u32 rq_wptr; /* rq_wptr - rq_rptr == count of */ u32 rq_rptr; /* pending wrs */ - u64 *rq_oldest_wr; /* oldest wr on the SW RQ */ + struct t3_swrq *rq_oldest_wr; /* oldest wr on the SW RQ */ u32 rq_size_log2; /* rq size */ u32 rq_addr; /* rq adapter address */ void __iomem *doorbell; /* kernel db */ u64 udb; /* user db if any */ + struct cxio_rdev *rdev; }; struct t3_cq { diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 3101e3c..72422ed 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1005,10 +1005,10 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, qhp->ibqp.qp_num = qhp->wq.qpid; init_timer(&(qhp->timer)); PDBG("%s sq_num_entries %d, rq_num_entries %d " - "qpid 0x%0x qhp %p dma_addr 0x%llx size %d\n", + "qpid 0x%0x qhp %p dma_addr 0x%llx size %d rq_addr 0x%x\n", __func__, qhp->attr.sq_num_entries, qhp->attr.rq_num_entries, qhp->wq.qpid, qhp, (unsigned long long) qhp->wq.dma_addr, - 1 << qhp->wq.size_log2); + 1 << qhp->wq.size_log2, qhp->wq.rq_addr); return &qhp->ibqp; } diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 169056e..0d0281d 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -33,10 +33,11 @@ #include "iwch.h" #include "iwch_cm.h" #include "cxio_hal.h" +#include "cxio_resource.h" #define NO_SUPPORT -1 -static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, u8 * flit_cnt) { int i; @@ -81,7 +82,7 @@ static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { int i; @@ -122,7 +123,7 @@ static int iwch_build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { if (wr->num_sge > 1) @@ -140,7 +141,7 @@ static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt, int *wr_cnt, struct t3_wq *wq) { int i; @@ -182,7 +183,7 @@ static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { wqe->local_inv.stag = cpu_to_be32(wr->ex.invalidate_rkey); @@ -241,23 +242,107 @@ static int iwch_sgl2pbl_map(struct iwch_dev *rhp, struct ib_sge *sg_list, return 0; } -static int iwch_build_rdma_recv(struct iwch_dev *rhp, union t3_wr *wqe, +static int build_rdma_recv(struct iwch_qp *qhp, union t3_wr *wqe, struct ib_recv_wr *wr) { - int i; - if (wr->num_sge > T3_MAX_SGE) - return -EINVAL; + int i, err = 0; + u32 pbl_addr[T3_MAX_SGE]; + u8 page_size[T3_MAX_SGE]; + + err = iwch_sgl2pbl_map(qhp->rhp, wr->sg_list, wr->num_sge, pbl_addr, + page_size); + if (err) + return err; + wqe->recv.pagesz[0] = page_size[0]; + wqe->recv.pagesz[1] = page_size[1]; + wqe->recv.pagesz[2] = page_size[2]; + wqe->recv.pagesz[3] = page_size[3]; wqe->recv.num_sgle = cpu_to_be32(wr->num_sge); for (i = 0; i < wr->num_sge; i++) { wqe->recv.sgl[i].stag = cpu_to_be32(wr->sg_list[i].lkey); wqe->recv.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); + + /* to in the WQE == the offset into the page */ + wqe->recv.sgl[i].to = cpu_to_be64(((u32) wr->sg_list[i].addr) % + (1UL << (12 + page_size[i]))); + + /* pbl_addr is the adapters address in the PBL */ + wqe->recv.pbl_addr[i] = cpu_to_be32(pbl_addr[i]); + } + for (; i < T3_MAX_SGE; i++) { + wqe->recv.sgl[i].stag = 0; + wqe->recv.sgl[i].len = 0; + wqe->recv.sgl[i].to = 0; + wqe->recv.pbl_addr[i] = 0; + } + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].wr_id = wr->wr_id; + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].pbl_addr = 0; + return 0; +} + +static int build_zero_stag_recv(struct iwch_qp *qhp, union t3_wr *wqe, + struct ib_recv_wr *wr) +{ + int i; + u32 pbl_addr; + u32 pbl_offset; + + + /* + * The T3 HW requires the PBL in the hw recv descriptor to reference + * a PBL entry. So we allocate the max needed PBL memory here and pass + * it to the uP in the recv WR. The uP will build the PBL and setup + * the hw recv descriptor. + */ + pbl_addr = cxio_hal_pblpool_alloc(&qhp->rhp->rdev, T3_STAG0_PBL_SIZE); + if (!pbl_addr) { + return -ENOMEM; + } + + /* + * Compute the 8B aligned offset. + */ + pbl_offset = (pbl_addr - qhp->rhp->rdev.rnic_info.pbl_base) >> 3; + + wqe->recv.num_sgle = cpu_to_be32(wr->num_sge); + + for (i = 0; i < wr->num_sge; i++) { + + /* + * Use a 128MB page size. This and an imposed 128MB + * sge length limit allows us to require only a 2-entry HW + * PBL for each SGE. This restriction is acceptable since + * since it is not possible to allocate 128MB of contiguous + * DMA coherent memory! + */ + if (wr->sg_list[i].length > T3_STAG0_MAX_PBE_LEN) + return -EINVAL; + wqe->recv.pagesz[i] = T3_STAG0_PAGE_SHIFT; + + /* + * T3 restricts a recv to all zero-stag or all non-zero-stag. + */ + if (wr->sg_list[i].lkey != 0) + return -EINVAL; + wqe->recv.sgl[i].stag = 0; + wqe->recv.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); wqe->recv.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); + wqe->recv.pbl_addr[i] = cpu_to_be32(pbl_offset); + pbl_offset += 2; } for (; i < T3_MAX_SGE; i++) { + wqe->recv.pagesz[i] = 0; wqe->recv.sgl[i].stag = 0; wqe->recv.sgl[i].len = 0; wqe->recv.sgl[i].to = 0; + wqe->recv.pbl_addr[i] = 0; } + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].wr_id = wr->wr_id; + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].pbl_addr = pbl_addr; return 0; } @@ -309,17 +394,17 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, if (wr->send_flags & IB_SEND_FENCE) t3_wr_flags |= T3_READ_FENCE_FLAG; t3_wr_opcode = T3_WR_SEND; - err = iwch_build_rdma_send(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_send(wqe, wr, &t3_wr_flit_cnt); break; case IB_WR_RDMA_WRITE: case IB_WR_RDMA_WRITE_WITH_IMM: t3_wr_opcode = T3_WR_WRITE; - err = iwch_build_rdma_write(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_write(wqe, wr, &t3_wr_flit_cnt); break; case IB_WR_RDMA_READ: t3_wr_opcode = T3_WR_READ; t3_wr_flags = 0; /* T3 reads are always signaled */ - err = iwch_build_rdma_read(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_read(wqe, wr, &t3_wr_flit_cnt); if (err) break; sqp->read_len = wqe->read.local_len; @@ -328,14 +413,14 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case IB_WR_FAST_REG_MR: t3_wr_opcode = T3_WR_FASTREG; - err = iwch_build_fastreg(wqe, wr, &t3_wr_flit_cnt, + err = build_fastreg(wqe, wr, &t3_wr_flit_cnt, &wr_cnt, &qhp->wq); break; case IB_WR_LOCAL_INV: if (wr->send_flags & IB_SEND_FENCE) t3_wr_flags |= T3_LOCAL_FENCE_FLAG; t3_wr_opcode = T3_WR_INV_STAG; - err = iwch_build_inv_stag(wqe, wr, &t3_wr_flit_cnt); + err = build_inv_stag(wqe, wr, &t3_wr_flit_cnt); break; default: PDBG("%s post of type=%d TBD!\n", __func__, @@ -394,18 +479,24 @@ int iwch_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, return -EINVAL; } while (wr) { + if (wr->num_sge > T3_MAX_SGE) { + err = -EINVAL; + *bad_wr = wr; + break; + } idx = Q_PTR2IDX(qhp->wq.wptr, qhp->wq.size_log2); wqe = (union t3_wr *) (qhp->wq.queue + idx); if (num_wrs) - err = iwch_build_rdma_recv(qhp->rhp, wqe, wr); + if (wr->sg_list[0].lkey) + err = build_rdma_recv(qhp, wqe, wr); + else + err = build_zero_stag_recv(qhp, wqe, wr); else err = -ENOMEM; if (err) { *bad_wr = wr; break; } - qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, qhp->wq.rq_size_log2)] = - wr->wr_id; build_fw_riwrh((void *) wqe, T3_WR_RCV, T3_COMPLETION_FLAG, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), 0, sizeof(struct t3_receive_wr) >> 3, T3_SOPEOP); @@ -806,7 +897,8 @@ static int rdma_init(struct iwch_dev *rhp, struct iwch_qp *qhp, init_attr.qp_dma_size = (1UL << qhp->wq.size_log2); init_attr.rqe_count = iwch_rqes_posted(qhp); init_attr.flags = qhp->attr.mpa_attr.initiator ? MPA_INITIATOR : 0; - init_attr.flags |= capable(CAP_NET_BIND_SERVICE) ? PRIV_QP : 0; + if (!qhp->ibqp.uobject) + init_attr.flags |= PRIV_QP; if (peer2peer) { init_attr.rtr_type = RTR_READ; if (init_attr.ord == 0 && qhp->attr.mpa_attr.initiator) From ErvinrailbirdBriggs at poets.org Thu Jun 26 13:13:57 2008 From: ErvinrailbirdBriggs at poets.org (Caleb Briggs) Date: Thu, 26 Jun 2008 23:13:57 +0300 Subject: [ofa-general] Obtain an accredited BA, MA or PhD in 30 days! Message-ID: <0IX074EJXVWDA243@poets.org> A degree with no classes or test in your spare time Get your degree without classes or test Call 24/7 on.... For US: 1-419-735-9250 Outside US: +1-419-735-9250 From a-akatz at acaciavp.com Thu Jun 26 21:45:35 2008 From: a-akatz at acaciavp.com (Fernando Mathis) Date: Fri, 27 Jun 2008 12:45:35 +0800 Subject: [ofa-general] did you miss me Message-ID: <018155522.67331116724124@acaciavp.com> Hello! I am bored this evening. I am nice girl that would like to chat with you. Email me at Anneli at YourGLo.info only, because I am using my friend's email to write this. Hope you like my pictures. -------------- next part -------------- A non-text attachment was scrubbed... Name: me Type: image/gif Size: 42483 bytes Desc: not available URL: From 51juanitavr at bestmed.co.za Thu Jun 26 20:15:16 2008 From: 51juanitavr at bestmed.co.za (elliot hartley) Date: Fri, 27 Jun 2008 03:15:16 +0000 Subject: [ofa-general] Best of Chloe, Paul Smith, UGG Message-ID: <000401c8d813$03a989c6$aadf63a1@umcinbd> The most popular and broad luxury store for shoes and bags is just one click away. Recommended by hundred thousands of satisfied customers all around the world, we carry many of populars brands including: Chanel Adidas Coach Dior Prada Shoes Here you willc see thousands of stunning designs for shoes, and leather products, at at cheapest priceE. Sale ends this weekend, so visit web site today and start pampering yourself and your loved second half! - Visit our site: www.pickatell[DOT]com (copy this link then replace "[DOT]" to ".") From PennynomenclatureCorrea at careerbuilder.com Fri Jun 27 07:12:32 2008 From: PennynomenclatureCorrea at careerbuilder.com (Carole Gunn) Date: Fri, 27 Jun 2008 12:12:32 -0200 Subject: [ofa-general] 30 days to a new Bachelor, Master or PhD Message-ID: <5IX892EJXVWDA879@careerbuilder.com> Get your degree without classes or test Need a better job? Get a better degree! Call 24/7 on.... For US: 1-419-735-9250 Outside US: +1-419-735-9250 From vlad at lists.openfabrics.org Fri Jun 27 02:38:10 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 27 Jun 2008 02:38:10 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080627-0200 daily build status Message-ID: <20080627093810.32B8BE60CD2@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26-rc6 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080627-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From Sumit.Gaur at Sun.COM Fri Jun 27 04:31:52 2008 From: Sumit.Gaur at Sun.COM (Sumit Gaur - Sun Microsystem) Date: Fri, 27 Jun 2008 17:01:52 +0530 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <1214485794.13056.592.camel@hrosenstock-ws.xsigo.com> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> <48633946.3070706@Sun.COM> <1214485794.13056.592.camel@hrosenstock-ws.xsigo.com> Message-ID: <4864CFA8.4010300@Sun.COM> Find my answers below:- Hal Rosenstock wrote: > Hi Sumit, > > On Thu, 2008-06-26 at 12:07 +0530, Sumit Gaur - Sun Microsystem wrote: > >>Hi Hal, >> >>Hal Rosenstock wrote: >> >>>>>>I am sending only request for >>>>>> >>>>>> rpc.mgtclass = IB_PERFORMANCE_CLASS; >>>>>> rpc.method = IB_MAD_METHOD_GET; >>>>>> >>>>>>at every one second. >>> >>> >>>Does perfquery work reliably with the same node(s) you are having >>>trouble with ? >>> >>>Does your app follow what perfquery does ? >> >>Yes, perfquery works fine and I am following similar way of implementation. Here >>is the output. I think difference is there in Load. I am sending 4 GS request >>per second basis and some got passed and some got timeout(110) or recv failed. > > > Can you elaborate on the multiple sends ? Are they outstanding > concurrently ? Are they to the same destination or different ones ? Are > they from a single or multiple threads ? No they are sending sequentially(mutex enabled) no concurrency but timeout for umad_recv is 100ms. Yes they are for same destination. They all are from single threads. I still point out same I configure for SMP and no failure. > > >># perfquery >># Port counters: Lid 393 port 1 >>PortSelect:......................1 >>CounterSelect:...................0x0000 >>SymbolErrors:....................0 >>LinkRecovers:....................0 >>LinkDowned:......................0 >>RcvErrors:.......................0 >>RcvRemotePhysErrors:.............0 >>RcvSwRelayErrors:................0 >>XmtDiscards:.....................0 >>XmtConstraintErrors:.............0 >>RcvConstraintErrors:.............0 >>LinkIntegrityErrors:.............0 >>ExcBufOverrunErrors:.............0 >>VL15Dropped:.....................0 >>XmtData:.........................65899728 >>RcvData:.........................65899656 >>XmtPkts:.........................915274 >>RcvPkts:.........................915273 >> >> > > > OK but you had said the received packet was corrupted. Maybe a nit, but > with timeout and other errors, the receive packet is invalid rather than > corrupted (an app shouldn't be looking at the response in the error > cases). > > >>>The underlying question is why are you getting the timeout relatively >>>frequently so I recommend checking all the error counters along the >>>path. >> >># Checking Ca: nodeguid 0x00144fa5e9ce001c >>Node check lid 392: OK >>Error check on lid 392 (HCA-1) port all: OK > > > Is that the requester or responder ? It's not the entire path. Maybe the > simplest thing is: what does ibchecknet or ibcheckerrors say ? > I am using lid 106 [root at o4test65 tmp]# ibchecknet #warn: counter SymbolErrors = 65535 (threshold 10) Error check on lid 106 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 18 (threshold 10) Error check on lid 4 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 19 (threshold 10) Error check on lid 9 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 26 (threshold 10) #warn: counter LinkDowned = 13 (threshold 10) #warn: counter RcvErrors = 27 (threshold 10) Error check on lid 10 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 255 (threshold 10) Error check on lid 1 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 1968 (threshold 10) Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 1967 (threshold 10) Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port 15: FAILED # Checked Switch: nodeguid 0x00144f0000a61390 with failure #warn: counter SymbolErrors = 65535 (threshold 10) Error check on lid 7 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkDowned = 12 (threshold 10) Error check on lid 3 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 15 (threshold 10) #warn: counter LinkDowned = 12 (threshold 10) Error check on lid 2 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 15 (threshold 10) #warn: counter LinkDowned = 12 (threshold 10) Error check on lid 11 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) Error check on lid 6 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 255 (threshold 10) #warn: counter RcvErrors = 445 (threshold 10) Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: Logical link state is Initialize Port check lid 12 port 15: FAILED # Checked Switch: nodeguid 0x00144f0000a61397 with failure #warn: Logical link state is Initialize Port check lid 12 port 14: FAILED #warn: Logical link state is Initialize Port check lid 12 port 13: FAILED #warn: counter LinkRecovers = 11 (threshold 10) Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port 13: FAILED # Checking Ca: nodeguid 0x00144fa5e9ce001c # Checking Ca: nodeguid 0x00144fa5e9ce000c # Checking Ca: nodeguid 0x00144fa5e9ce0014 # Checking Ca: nodeguid 0x00144fa5e9ce0004 ## Summary: 29 nodes checked, 0 bad nodes found ## 359 ports checked, 3 bad ports found ## 2 ports have errors beyond threshold > In any case, based on your comments above about perfquery working > reliably, I'm skeptical whether this is the issue but it's best to rule > it out. [root at o4test65 tmp]# ibcheckerrors #warn: counter SymbolErrors = 65535 (threshold 10) Error check on lid 106 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 18 (threshold 10) Error check on lid 4 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 19 (threshold 10) Error check on lid 9 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 26 (threshold 10) #warn: counter LinkDowned = 13 (threshold 10) #warn: counter RcvErrors = 27 (threshold 10) Error check on lid 10 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 255 (threshold 10) Error check on lid 1 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 2081 (threshold 10) Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 2080 (threshold 10) Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port 15: FAILED # Checked Switch: nodeguid 0x00144f0000a61390 with failure #warn: counter SymbolErrors = 65535 (threshold 10) Error check on lid 7 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkDowned = 12 (threshold 10) Error check on lid 3 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 15 (threshold 10) #warn: counter LinkDowned = 12 (threshold 10) Error check on lid 2 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 15 (threshold 10) #warn: counter LinkDowned = 12 (threshold 10) Error check on lid 11 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) Error check on lid 6 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter SymbolErrors = 65535 (threshold 10) #warn: counter LinkRecovers = 255 (threshold 10) #warn: counter RcvErrors = 445 (threshold 10) Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port all: FAILED #warn: counter LinkRecovers = 11 (threshold 10) Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port 13: FAILED # Checked Switch: nodeguid 0x00144f0000a61397 with failure ## Summary: 29 nodes checked, 0 bad nodes found ## 359 ports checked, 2 ports have errors beyond threshold > > >>>Are you sure the request gets to the responder ? Does the responder >>>respond and it doesn't make it back ? >> >>yes As I told It is not 100% failure, It is 30% to 40% failure. But Why ? > > > I don't know enough about what is different about your app yet to say > more right now. > > -- Hal > > >>>-- Hal >>> >>> > > From hrosenstock at xsigo.com Fri Jun 27 06:15:01 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 06:15:01 -0700 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <4864CFA8.4010300@Sun.COM> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> <48633946.3070706@Sun.COM> <1214485794.13056.592.camel@hrosenstock-ws.xsigo.com> <4864CFA8.4010300@Sun.COM> Message-ID: <1214572501.13056.678.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-27 at 17:01 +0530, Sumit Gaur - Sun Microsystem wrote: > Find my answers below:- > > Hal Rosenstock wrote: > > Hi Sumit, > > > > On Thu, 2008-06-26 at 12:07 +0530, Sumit Gaur - Sun Microsystem wrote: > > > >>Hi Hal, > >> > >>Hal Rosenstock wrote: > >> > >>>>>>I am sending only request for > >>>>>> > >>>>>> rpc.mgtclass = IB_PERFORMANCE_CLASS; > >>>>>> rpc.method = IB_MAD_METHOD_GET; > >>>>>> > >>>>>>at every one second. > >>> > >>> > >>>Does perfquery work reliably with the same node(s) you are having > >>>trouble with ? > >>> > >>>Does your app follow what perfquery does ? > >> > >>Yes, perfquery works fine and I am following similar way of implementation. Here > >>is the output. I think difference is there in Load. I am sending 4 GS request > >>per second basis and some got passed and some got timeout(110) or recv failed. > > Can you elaborate on the multiple sends ? Are they outstanding > > concurrently ? Are they to the same destination or different ones ? Are > > they from a single or multiple threads ? > No they are sending sequentially(mutex enabled) no concurrency but timeout for > umad_recv is 100ms. Can you try increasing that to see if there is some threshold where it works more reliably ? Does it work better at say 200 msec (as you said your rate was 4/sec) ? The default timeout used in the diags is 1 sec. BTW, this could explain the timeouts but I'm not sure about the other errors you mentioned. > Yes they are for same destination. They all are from single > threads. I still point out same I configure for SMP and no failure. > > > > > >># perfquery > >># Port counters: Lid 393 port 1 > >>PortSelect:......................1 > >>CounterSelect:...................0x0000 > >>SymbolErrors:....................0 > >>LinkRecovers:....................0 > >>LinkDowned:......................0 > >>RcvErrors:.......................0 > >>RcvRemotePhysErrors:.............0 > >>RcvSwRelayErrors:................0 > >>XmtDiscards:.....................0 > >>XmtConstraintErrors:.............0 > >>RcvConstraintErrors:.............0 > >>LinkIntegrityErrors:.............0 > >>ExcBufOverrunErrors:.............0 > >>VL15Dropped:.....................0 > >>XmtData:.........................65899728 > >>RcvData:.........................65899656 > >>XmtPkts:.........................915274 > >>RcvPkts:.........................915273 > >> > >> > > > > > > > > OK but you had said the received packet was corrupted. Maybe a nit, but > > with timeout and other errors, the receive packet is invalid rather than > > corrupted (an app shouldn't be looking at the response in the error > > cases). > > > > > >>>The underlying question is why are you getting the timeout relatively > >>>frequently so I recommend checking all the error counters along the > >>>path. > >> > >># Checking Ca: nodeguid 0x00144fa5e9ce001c > >>Node check lid 392: OK > >>Error check on lid 392 (HCA-1) port all: OK > > > > > > Is that the requester or responder ? It's not the entire path. Maybe the > > simplest thing is: what does ibchecknet or ibcheckerrors say ? > > > > I am using lid 106 > > [root at o4test65 tmp]# ibchecknet > #warn: counter SymbolErrors = 65535 (threshold 10) > Error check on lid 106 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 18 (threshold 10) > Error check on lid 4 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 19 (threshold 10) > Error check on lid 9 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 26 (threshold 10) > #warn: counter LinkDowned = 13 (threshold 10) > #warn: counter RcvErrors = 27 (threshold 10) > Error check on lid 10 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 255 (threshold 10) > Error check on lid 1 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 1968 (threshold 10) > Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 1967 (threshold 10) > Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port 15: > FAILED > # Checked Switch: nodeguid 0x00144f0000a61390 with failure > #warn: counter SymbolErrors = 65535 (threshold 10) > Error check on lid 7 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkDowned = 12 (threshold 10) > Error check on lid 3 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 15 (threshold 10) > #warn: counter LinkDowned = 12 (threshold 10) > Error check on lid 2 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 15 (threshold 10) > #warn: counter LinkDowned = 12 (threshold 10) > Error check on lid 11 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > Error check on lid 6 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 255 (threshold 10) > #warn: counter RcvErrors = 445 (threshold 10) > Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: Logical link state is Initialize > Port check lid 12 port 15: FAILED > # Checked Switch: nodeguid 0x00144f0000a61397 with failure > #warn: Logical link state is Initialize > Port check lid 12 port 14: FAILED > #warn: Logical link state is Initialize > Port check lid 12 port 13: FAILED > #warn: counter LinkRecovers = 11 (threshold 10) > Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port 13: > FAILED > > # Checking Ca: nodeguid 0x00144fa5e9ce001c > > # Checking Ca: nodeguid 0x00144fa5e9ce000c > > # Checking Ca: nodeguid 0x00144fa5e9ce0014 > > # Checking Ca: nodeguid 0x00144fa5e9ce0004 > > ## Summary: 29 nodes checked, 0 bad nodes found > ## 359 ports checked, 3 bad ports found > ## 2 ports have errors beyond threshold > > > In any case, based on your comments above about perfquery working > > reliably, I'm skeptical whether this is the issue but it's best to rule > > it out. > [root at o4test65 tmp]# ibcheckerrors > #warn: counter SymbolErrors = 65535 (threshold 10) > Error check on lid 106 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 18 (threshold 10) > Error check on lid 4 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 19 (threshold 10) > Error check on lid 9 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 26 (threshold 10) > #warn: counter LinkDowned = 13 (threshold 10) > #warn: counter RcvErrors = 27 (threshold 10) > Error check on lid 10 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 255 (threshold 10) > Error check on lid 1 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 2081 (threshold 10) > Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 2080 (threshold 10) > Error check on lid 5 (MT47396 Infiniscale-III Mellanox Technologies) port 15: > FAILED > # Checked Switch: nodeguid 0x00144f0000a61390 with failure > #warn: counter SymbolErrors = 65535 (threshold 10) > Error check on lid 7 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkDowned = 12 (threshold 10) > Error check on lid 3 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 15 (threshold 10) > #warn: counter LinkDowned = 12 (threshold 10) > Error check on lid 2 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 15 (threshold 10) > #warn: counter LinkDowned = 12 (threshold 10) > Error check on lid 11 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > Error check on lid 6 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter SymbolErrors = 65535 (threshold 10) > #warn: counter LinkRecovers = 255 (threshold 10) > #warn: counter RcvErrors = 445 (threshold 10) > Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port all: > FAILED > #warn: counter LinkRecovers = 11 (threshold 10) > Error check on lid 12 (MT47396 Infiniscale-III Mellanox Technologies) port 13: > FAILED > # Checked Switch: nodeguid 0x00144f0000a61397 with failure > > ## Summary: 29 nodes checked, 0 bad nodes found > ## 359 ports checked, 2 ports have errors beyond threshold Looks like there are some issues here to debug in your subnet. It might help to clear the counters and see what is actively going on to isolate these issues. This could factor into those other errors you are seeing. -- Hal > >>>Are you sure the request gets to the responder ? Does the responder > >>>respond and it doesn't make it back ? > >> > >>yes As I told It is not 100% failure, It is 30% to 40% failure. But Why ? > > > > > > I don't know enough about what is different about your app yet to say > > more right now. > > > > -- Hal > > > > > >>>-- Hal > >>> > >>> > > > > From Sumit.Gaur at Sun.COM Fri Jun 27 06:58:47 2008 From: Sumit.Gaur at Sun.COM (Sumit Gaur - Sun Microsystem) Date: Fri, 27 Jun 2008 19:28:47 +0530 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <1214572501.13056.678.camel@hrosenstock-ws.xsigo.com> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> <48633946.3070706@Sun.COM> <1214485794.13056.592.camel@hrosenstock-ws.xsigo.com> <4864CFA8.4010300@Sun.COM> <1214572501.13056.678.camel@hrosenstock-ws.xsigo.com> Message-ID: <4864F217.8080508@Sun.COM> hi Hal, Hal Rosenstock wrote: > >>>Can you elaborate on the multiple sends ? Are they outstanding >>>concurrently ? Are they to the same destination or different ones ? Are >>>they from a single or multiple threads ? >> >>No they are sending sequentially(mutex enabled) no concurrency but timeout for >>umad_recv is 100ms. > > > Can you try increasing that to see if there is some threshold where it > works more reliably ? Does it work better at say 200 msec (as you said > your rate was 4/sec) ? The default timeout used in the diags is 1 sec. yes, I tried increasing it upto 3000ms but condition not improved much (it *reduced* timeout failures but *no reduction* in recv_fail and this piling up number of request per second too at client side). Also I tried same code on two separate subnet and similar problem come across. Could be a network issue but is it a fact that GMP request takes much more time then SMP request (for ideal subnet) ? > > Looks like there are some issues here to debug in your subnet. It might > help to clear the counters and see what is actively going on to isolate > these issues. This could factor into those other errors you are seeing. > > -- Hal > > From sashak at voltaire.com Fri Jun 27 08:03:00 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 18:03:00 +0300 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <1214503571.13056.631.camel@hrosenstock-ws.xsigo.com> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <1214485229.13056.581.camel@hrosenstock-ws.xsigo.com> <20080626105425.1cad6736.weiny2@llnl.gov> <1214503571.13056.631.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627150300.GB29456@sashak.voltaire.com> On 11:06 Thu 26 Jun , Hal Rosenstock wrote: > > but does Windows really not support this? > > Not AFAIT although a wrapper looks pretty simple. I agree that it should be something simple. Sasha From sashak at voltaire.com Fri Jun 27 08:08:11 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 18:08:11 +0300 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080625160934.5025c073.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> Message-ID: <20080627150811.GC29456@sashak.voltaire.com> Hi Ira, On 16:09 Wed 25 Jun , Ira Weiny wrote: > > >From ec5c867213d58a2a297a2ca817b50266f55a35bd Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Tue, 24 Jun 2008 14:43:24 -0700 > Subject: [PATCH] libibmad: print warning containing destination port when _do_madrpc fails > > > Signed-off-by: Ira K. Weiny Applied. Thanks. > >From 31c73fc09fab80e599559e5de501708dc0f0899e Mon Sep 17 00:00:00 2001 > From: Ira K. Weiny > Date: Wed, 25 Jun 2008 15:08:13 -0700 > Subject: [PATCH] libibmad: update portid2str to use inet_ntop and drpath2str functions for gid > > and drpath printing respectively > > Signed-off-by: Ira K. Weiny Applied this with change Hal mentioned and removing unsuned 'i' variable. Thanks. Also do you think that intermidiate buffer could be removed here with something like below? Sasha diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c index 1c0e38a..c6b5135 100644 --- a/libibmad/src/portid.c +++ b/libibmad/src/portid.c @@ -66,23 +66,22 @@ char * portid2str(ib_portid_t *portid) { static char buf[1024] = "local"; - char drpath[512]; - char *s = buf; + int n = 0; if (portid->lid > 0) { - s += sprintf(s, "Lid %d", portid->lid); + n += sprintf(buf + n, "Lid %d", portid->lid); if (portid->grh_present) { char gid[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; if (inet_ntop(AF_INET6, portid->gid, gid, sizeof(gid))) - s += sprintf(s, " Gid %s", gid); + n += sprintf(buf + n, " Gid %s", gid); } if (portid->drpath.cnt) - s += sprintf(s, " "); + n += sprintf(buf + n, " "); else return buf; } - s += sprintf(s, "DR path %s", - drpath2str(&(portid->drpath), drpath, sizeof(drpath))); + n += sprintf(buf + n, "DR path "); + drpath2str(&(portid->drpath), buf + n, sizeof(buf) - n); return buf; } From hrosenstock at xsigo.com Fri Jun 27 08:10:11 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 08:10:11 -0700 Subject: [ofa-general] Is response time of GMP is more than SMP In-Reply-To: <4864F217.8080508@Sun.COM> References: <4860B59C.5010306@Sun.COM> <1214322429.13056.296.camel@hrosenstock-ws.xsigo.com> <4861E8A0.7000704@Sun.COM> <1214405310.13056.396.camel@hrosenstock-ws.xsigo.com> <48625B5B.7080207@Sun.COM> <1214409331.13056.444.camel@hrosenstock-ws.xsigo.com> <48633946.3070706@Sun.COM> <1214485794.13056.592.camel@hrosenstock-ws.xsigo.com> <4864CFA8.4010300@Sun.COM> <1214572501.13056.678.camel@hrosenstock-ws.xsigo.com> <4864F217.8080508@Sun.COM> Message-ID: <1214579411.13056.682.camel@hrosenstock-ws.xsigo.com> Hi Sumit, On Fri, 2008-06-27 at 19:28 +0530, Sumit Gaur - Sun Microsystem wrote: > hi Hal, > > Hal Rosenstock wrote: > > > >>>Can you elaborate on the multiple sends ? Are they outstanding > >>>concurrently ? Are they to the same destination or different ones ? Are > >>>they from a single or multiple threads ? > >> > >>No they are sending sequentially(mutex enabled) no concurrency but timeout for > >>umad_recv is 100ms. > > > > > > Can you try increasing that to see if there is some threshold where it > > works more reliably ? Does it work better at say 200 msec (as you said > > your rate was 4/sec) ? The default timeout used in the diags is 1 sec. > > yes, I tried increasing it upto 3000ms but condition not improved much (it > *reduced* timeout failures but *no reduction* in recv_fail and this piling up > number of request per second too at client side). Right; I'm not sure you can send the subsequent request until the former one either completes or times out. > Also I tried same code on two > separate subnet and similar problem come across. Is the second subnet any better/cleaner than the first ? > Could be a network issue but is > it a fact that GMP request takes much more time then SMP request (for ideal > subnet) ? Other than SMPs being VL15 with no flow control and GMPs being data VL (usually VL0) with flow control, there should be no difference for SMA v. PMA requests AFAIK. -- Hal > > Looks like there are some issues here to debug in your subnet. It might > > help to clear the counters and see what is actively going on to isolate > > these issues. This could factor into those other errors you are seeing. > > > > -- Hal > > > > From hrosenstock at xsigo.com Fri Jun 27 08:14:46 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 08:14:46 -0700 Subject: [ofa-general] [PATCH] opensm: Convert unicast LIDs in long messages from hex to decimal Message-ID: <1214579686.13056.685.camel@hrosenstock-ws.xsigo.com> opensm: Convert unicast LIDs in long messages from hex to decimal First pass; a few more changes related to this separately later Signed-off-by: Hal Rosenstock diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 7f0b00f..d3206e5 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -438,7 +438,7 @@ static void querylid_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) if (!p_port) goto invalid_lid; - fprintf(out, "Query results for LID %d\n", lid); + fprintf(out, "Query results for LID %u\n", lid); fprintf(out, " GUID : 0x%016" PRIx64 "\n" " Node Desc : %s\n" diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index 2d8d6e9..a7b2553 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -196,7 +196,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, - "Clearing abandoned LID range [0x%X,0x%X]\n", + "Clearing abandoned LID range [%u,%u]\n", min_lid_ho, max_lid_ho); p_port_lid_tbl = &sm->p_subn->port_lid_tbl; @@ -254,7 +254,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) if (osm_log_is_active(sm->p_log, OSM_LOG_INFO)) { OSM_LOG(sm->p_log, OSM_LOG_INFO, "Removed port with GUID:0x%016" PRIx64 - " LID range [0x%X,0x%X] of node:%s\n", + " LID range [%u, %u] of node:%s\n", cl_ntoh64(port_gid.unicast.interface_id), min_lid_ho, max_lid_ho, p_node ? p_node->print_desc : "UNKNOWN"); diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c index 60c6d25..9447cc8 100644 --- a/opensm/opensm/osm_dump.c +++ b/opensm/opensm/osm_dump.c @@ -330,7 +330,7 @@ static void dump_ucast_lfts(cl_map_item_t *p_map_item, FILE *file, void *cxt) uint16_t lid; uint8_t port; - fprintf(file, "Unicast lids [0x0-0x%x] of switch Lid %u guid 0x%016" + fprintf(file, "Unicast lids [0-%u] of switch Lid %u guid 0x%016" PRIx64 " (\'%s\'):\n", max_lid, cl_ntoh16(osm_node_get_base_lid(p_node, 0)), cl_ntoh64(osm_node_get_node_guid(p_node)), p_node->print_desc); diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c index 1b20638..f91e1e8 100644 --- a/opensm/opensm/osm_helper.c +++ b/opensm/opensm/osm_helper.c @@ -740,8 +740,8 @@ osm_dump_port_info(IN osm_log_t * const p_log, "\t\t\t\tport_guid...............0x%016" PRIx64 "\n" "\t\t\t\tm_key...................0x%016" PRIx64 "\n" "\t\t\t\tsubnet_prefix...........0x%016" PRIx64 "\n" - "\t\t\t\tbase_lid................0x%X\n" - "\t\t\t\tmaster_sm_base_lid......0x%X\n" + "\t\t\t\tbase_lid................%u\n" + "\t\t\t\tmaster_sm_base_lid......%u\n" "\t\t\t\tcapability_mask.........0x%X\n" "\t\t\t\tdiag_code...............0x%X\n" "\t\t\t\tm_key_lease_period......0x%X\n" @@ -824,14 +824,14 @@ osm_dump_portinfo_record(IN osm_log_t * const p_log, osm_log(p_log, log_level, "PortInfo Record dump:\n" "\t\t\t\tRID\n" - "\t\t\t\tEndPortLid..............0x%X\n" + "\t\t\t\tEndPortLid..............%u\n" "\t\t\t\tPortNum.................0x%X\n" "\t\t\t\tReserved................0x%X\n" "\t\t\t\tPortInfo dump:\n" "\t\t\t\tm_key...................0x%016" PRIx64 "\n" "\t\t\t\tsubnet_prefix...........0x%016" PRIx64 "\n" - "\t\t\t\tbase_lid................0x%X\n" - "\t\t\t\tmaster_sm_base_lid......0x%X\n" + "\t\t\t\tbase_lid................%u\n" + "\t\t\t\tmaster_sm_base_lid......%u\n" "\t\t\t\tcapability_mask.........0x%X\n" "\t\t\t\tdiag_code...............0x%X\n" "\t\t\t\tm_key_lease_period......0x%X\n" @@ -911,7 +911,7 @@ osm_dump_guidinfo_record(IN osm_log_t * const p_log, osm_log(p_log, log_level, "GUIDInfo Record dump:\n" "\t\t\t\tRID\n" - "\t\t\t\tLid.....................0x%X\n" + "\t\t\t\tLid.....................%u\n" "\t\t\t\tBlockNum................0x%X\n" "\t\t\t\tReserved................0x%X\n" "\t\t\t\tGUIDInfo dump:\n" @@ -996,7 +996,7 @@ osm_dump_node_record(IN osm_log_t * const p_log, osm_log(p_log, log_level, "Node Record dump:\n" "\t\t\t\tRID\n" - "\t\t\t\tLid.....................0x%X\n" + "\t\t\t\tLid.....................%u\n" "\t\t\t\tReserved................0x%X\n" "\t\t\t\tNodeInfo dump:\n" "\t\t\t\tbase_version............0x%X\n" @@ -1047,7 +1047,7 @@ osm_dump_path_record(IN osm_log_t * const p_log, "\t\t\t\tsgid....................0x%016" PRIx64 " : " "0x%016" PRIx64 "\n" "\t\t\t\tdlid....................0x%X\n" - "\t\t\t\tslid....................0x%X\n" + "\t\t\t\tslid....................%u\n" "\t\t\t\thop_flow_raw............0x%X\n" "\t\t\t\ttclass..................0x%X\n" "\t\t\t\tnum_path_revers.........0x%X\n" @@ -1322,8 +1322,8 @@ osm_dump_inform_info(IN osm_log_t * const p_log, "InformInfo dump:\n" "\t\t\t\tgid.....................0x%016" PRIx64 " : 0x%016" PRIx64 "\n" - "\t\t\t\tlid_range_begin.........0x%X\n" - "\t\t\t\tlid_range_end...........0x%X\n" + "\t\t\t\tlid_range_begin.........%u\n" + "\t\t\t\tlid_range_end...........%u\n" "\t\t\t\tis_generic..............0x%X\n" "\t\t\t\tsubscribe...............0x%X\n" "\t\t\t\ttrap_type...............0x%X\n" @@ -1346,8 +1346,8 @@ osm_dump_inform_info(IN osm_log_t * const p_log, "InformInfo dump:\n" "\t\t\t\tgid.....................0x%016" PRIx64 " : 0x%016" PRIx64 "\n" - "\t\t\t\tlid_range_begin.........0x%X\n" - "\t\t\t\tlid_range_end...........0x%X\n" + "\t\t\t\tlid_range_begin.........%u\n" + "\t\t\t\tlid_range_end...........%u\n" "\t\t\t\tis_generic..............0x%X\n" "\t\t\t\tsubscribe...............0x%X\n" "\t\t\t\ttrap_type...............0x%X\n" @@ -1395,8 +1395,8 @@ osm_dump_inform_info_record(IN osm_log_t * const p_log, "\t\t\t\tInformInfo dump:\n" "\t\t\t\tgid.....................0x%016" PRIx64 " : 0x%016" PRIx64 "\n" - "\t\t\t\tlid_range_begin.........0x%X\n" - "\t\t\t\tlid_range_end...........0x%X\n" + "\t\t\t\tlid_range_begin.........%u\n" + "\t\t\t\tlid_range_end...........%u\n" "\t\t\t\tis_generic..............0x%X\n" "\t\t\t\tsubscribe...............0x%X\n" "\t\t\t\ttrap_type...............0x%X\n" @@ -1433,8 +1433,8 @@ osm_dump_inform_info_record(IN osm_log_t * const p_log, "\t\t\t\tInformInfo dump:\n" "\t\t\t\tgid.....................0x%016" PRIx64 " : 0x%016" PRIx64 "\n" - "\t\t\t\tlid_range_begin.........0x%X\n" - "\t\t\t\tlid_range_end...........0x%X\n" + "\t\t\t\tlid_range_begin.........%u\n" + "\t\t\t\tlid_range_end...........%u\n" "\t\t\t\tis_generic..............0x%X\n" "\t\t\t\tsubscribe...............0x%X\n" "\t\t\t\ttrap_type...............0x%X\n" @@ -1475,10 +1475,10 @@ osm_dump_link_record(IN osm_log_t * const p_log, if (osm_log_is_active(p_log, log_level)) { osm_log(p_log, log_level, "Link Record dump:\n" - "\t\t\t\tfrom_lid................0x%X\n" + "\t\t\t\tfrom_lid................%u\n" "\t\t\t\tfrom_port_num...........0x%X\n" "\t\t\t\tto_port_num.............0x%X\n" - "\t\t\t\tto_lid..................0x%X\n" + "\t\t\t\tto_lid..................%u\n" "", cl_ntoh16(p_lr->from_lid), p_lr->from_port_num, @@ -1533,7 +1533,7 @@ osm_dump_switch_info_record(IN osm_log_t * const p_log, osm_log(p_log, log_level, "SwitchInfo Record dump:\n" "\t\t\t\tRID\n" - "\t\t\t\tlid.....................0x%X\n" + "\t\t\t\tlid.....................%u\n" "\t\t\t\tSwitchInfo dump:\n" "\t\t\t\tlin_cap.................0x%X\n" "\t\t\t\trand_cap................0x%X\n" @@ -1686,7 +1686,7 @@ osm_dump_sm_info_record(IN osm_log_t * const p_log, osm_log(p_log, OSM_LOG_DEBUG, "SMInfo Record dump:\n" "\t\t\t\tRID\n" - "\t\t\t\tLid.....................0x%X\n" + "\t\t\t\tLid.....................%u\n" "\t\t\t\tReserved................0x%X\n" "\t\t\t\tSMInfo dump:\n" "\t\t\t\tguid....................0x%016" PRIx64 "\n" @@ -1735,7 +1735,7 @@ osm_dump_notice(IN osm_log_t * const p_log, break; case 128: sprintf(buff, - "\t\t\t\tsw_lid...................0x%04X\n", + "\t\t\t\tsw_lid...................%u\n", cl_ntoh16(p_ntci->data_details.ntc_128. sw_lid)); break; @@ -1743,7 +1743,7 @@ osm_dump_notice(IN osm_log_t * const p_log, case 130: case 131: sprintf(buff, - "\t\t\t\tlid......................0x%04X\n" + "\t\t\t\tlid......................%u\n" "\t\t\t\tport_num.................%u\n", cl_ntoh16(p_ntci->data_details. ntc_129_131.lid), @@ -1752,7 +1752,7 @@ osm_dump_notice(IN osm_log_t * const p_log, break; case 144: sprintf(buff, - "\t\t\t\tlid......................0x%04x\n" + "\t\t\t\tlid......................%u\n" "\t\t\t\tnew_cap_mask.............0x%08x\n", cl_ntoh16(p_ntci->data_details.ntc_144. lid), @@ -1761,7 +1761,7 @@ osm_dump_notice(IN osm_log_t * const p_log, break; case 145: sprintf(buff, - "\t\t\t\tlid......................0x%04X\n" + "\t\t\t\tlid......................%u\n" "\t\t\t\tnew_sys_guid.............0x%016" PRIx64 "\n", cl_ntoh16(p_ntci->data_details.ntc_145. @@ -1853,8 +1853,8 @@ osm_dump_dr_smp(IN osm_log_t * const p_log, if (p_smp->mgmt_class == IB_MCLASS_SUBN_DIR) { sprintf(line, - "\t\t\t\tdr_slid.................0x%X\n" - "\t\t\t\tdr_dlid.................0x%X\n", + "\t\t\t\tdr_slid.................%u\n" + "\t\t\t\tdr_dlid.................%u\n", cl_ntoh16(p_smp->dr_slid), cl_ntoh16(p_smp->dr_dlid) ); diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index 1a4f18f..e6146f7 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -323,8 +323,8 @@ static ib_api_status_t __osm_send_report(IN osm_infr_t * p_infr_rec, /* the info /* it is better to use LIDs since the GIDs might not be there for SMI traps */ OSM_LOG(p_log, OSM_LOG_DEBUG, - "Forwarding Notice Event from LID:0x%X" - " to InformInfo LID: 0x%X TID:0x%X\n", + "Forwarding Notice Event from LID:%u" + " to InformInfo LID: %u TID:0x%X\n", cl_ntoh16(p_ntc->issuer_lid), cl_ntoh16(p_infr_rec->report_addr.dest_lid), trap_fwd_trans_id); @@ -422,7 +422,7 @@ __match_notice_to_inf_rec(IN cl_list_item_t * const p_list_item, || (cl_hton16(p_ntc->issuer_lid) > cl_hton16(p_ii->lid_range_end))) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Mismatch by LID Range. Needed: 0x%X <= 0x%X <= 0x%X\n", + "Mismatch by LID Range. Needed: %u <= %u <= %u\n", cl_hton16(p_ii->lid_range_begin), cl_hton16(p_ntc->issuer_lid), cl_hton16(p_ii->lid_range_end) @@ -519,7 +519,7 @@ __match_notice_to_inf_rec(IN cl_list_item_t * const p_list_item, cl_ntoh16(p_infr_rec->report_addr.dest_lid)); if (!p_dest_port) { OSM_LOG(p_log, OSM_LOG_INFO, - "Cannot find destination port with LID:0x%04x\n", + "Cannot find destination port with LID:%u\n", cl_ntoh16(p_infr_rec->report_addr.dest_lid)); goto Exit; } @@ -582,7 +582,7 @@ osm_report_notice(IN osm_log_t * const p_log, if (ib_notice_is_generic(p_ntc)) { OSM_LOG(p_log, OSM_LOG_INFO, "Reporting Generic Notice type:%u num:%u" - " from LID:0x%04X GID:0x%016" PRIx64 + " from LID:%u GID:0x%016" PRIx64 ",0x%016" PRIx64 "\n", ib_notice_get_type(p_ntc), cl_ntoh16(p_ntc->g_or_v.generic.trap_num), @@ -593,7 +593,7 @@ osm_report_notice(IN osm_log_t * const p_log, } else { OSM_LOG(p_log, OSM_LOG_INFO, "Reporting Vendor Notice type:%u vend:%u dev:%u" - " from LID:0x%04X GID:0x%016" PRIx64 + " from LID:%u GID:0x%016" PRIx64 ",0x%016" PRIx64 "\n", ib_notice_get_type(p_ntc), cl_ntoh32(ib_notice_get_vend_id(p_ntc)), diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index af8f7b3..d66ced3 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -173,7 +173,7 @@ static void __osm_lid_mgr_validate_db(IN osm_lid_mgr_t * p_mgr) || (p_item->guid == 0) || (max_lid > p_mgr->p_subn->max_ucast_lid_ho)) { OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0312: " - "Illegal LID range [0x%x:0x%x] for " + "Illegal LID range [%u:%u] for " "guid:0x%016" PRIx64 "\n", min_lid, max_lid, p_item->guid); lids_ok = FALSE; @@ -182,7 +182,7 @@ static void __osm_lid_mgr_validate_db(IN osm_lid_mgr_t * p_mgr) /* check that if the lids define a range that is valid for the current LMC mask */ OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 0313: " - "LID range [0x%x:0x%x] for guid:0x%016" + "LID range [%u:%u] for guid:0x%016" PRIx64 " is not aligned according to mask:0x%04x\n", min_lid, max_lid, p_item->guid, @@ -774,13 +774,13 @@ __osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, if (min_lid == cl_ntoh16(osm_port_get_base_lid(p_port))) { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "0x%016" PRIx64 - " matches its known lid:0x%04x\n", guid, + " matches its known lid:%u\n", guid, min_lid); goto Exit; } else { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "0x%016" PRIx64 " with lid:0x%04x " - "does not match its known lid:0x%04x\n", + "0x%016" PRIx64 " with lid:%u " + "does not match its known lid:%u\n", guid, cl_ntoh16(osm_port_get_base_lid(p_port)), min_lid); __osm_lid_mgr_cleanup_discovered_port_lid_range(p_mgr, @@ -1179,7 +1179,7 @@ __osm_lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * const p_mgr) */ __osm_lid_mgr_get_port_lid(p_mgr, p_port, &min_lid_ho, &max_lid_ho); OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Current base LID is 0x%X\n", min_lid_ho); + "Current base LID is %u\n", min_lid_ho); /* Update subnet object. */ @@ -1189,7 +1189,7 @@ __osm_lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * const p_mgr) if (osm_log_is_active(p_mgr->p_log, OSM_LOG_VERBOSE)) OSM_LOG(p_mgr->p_log, OSM_LOG_VERBOSE, "Assigning SM's port 0x%016" PRIx64 - "\n\t\t\t\tto LID range [0x%X,0x%X]\n", + "\n\t\t\t\tto LID range [%u,%u]\n", cl_ntoh64(osm_port_get_guid(p_port)), min_lid_ho, max_lid_ho); @@ -1301,7 +1301,7 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr) if required */ OSM_LOG(p_mgr->p_log, OSM_LOG_VERBOSE, "Assigned port 0x%016" PRIx64 - ", LID [0x%X,0x%X]\n", cl_ntoh64(port_guid), + ", LID [%u,%u]\n", cl_ntoh64(port_guid), min_lid_ho, max_lid_ho); /* the proc returns the fact it sent a set port info */ diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index c3167a8..7ab05c3 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -466,7 +466,7 @@ __osm_mcast_mgr_subdivide(osm_sm_t * sm, "Error routing MLID 0x%X through switch 0x%" PRIx64 "\n" "\t\t\t\tNo multicast paths from this switch for port " - "with LID 0x%X\n", mlid_ho, node_guid_ho, + "with LID %u\n", mlid_ho, node_guid_ho, cl_ntoh16(osm_port_get_base_lid (p_wobj->p_port))); @@ -481,7 +481,7 @@ __osm_mcast_mgr_subdivide(osm_sm_t * sm, "Error routing MLID 0x%X through switch 0x%" PRIx64 "\n" "\t\t\t\tNo multicast paths from this switch to port " - "with LID 0x%X\n", mlid_ho, node_guid_ho, + "with LID %u\n", mlid_ho, node_guid_ho, cl_ntoh16(osm_port_get_base_lid (p_wobj->p_port))); @@ -567,7 +567,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, if (depth >= 64) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, - "Maximal hops number is reached MLID 0x%x." + "Maximal hops number is reached for MLID 0x%x." " Break processing.", mlid_ho); __osm_mcast_mgr_purge_list(sm, p_list); goto Exit; diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index 315a065..9892335 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -1113,7 +1113,7 @@ static void osm_pc_rcv_process(void *context, void *data) ib_api_status_t status; OSM_LOG(pm->log, OSM_LOG_VERBOSE, - "Redirection to LID 0x%x " + "Redirection to LID %u " "GID 0x%016" PRIx64 " : 0x%016" PRIx64 " QP 0x%x received\n", cl_ntoh16(cpi->redir_lid), diff --git a/opensm/opensm/osm_port_info_rcv.c b/opensm/opensm/osm_port_info_rcv.c index c2ddfb0..2e3d94e 100644 --- a/opensm/opensm/osm_port_info_rcv.c +++ b/opensm/opensm/osm_port_info_rcv.c @@ -94,8 +94,8 @@ static void pi_rcv_check_and_fix_lid(osm_log_t *log, ib_port_info_t * const pi, { if (cl_ntoh16(pi->base_lid) > IB_LID_UCAST_END_HO) { OSM_LOG(log, OSM_LOG_ERROR, "ERR 0F04: " - "Got invalid base LID 0x%x from the network. " - "Corrected to 0x%x.\n", cl_ntoh16(pi->base_lid), + "Got invalid base LID %u from the network. " + "Corrected to %u.\n", cl_ntoh16(pi->base_lid), cl_ntoh16(p->port_info.base_lid)); pi->base_lid = p->port_info.base_lid; } diff --git a/opensm/opensm/osm_sa_guidinfo_record.c b/opensm/opensm/osm_sa_guidinfo_record.c index 7760f42..aaf4d6c 100644 --- a/opensm/opensm/osm_sa_guidinfo_record.c +++ b/opensm/opensm/osm_sa_guidinfo_record.c @@ -140,7 +140,7 @@ __osm_sa_gir_create_gir(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Looking for GUIDRecord with LID: 0x%X GUID:0x%016" + "Looking for GUIDRecord with LID: %u GUID:0x%016" PRIx64 "\n", cl_ntoh16(match_lid), cl_ntoh64(match_port_guid) ); @@ -203,7 +203,7 @@ __osm_sa_gir_create_gir(IN osm_sa_t * sa, */ if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Comparing LID: 0x%X <= 0x%X <= 0x%X\n", + "Comparing LID: %u <= %u <= %u\n", base_lid_ho, match_lid_ho, max_lid_ho); } diff --git a/opensm/opensm/osm_sa_informinfo.c b/opensm/opensm/osm_sa_informinfo.c index 798f5db..3bb2295 100644 --- a/opensm/opensm/osm_sa_informinfo.c +++ b/opensm/opensm/osm_sa_informinfo.c @@ -160,7 +160,7 @@ __validate_ports_access_rights(IN osm_sa_t * sa, } else { /* lid requested is out of range */ OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4302: " - "Given LID (0x%X) is out of range:0x%X\n", + "Given LID (%u) is out of range:%u\n", lid, cl_ptr_vector_get_size(p_tbl)); valid = FALSE; goto Exit; @@ -457,7 +457,7 @@ osm_infr_rcv_process_set_method(IN osm_sa_t * sa, cl_plock_release(sa->p_lock); OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4308 " - "Subscribe Request from unknown LID: 0x%04X\n", + "Subscribe Request from unknown LID: %u\n", cl_ntoh16(p_madw->mad_addr.dest_lid)); osm_sa_send_error(sa, p_madw, IB_SA_MAD_STATUS_REQ_INVALID); goto Exit; diff --git a/opensm/opensm/osm_sa_lft_record.c b/opensm/opensm/osm_sa_lft_record.c index f4b9b6e..fd7ccf6 100644 --- a/opensm/opensm/osm_sa_lft_record.c +++ b/opensm/opensm/osm_sa_lft_record.c @@ -182,7 +182,7 @@ __osm_lftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, /* compare the lids - if required */ if (comp_mask & IB_LFTR_COMPMASK_LID) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Comparing lid:0x%02X to port lid range: 0x%02X .. 0x%02X\n", + "Comparing lid:%u to port lid range: %u .. %u\n", cl_ntoh16(p_rcvd_rec->lid), min_lid_ho, max_lid_ho); /* ok we are ready for range check */ if (min_lid_ho > cl_ntoh16(p_rcvd_rec->lid) || diff --git a/opensm/opensm/osm_sa_link_record.c b/opensm/opensm/osm_sa_link_record.c index 0b761ba..f08eee1 100644 --- a/opensm/opensm/osm_sa_link_record.c +++ b/opensm/opensm/osm_sa_link_record.c @@ -404,7 +404,7 @@ __osm_lr_rcv_get_end_points(IN osm_sa_t * sa, Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No source port with LID = 0x%X\n", + "No source port with LID %u\n", cl_ntoh16(p_lr->from_lid)); sa_status = IB_SA_MAD_STATUS_NO_RECORDS; @@ -423,7 +423,7 @@ __osm_lr_rcv_get_end_points(IN osm_sa_t * sa, Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No dest port with LID = 0x%X\n", + "No dest port with LID %u\n", cl_ntoh16(p_lr->to_lid)); sa_status = IB_SA_MAD_STATUS_NO_RECORDS; diff --git a/opensm/opensm/osm_sa_mft_record.c b/opensm/opensm/osm_sa_mft_record.c index 57350a6..433f717 100644 --- a/opensm/opensm/osm_sa_mft_record.c +++ b/opensm/opensm/osm_sa_mft_record.c @@ -188,7 +188,7 @@ __osm_mftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, /* compare the lids - if required */ if (comp_mask & IB_MFTR_COMPMASK_LID) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Comparing lid:0x%02X to port lid range: 0x%02X .. 0x%02X\n", + "Comparing lid:%u to port lid range: %u .. %u\n", cl_ntoh16(p_rcvd_rec->lid), min_lid_ho, max_lid_ho); /* ok we are ready for range check */ if (min_lid_ho > cl_ntoh16(p_rcvd_rec->lid) || diff --git a/opensm/opensm/osm_sa_multipath_record.c b/opensm/opensm/osm_sa_multipath_record.c index bad1962..86a1d50 100644 --- a/opensm/opensm/osm_sa_multipath_record.c +++ b/opensm/opensm/osm_sa_multipath_record.c @@ -235,7 +235,7 @@ __osm_mpr_rcv_get_path_parms(IN osm_sa_t * sa, p_physp = osm_switch_get_route_by_lid(p_node->sw, dest_lid); if (p_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4514: " - "Can't find routing to LID 0x%X from switch for GUID 0x%016" + "Can't find routing to LID %u from switch for GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_NOT_FOUND; @@ -279,7 +279,7 @@ __osm_mpr_rcv_get_path_parms(IN osm_sa_t * sa, if (p_dest_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4515: " - "Can't find routing to LID 0x%X from switch for GUID 0x%016" + "Can't find routing to LID %u from switch for GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_NOT_FOUND; @@ -299,7 +299,7 @@ __osm_mpr_rcv_get_path_parms(IN osm_sa_t * sa, if (p_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4505: " - "Can't find remote phys port when routing to LID 0x%X from node GUID 0x%016" + "Can't find remote phys port when routing to LID %u from node GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_ERROR; @@ -346,7 +346,7 @@ __osm_mpr_rcv_get_path_parms(IN osm_sa_t * sa, p_physp = osm_switch_get_route_by_lid(p_node->sw, dest_lid); if (p_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4516: " - "Dead end on path to LID 0x%X from switch for GUID 0x%016" + "Dead end on path to LID %u from switch for GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_ERROR; @@ -822,7 +822,7 @@ __osm_mpr_rcv_get_lid_pair_path(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Src LID 0x%X, Dest LID 0x%X\n", + "Src LID %u, Dest LID %u\n", src_lid_ho, dest_lid_ho); p_pr_item = malloc(sizeof(*p_pr_item)); @@ -974,7 +974,7 @@ __osm_mpr_rcv_get_port_pair_paths(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Src LID [0x%X-0x%X], Dest LID [0x%X-0x%X]\n", + "Src LID [%u-%u], Dest LID [%u-%u]\n", src_lid_min_ho, src_lid_max_ho, dest_lid_min_ho, dest_lid_max_ho); @@ -1122,8 +1122,8 @@ __osm_mpr_rcv_get_apm_port_pair_paths(IN osm_sa_t * sa, dest_lid_ho += base_offs % dest_lids; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Src LIDs [0x%X-0x%X] hashed %d, " - "Dest LIDs [0x%X-0x%X] hashed %d\n", + "Src LIDs [%u-%u] hashed %u, " + "Dest LIDs [%u-%u] hashed %u\n", src_lid_min_ho, src_lid_max_ho, src_lid_ho, dest_lid_min_ho, dest_lid_max_ho, dest_lid_ho); @@ -1142,7 +1142,7 @@ __osm_mpr_rcv_get_apm_port_pair_paths(IN osm_sa_t * sa, if (p_pr_item) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Found matching path from Src LID 0x%X to Dest LID 0x%X with %d hops\n", + "Found matching path from Src LID %u to Dest LID %u with %d hops\n", src_lid_ho, dest_lid_ho, p_pr_item->hops); break; } diff --git a/opensm/opensm/osm_sa_node_record.c b/opensm/opensm/osm_sa_node_record.c index 91b748b..1ba418e 100644 --- a/opensm/opensm/osm_sa_node_record.c +++ b/opensm/opensm/osm_sa_node_record.c @@ -135,7 +135,7 @@ __osm_nr_rcv_create_nr(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Looking for NodeRecord with LID: 0x%X GUID:0x%016" + "Looking for NodeRecord with LID: %u GUID:0x%016" PRIx64 "\n", cl_ntoh16(match_lid), cl_ntoh64(match_port_guid) ); @@ -177,7 +177,7 @@ __osm_nr_rcv_create_nr(IN osm_sa_t * sa, */ if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Comparing LID: 0x%X <= 0x%X <= 0x%X\n", + "Comparing LID: %u <= %u <= %u\n", base_lid_ho, match_lid_ho, max_lid_ho); } diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 9af8a4f..f863735 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -242,7 +242,7 @@ __osm_pr_rcv_get_path_parms(IN osm_sa_t * sa, p_physp = osm_switch_get_route_by_lid(p_node->sw, dest_lid); if (p_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F02: " - "Cannot find routing to LID 0x%X from switch for GUID 0x%016" + "Cannot find routing to LID %u from switch for GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_NOT_FOUND; @@ -286,7 +286,7 @@ __osm_pr_rcv_get_path_parms(IN osm_sa_t * sa, if (p_dest_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F03: " - "Cannot find routing to LID 0x%X from switch for GUID 0x%016" + "Cannot find routing to LID %u from switch for GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_NOT_FOUND; @@ -306,7 +306,7 @@ __osm_pr_rcv_get_path_parms(IN osm_sa_t * sa, if (p_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F05: " - "Cannot find remote phys port when routing to LID 0x%X from node GUID 0x%016" + "Cannot find remote phys port when routing to LID %u from node GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_ERROR; @@ -352,7 +352,7 @@ __osm_pr_rcv_get_path_parms(IN osm_sa_t * sa, p_physp = osm_switch_get_route_by_lid(p_node->sw, dest_lid); if (p_physp == 0) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F07: " - "Dead end on path to LID 0x%X from switch for GUID 0x%016" + "Dead end on path to LID %u from switch for GUID 0x%016" PRIx64 "\n", dest_lid_ho, cl_ntoh64(osm_node_get_node_guid(p_node))); status = IB_ERROR; @@ -847,7 +847,7 @@ __osm_pr_rcv_get_lid_pair_path(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Src LID 0x%X, Dest LID 0x%X\n", + "Src LID %u, Dest LID %u\n", src_lid_ho, dest_lid_ho); p_pr_item = malloc(sizeof(*p_pr_item)); @@ -1026,7 +1026,7 @@ __osm_pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Src LIDs [0x%X-0x%X], Dest LIDs [0x%X-0x%X]\n", + "Src LIDs [%u-%u], Dest LIDs [%u-%u]\n", src_lid_min_ho, src_lid_max_ho, dest_lid_min_ho, dest_lid_max_ho); @@ -1221,7 +1221,7 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No source port with LID 0x%X\n", + "No source port with LID %u\n", cl_ntoh16(p_pr->slid)); sa_status = IB_SA_MAD_STATUS_NO_RECORDS; @@ -1332,7 +1332,7 @@ __osm_pr_rcv_get_end_points(IN osm_sa_t * sa, Return an error response to the client. */ OSM_LOG(sa->p_log, OSM_LOG_VERBOSE, - "No dest port with LID 0x%X\n", + "No dest port with LID %u\n", cl_ntoh16(p_pr->dlid)); sa_status = IB_SA_MAD_STATUS_NO_RECORDS; diff --git a/opensm/opensm/osm_sa_pkey_record.c b/opensm/opensm/osm_sa_pkey_record.c index d953369..db341f1 100644 --- a/opensm/opensm/osm_sa_pkey_record.c +++ b/opensm/opensm/osm_sa_pkey_record.c @@ -301,7 +301,7 @@ void osm_pkey_rec_rcv_process(IN void *ctx, IN void *data) context.p_req_physp = p_req_physp; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Got Query Lid:0x%04X(%02X), Block:0x%02X(%02X), Port:0x%02X(%02X)\n", + "Got Query Lid:%u(%02X), Block:0x%02X(%02X), Port:0x%02X(%02X)\n", cl_ntoh16(p_rcvd_rec->lid), (comp_mask & IB_PKEY_COMPMASK_LID) != 0, p_rcvd_rec->port_num, (comp_mask & IB_PKEY_COMPMASK_PORT) != 0, p_rcvd_rec->block_num, @@ -324,7 +324,7 @@ void osm_pkey_rec_rcv_process(IN void *ctx, IN void *data) if (status != IB_SUCCESS || p_port == NULL) { status = IB_NOT_FOUND; OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 460B: " - "No port found with LID 0x%x\n", + "No port found with LID %u\n", cl_ntoh16(p_rcvd_rec->lid)); } } diff --git a/opensm/opensm/osm_sa_portinfo_record.c b/opensm/opensm/osm_sa_portinfo_record.c index 3f3163d..267fe4a 100644 --- a/opensm/opensm/osm_sa_portinfo_record.c +++ b/opensm/opensm/osm_sa_portinfo_record.c @@ -149,7 +149,7 @@ __osm_sa_pir_create(IN osm_sa_t * sa, */ if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Comparing LID: 0x%X <= 0x%X <= 0x%X\n", + "Comparing LID: %u <= %u <= %u\n", base_lid_ho, match_lid_ho, max_lid_ho); if (match_lid_ho < base_lid_ho || match_lid_ho > max_lid_ho) @@ -543,7 +543,7 @@ void osm_pir_rcv_process(IN void *ctx, IN void *data) if ((status != IB_SUCCESS) || (p_port == NULL)) { status = IB_NOT_FOUND; OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 2109: " - "No port found with LID 0x%x\n", + "No port found with LID %u\n", cl_ntoh16(p_rcvd_rec->lid)); } } else if (comp_mask & IB_PIR_COMPMASK_BASELID) { @@ -554,7 +554,7 @@ void osm_pir_rcv_process(IN void *ctx, IN void *data) else { status = IB_NOT_FOUND; OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 2103: " - "Given LID (0x%X) is out of range:0x%X\n", + "Given LID (%u) is out of range:%u\n", cl_ntoh16(p_pi->base_lid), cl_ptr_vector_get_size(p_tbl)); } diff --git a/opensm/opensm/osm_sa_slvl_record.c b/opensm/opensm/osm_sa_slvl_record.c index f3b53dc..0b33307 100644 --- a/opensm/opensm/osm_sa_slvl_record.c +++ b/opensm/opensm/osm_sa_slvl_record.c @@ -272,7 +272,7 @@ void osm_slvl_rec_rcv_process(IN void *ctx, IN void *data) cl_plock_acquire(sa->p_lock); OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Got Query Lid:0x%04X(%02X), In-Port:0x%02X(%02X), Out-Port:0x%02X(%02X)\n", + "Got Query Lid:%u(%02X), In-Port:0x%02X(%02X), Out-Port:0x%02X(%02X)\n", cl_ntoh16(p_rcvd_rec->lid), (comp_mask & IB_SLVL_COMPMASK_LID) != 0, p_rcvd_rec->in_port_num, @@ -296,7 +296,7 @@ void osm_slvl_rec_rcv_process(IN void *ctx, IN void *data) if ((status != IB_SUCCESS) || (p_port == NULL)) { status = IB_NOT_FOUND; OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 2608: " - "No port found with LID 0x%x\n", + "No port found with LID %u\n", cl_ntoh16(p_rcvd_rec->lid)); } } diff --git a/opensm/opensm/osm_sa_sminfo_record.c b/opensm/opensm/osm_sa_sminfo_record.c index 6eddc59..f23c3e0 100644 --- a/opensm/opensm/osm_sa_sminfo_record.c +++ b/opensm/opensm/osm_sa_sminfo_record.c @@ -252,7 +252,7 @@ void osm_smir_rcv_process(IN void *ctx, IN void *data) if ((status != IB_SUCCESS) || (p_port == NULL)) { status = IB_NOT_FOUND; OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 2806: " - "No port found with LID 0x%x\n", + "No port found with LID %u\n", cl_ntoh16(p_rcvd_rec->lid)); } } diff --git a/opensm/opensm/osm_sa_sw_info_record.c b/opensm/opensm/osm_sa_sw_info_record.c index a05c814..86471e3 100644 --- a/opensm/opensm/osm_sa_sw_info_record.c +++ b/opensm/opensm/osm_sa_sw_info_record.c @@ -143,7 +143,7 @@ __osm_sir_rcv_create_sir(IN osm_sa_t * sa, if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Looking for SwitchInfoRecord with LID: 0x%X\n", + "Looking for SwitchInfoRecord with LID: %u\n", cl_ntoh16(match_lid) ); } @@ -184,7 +184,7 @@ __osm_sir_rcv_create_sir(IN osm_sa_t * sa, */ if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Comparing LID: 0x%X <= 0x%X <= 0x%X\n", + "Comparing LID: %u <= %u <= %u\n", min_lid_ho, match_lid_ho, max_lid_ho); } diff --git a/opensm/opensm/osm_sa_vlarb_record.c b/opensm/opensm/osm_sa_vlarb_record.c index 6b97a2e..eb4dd81 100644 --- a/opensm/opensm/osm_sa_vlarb_record.c +++ b/opensm/opensm/osm_sa_vlarb_record.c @@ -287,7 +287,7 @@ void osm_vlarb_rec_rcv_process(IN void *ctx, IN void *data) context.p_req_physp = p_req_physp; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, - "Got Query Lid:0x%04X(%02X), Port:0x%02X(%02X), Block:0x%02X(%02X)\n", + "Got Query Lid:%u(%02X), Port:0x%02X(%02X), Block:0x%02X(%02X)\n", cl_ntoh16(p_rcvd_rec->lid), (comp_mask & IB_VLA_COMPMASK_LID) != 0, p_rcvd_rec->port_num, (comp_mask & IB_VLA_COMPMASK_OUT_PORT) != 0, @@ -312,7 +312,7 @@ void osm_vlarb_rec_rcv_process(IN void *ctx, IN void *data) if ((status != IB_SUCCESS) || (p_port == NULL)) { status = IB_NOT_FOUND; OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 2A09: " - "No port found with LID 0x%x\n", + "No port found with LID %u\n", cl_ntoh16(p_rcvd_rec->lid)); } } diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index 780d8db..9a8409b 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -833,7 +833,7 @@ static void __osm_state_mgr_report_new_ports(IN osm_sm_t * sm) osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); OSM_LOG(sm->p_log, OSM_LOG_INFO, "Discovered new port with GUID:0x%016" PRIx64 - " LID range [0x%X,0x%X] of node:%s\n", + " LID range [%u,%u] of node:%s\n", cl_ntoh64(port_gid.unicast.interface_id), min_lid_ho, max_lid_ho, p_port->p_node ? p_port->p_node-> diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 73b476f..976e12b 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -243,7 +243,7 @@ osm_get_gid_by_mad_addr(IN osm_log_t * p_log, cl_ptr_vector_get(p_tbl, cl_ntoh16(p_mad_addr->dest_lid)); if (p_port == NULL) { OSM_LOG(p_log, OSM_LOG_DEBUG, - "Did not find any port with LID: 0x%X\n", + "Did not find any port with LID: %u\n", cl_ntoh16(p_mad_addr->dest_lid)); return (IB_INVALID_PARAMETER); } @@ -252,7 +252,7 @@ osm_get_gid_by_mad_addr(IN osm_log_t * p_log, } else { /* The dest_lid is not in the subnet table - this is an error */ OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 7501: " - "LID is out of range: 0x%X\n", + "LID is out of range: %u\n", cl_ntoh16(p_mad_addr->dest_lid)); return (IB_INVALID_PARAMETER); } @@ -283,7 +283,7 @@ osm_physp_t *osm_get_physp_by_mad_addr(IN osm_log_t * p_log, if (p_port == NULL) { /* The port is not in the port_lid table - this is an error */ OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 7502: " - "Cannot locate port object by lid: 0x%X\n", + "Cannot locate port object by lid: %u\n", cl_ntoh16(p_mad_addr->dest_lid)); goto Exit; @@ -292,7 +292,7 @@ osm_physp_t *osm_get_physp_by_mad_addr(IN osm_log_t * p_log, } else { /* The dest_lid is not in the subnet table - this is an error */ OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 7503: " - "Lid is out of range: 0x%X\n", + "Lid is out of range: %u\n", cl_ntoh16(p_mad_addr->dest_lid)); } @@ -322,7 +322,7 @@ osm_port_t *osm_get_port_by_mad_addr(IN osm_log_t * p_log, } else { /* The dest_lid is not in the subnet table - this is an error */ OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 7504: " - "Lid is out of range: 0x%X\n", + "Lid is out of range: %u\n", cl_ntoh16(p_mad_addr->dest_lid)); } diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index 1ff8e6e..ae7ab05 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -310,7 +310,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, goto Exit; } OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Received SLID=0 Trap. Using local LID:0x%04X instead\n", + "Received SLID=0 Trap. Using local LID:%u instead\n", cl_ntoh16(sm->p_subn->sm_base_lid) ); tmp_madw.mad_addr.addr_type.smi.source_lid = @@ -328,7 +328,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, CL_HTON16(131))) OSM_LOG(sm->p_log, OSM_LOG_ERROR, "Received Generic Notice type:0x%02X num:%u Producer:%u (%s) " - "from LID:0x%04X Port %d TID:0x%016" + "from LID:%u Port %d TID:0x%016" PRIx64 "\n", ib_notice_get_type(p_ntci), cl_ntoh16(p_ntci->g_or_v.generic. trap_num), @@ -343,7 +343,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, else OSM_LOG(sm->p_log, OSM_LOG_ERROR, "Received Generic Notice type:0x%02X num:%u Producer:%u (%s) " - "from LID:0x%04X TID:0x%016" PRIx64 + "from LID:%u TID:0x%016" PRIx64 "\n", ib_notice_get_type(p_ntci), cl_ntoh16(p_ntci->g_or_v.generic. trap_num), @@ -357,7 +357,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, } else OSM_LOG(sm->p_log, OSM_LOG_ERROR, "Received Vendor Notice type:0x%02X vend:0x%06X dev:%u " - "from LID:0x%04X TID:0x%016" PRIx64 "\n", + "from LID:%u TID:0x%016" PRIx64 "\n", ib_notice_get_type(p_ntci), cl_ntoh32(ib_notice_get_vend_id(p_ntci)), cl_ntoh16(p_ntci->g_or_v.vend.dev_id), @@ -442,7 +442,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, if (!p_physp) OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3805: " - "Failed to find physical port by lid:0x%02X num:%u\n", + "Failed to find physical port by lid:%u num:%u\n", cl_ntoh16(p_ntci->data_details. ntc_129_131.lid), p_ntci->data_details. @@ -465,7 +465,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3810: " - "Disabling physical port lid:0x%02X num:%u\n", + "Disabling physical port lid:%u num:%u\n", cl_ntoh16(p_ntci-> data_details. ntc_129_131. @@ -524,7 +524,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, } OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, - "Marking unhealthy physical port by lid:0x%02X num:%u\n", + "Marking unhealthy physical port by lid:%u num:%u\n", cl_ntoh16(p_ntci->data_details. ntc_129_131.lid), p_ntci->data_details. @@ -640,7 +640,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, cl_ntoh16(source_lid)) { /* the source lid is out of range */ OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, - "source lid is out of range:0x%X\n", + "source lid is out of range:%u\n", cl_ntoh16(source_lid)); goto Exit; @@ -649,7 +649,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, if (p_port == 0) { /* We have the lid - but no corresponding port */ OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, - "Cannot find port corresponding to lid:0x%X\n", + "Cannot find port corresponding to lid:%u\n", cl_ntoh16(source_lid)); goto Exit; diff --git a/opensm/opensm/osm_ucast_file.c b/opensm/opensm/osm_ucast_file.c index 43bcbc5..dee0e01 100644 --- a/opensm/opensm/osm_ucast_file.c +++ b/opensm/opensm/osm_ucast_file.c @@ -86,7 +86,7 @@ static void add_path(osm_opensm_t * p_osm, if (old_port != OSM_NO_PATH && old_port != port_num) { OSM_LOG(&p_osm->log, OSM_LOG_VERBOSE, "LID collision is detected on switch " - "0x016%" PRIx64 ", will overwrite LID 0x%x entry\n", + "0x016%" PRIx64 ", will overwrite LID %u entry\n", cl_ntoh64(osm_node_get_node_guid(p_sw->p_node)), new_lid); } diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 9604381..7502e04 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -55,7 +55,7 @@ /* * FatTree rank is bounded between 2 and 8: - * - Tree of rank 1 has only trivial routing pathes, + * - Tree of rank 1 has only trivial routing paths, * so no need to use FatTree routing. * - Why maximum rank is 8: * Each node (switch) is assigned a unique tuple. @@ -1222,7 +1222,7 @@ static void __osm_ftree_fabric_dump_general_info(IN ftree_fabric_t * p_ftree) if (p_sw->rank == 0) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, " GUID: 0x%016" PRIx64 - ", LID: 0x%04x, Index %s\n", + ", LID: %u, Index %s\n", __osm_ftree_sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), __osm_ftree_tuple_to_str(p_sw->tuple)); @@ -1233,7 +1233,7 @@ static void __osm_ftree_fabric_dump_general_info(IN ftree_fabric_t * p_ftree) for (i = 0; i < p_ftree->leaf_switches_num; i++) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, " GUID: 0x%016" PRIx64 - ", LID: 0x%04x, Index %s\n", + ", LID: %u, Index %s\n", __osm_ftree_sw_get_guid_ho(p_ftree-> leaf_switches[i]), cl_ntoh16(p_ftree->leaf_switches[i]->base_lid), @@ -1492,7 +1492,7 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) "Indexing starting point:\n" " - Switch rank : %u\n" " - Switch index : %s\n" - " - Node LID : 0x%04x\n" + " - Node LID : %u\n" " - Node GUID : 0x%016" PRIx64 "\n", p_sw->rank, __osm_ftree_tuple_to_str(p_sw->tuple), cl_ntoh16(p_sw->base_lid), __osm_ftree_sw_get_guid_ho(p_sw)); @@ -1778,9 +1778,9 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB09: Different number of upward port groups on switches:\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u groups\n" + ", LID %u, Index %s - %u groups\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u groups\n", + ", LID %u, Index %s - %u groups\n", __osm_ftree_sw_get_guid_ho (reference_sw_arr[p_sw->rank]), cl_ntoh16(reference_sw_arr[p_sw->rank]-> @@ -1805,9 +1805,9 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB0A: Different number of downward port groups on switches:\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u port groups\n" + ", LID %u, Index %s - %u port groups\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u port groups\n", + ", LID %u, Index %s - %u port groups\n", __osm_ftree_sw_get_guid_ho (reference_sw_arr[p_sw->rank]), cl_ntoh16(reference_sw_arr[p_sw->rank]-> @@ -1840,10 +1840,10 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * "ERR AB0B: Different number of ports in an upward port group on switches:\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u ports\n" + ", LID %u, Index %s - %u ports\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u ports\n", + ", LID %u, Index %s - %u ports\n", __osm_ftree_sw_get_guid_ho (reference_sw_arr [p_sw->rank]), @@ -1887,10 +1887,10 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * "ERR AB0C: Different number of ports in an downward port group on switches:\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u ports\n" + ", LID %u, Index %s - %u ports\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, Index %s - %u ports\n", + ", LID %u, Index %s - %u ports\n", __osm_ftree_sw_get_guid_ho (reference_sw_arr [p_sw->rank]), @@ -2045,7 +2045,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, on our way UP, and now we see it again on our way DOWN */ OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Loop of lenght %d in the fabric:\n " - "Switch %s (LID 0x%04x) closes loop through switch %s (LID 0x%04x)\n", + "Switch %s (LID %u) closes loop through switch %s (LID %u)\n", (p_remote_sw->rank - highest_rank_in_route) * 2, __osm_ftree_tuple_to_str(p_remote_sw->tuple), cl_ntoh16(p_group->base_lid), @@ -2097,7 +2097,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, p_min_port-> remote_port_num); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to CA LID 0x%04x through port %u\n", + "Switch %s: set path to CA LID %u through port %u\n", __osm_ftree_tuple_to_str(p_remote_sw->tuple), cl_ntoh16(target_lid), p_min_port->remote_port_num); @@ -2260,7 +2260,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, if (is_main_path) { if (p_sw->is_leaf) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - " - Routing MAIN path for %s CA LID 0x%04x: %s --> %s\n", + " - Routing MAIN path for %s CA LID %u: %s --> %s\n", (is_real_lid) ? "real" : "DUMMY", cl_ntoh16(target_lid), __osm_ftree_tuple_to_str(p_sw->tuple), @@ -2279,7 +2279,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, p_min_port-> remote_port_num); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to CA LID 0x%04x through port %u\n", + "Switch %s: set path to CA LID %u through port %u\n", __osm_ftree_tuple_to_str(p_remote_sw->tuple), cl_ntoh16(target_lid), p_min_port->remote_port_num); @@ -2358,7 +2358,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, if (p_sw->is_leaf) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - " - Routing SECONDARY path for LID 0x%04x: %s --> %s\n", + " - Routing SECONDARY path for LID %u: %s --> %s\n", cl_ntoh16(target_lid), __osm_ftree_tuple_to_str(p_sw->tuple), __osm_ftree_tuple_to_str(p_remote_sw->tuple)); @@ -2368,7 +2368,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, switch-to-switch or switch-to-CA paths. We can safely assume that switch will initiate very few traffic, so there's no point waisting runtime on - trying to ballance these routes - always pick port 0. */ + trying to balance these routes - always pick port 0. */ cl_ptr_vector_at(&p_group->ports, 0, (void **)&p_port); __osm_ftree_sw_set_fwd_table_block(p_remote_sw, @@ -2468,7 +2468,7 @@ static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) cl_ntoh16(hca_lid), p_port->port_num); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to CN LID 0x%04x through port %u\n", + "Switch %s: set path to CN LID %u through port %u\n", __osm_ftree_tuple_to_str(p_sw->tuple), cl_ntoh16(hca_lid), p_port->port_num); @@ -2479,7 +2479,7 @@ static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) /* Assign downgoing ports by stepping up. Since we're routing here only CNs, we're routing it as REAL - LID and updating fat-tree ballancing counters. */ + LID and updating fat-tree balancing counters. */ __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ hca_lid, /* LID that we're routing to */ @@ -2579,7 +2579,7 @@ static void __osm_ftree_fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) port_num_on_switch); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s: set path to non-CN HCA LID 0x%04x through port %u\n", + "Switch %s: set path to non-CN HCA LID %u through port %u\n", __osm_ftree_tuple_to_str(p_sw->tuple), cl_ntoh16(hca_lid), port_num_on_switch); @@ -2637,7 +2637,7 @@ static void __osm_ftree_fabric_route_to_switches(IN ftree_fabric_t * p_ftree) 0); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, - "Switch %s (LID 0x%04x): routing switch-to-switch pathes\n", + "Switch %s (LID %u): routing switch-to-switch paths\n", __osm_ftree_tuple_to_str(p_sw->tuple), cl_ntoh16(p_sw->base_lid)); @@ -2845,7 +2845,7 @@ __osm_ftree_rank_leaf_switches(IN ftree_fabric_t * p_ftree, PRIx64 "\n" " - Switch guid: 0x%016" PRIx64 "\n" - " - Switch LID : 0x%04x\n", + " - Switch LID : %u\n", __osm_ftree_hca_get_guid_ho(p_hca), __osm_ftree_sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid)); @@ -3020,7 +3020,7 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, if (p_node == p_remote_node) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Ignoring loopback on switch GUID 0x%016" PRIx64 - ", LID 0x%04x, rank %u\n", + ", LID %u, rank %u\n", __osm_ftree_sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), p_sw->rank); @@ -3065,9 +3065,9 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, "ERR AB16: " "Illegal link between switches with ranks %u and %u:\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, rank %u\n" + ", LID %u, rank %u\n" " GUID 0x%016" PRIx64 - ", LID 0x%04x, rank %u\n", p_sw->rank, + ", LID %u, rank %u\n", p_sw->rank, p_remote_sw->rank, __osm_ftree_sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), p_sw->rank, @@ -3647,7 +3647,7 @@ static int __osm_ftree_construct_fabric(IN void *context) } OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, - "Max LID in switch LFTs (in host order): 0x%04x\n", + "Max LID in switch LFTs: %u\n", p_ftree->lft_max_lid_ho); Exit: @@ -3695,7 +3695,7 @@ static int __osm_ftree_do_routing(IN void *context) __osm_ftree_fabric_route_to_non_cns(p_ftree); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, - "Filling switch forwarding tables for switch-to-switch pathes\n"); + "Filling switch forwarding tables for switch-to-switch paths\n"); __osm_ftree_fabric_route_to_switches(p_ftree); /* for each switch, set its fwd table */ diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index 39ce2bd..9f54486 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -246,7 +246,7 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Processing port 0x%" PRIx64 " (\'%s\' port %u), LID %u [0x%X,0x%X]\n", + "Processing port 0x%" PRIx64 " (\'%s\' port %u), LID %u [%u,%u]\n", cl_ntoh64(osm_port_get_guid(p_port)), p_port->p_node->print_desc, p_port->p_physp->port_num, lid_ho, min_lid_ho, max_lid_ho); @@ -276,17 +276,17 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, switches so we do not report that as an error in that case */ if (!p_routing_eng->build_lid_matrices) { OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A08: " - "No path to get to LID 0x%X from switch 0x%" + "No path to get to LID %u from switch 0x%" PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); /* trigger a new sweep - try again ... */ p_mgr->p_subn->subnet_initialization_error = TRUE; } else OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "No path to get to LID 0x%X from switch 0x%" + "No path to get to LID %u from switch 0x%" PRIx64 "\n", lid_ho, cl_ntoh64(node_guid)); } else { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Routing LID 0x%X to port 0x%X" + "Routing LID %u to port 0x%X" " for switch 0x%" PRIx64 "\n", lid_ho, port, cl_ntoh64(node_guid)); @@ -384,7 +384,7 @@ osm_ucast_mgr_set_fwd_table(IN osm_ucast_mgr_t * const p_mgr, if (set_swinfo_require) { if (osm_log_is_active(p_mgr->p_log, OSM_LOG_DEBUG)) { OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Setting switch FT top to LID 0x%X\n", + "Setting switch FT top to LID %u\n", p_sw->max_lid_ho); } From sashak at voltaire.com Fri Jun 27 08:14:32 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 18:14:32 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/include/opensm/osm_port.h: Commentary change In-Reply-To: <1214489305.13056.602.camel@hrosenstock-ws.xsigo.com> References: <1214489305.13056.602.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627151432.GD29456@sashak.voltaire.com> On 07:08 Thu 26 Jun , Hal Rosenstock wrote: > opensm/include/opensm/osm_port.h: Commentary change > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hrosenstock at xsigo.com Fri Jun 27 08:16:49 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 08:16:49 -0700 Subject: [ofa-general] [RFC] OpenSM: port number format in log messages Message-ID: <1214579809.13056.688.camel@hrosenstock-ws.xsigo.com> Currently, port numbers are handled inconsistently in the OpenSM logging. In some places they are decimal and others hex. It seems the simplest thing here is decimal. Comments ? -- Hal From alypin at sim2-c3x.com Fri Jun 27 08:25:21 2008 From: alypin at sim2-c3x.com (Joanne Chase) Date: Fri, 27 Jun 2008 15:25:21 -0000 Subject: [ofa-general] "Premiere Pro CS 3" Message-ID: <000401c8d869$47eded80$0100007f@srqyl> " Adobe CS3 Master Collection for PC or MAC includes: " InDesign CS3 " Photoshop CS3 " Illustrator CS3 " Acrobat 8 Professional " Flash CS3 Professional " Dreamweaver CS3 " Fireworks CS3 " Contribute CS3 " After Effects CS3 Professional " Premiere Pro CS3 " Encore DVD CS3 " Soundbooth CS3 " esoftcheap . com in your Internet Exp1orer " System Requirements " For PC: " Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core " Duo (or compatible) processor; SSE2-enabled processor required for AMD systems " Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) " 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components " 38GB of available hard-disk space (additional free space required during installation) " Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred " Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) " 1,280x1,024 monitor resolution with 32-bit color adapter " DVD-ROM drive " For MAC: " PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) " Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server " 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components " 36GB of available hard-disk space (additional free space required during installation) " Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred " Core Audio compatible sound card " 1,280x1,024 monitor resolution with 32-bit color adapter " DVD-ROM drive" DVD+-R burner required for DVD creation Polls open in Zimbabwe on Friday for the disputed one-man presidential run-off. Opposition candidate Morgan Tsvangirai pulled out of the race citing concerns for his supporters' lives. That leaves President Robert Mugabe as the only candidate on the ballot. International leaders have condemned the election. Over four weeks, more than half a million ballots were cast at Foreign Policy's website. Do you know who's number one? From weiny2 at llnl.gov Fri Jun 27 09:12:24 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 09:12:24 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080627150811.GC29456@sashak.voltaire.com> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <20080627150811.GC29456@sashak.voltaire.com> Message-ID: <20080627091224.2b18659b.weiny2@llnl.gov> On Fri, 27 Jun 2008 18:08:11 +0300 Sasha Khapyorsky wrote: > Hi Ira, > > On 16:09 Wed 25 Jun , Ira Weiny wrote: > > > > Subject: [PATCH] libibmad: update portid2str to use inet_ntop and drpath2str functions for gid > > > > and drpath printing respectively > > > > Signed-off-by: Ira K. Weiny > > Applied this with change Hal mentioned and removing unsuned 'i' variable. > Thanks. > > Also do you think that intermidiate buffer could be removed here with > something like below? Yea that looks like it would work. Looks more efficient as well... ;-) Ira > > Sasha > > > diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c > index 1c0e38a..c6b5135 100644 > --- a/libibmad/src/portid.c > +++ b/libibmad/src/portid.c > @@ -66,23 +66,22 @@ char * > portid2str(ib_portid_t *portid) > { > static char buf[1024] = "local"; > - char drpath[512]; > - char *s = buf; > + int n = 0; > > if (portid->lid > 0) { > - s += sprintf(s, "Lid %d", portid->lid); > + n += sprintf(buf + n, "Lid %d", portid->lid); > if (portid->grh_present) { > char gid[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"]; > if (inet_ntop(AF_INET6, portid->gid, gid, sizeof(gid))) > - s += sprintf(s, " Gid %s", gid); > + n += sprintf(buf + n, " Gid %s", gid); > } > if (portid->drpath.cnt) > - s += sprintf(s, " "); > + n += sprintf(buf + n, " "); > else > return buf; > } > - s += sprintf(s, "DR path %s", > - drpath2str(&(portid->drpath), drpath, sizeof(drpath))); > + n += sprintf(buf + n, "DR path "); > + drpath2str(&(portid->drpath), buf + n, sizeof(buf) - n); > > return buf; > } From sashak at voltaire.com Fri Jun 27 09:19:51 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:19:51 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] osm_port_profile: Change some variable names for consistency In-Reply-To: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> References: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627161951.GE29456@sashak.voltaire.com> Hi Hal, On 07:07 Thu 26 Jun , Hal Rosenstock wrote: > osm_port_profile: Change some variable names for consistency Interesting. Looking at the code I see that this file should list switch's external ports where paths counting will not be done. If so should this be clarified in manual page? Also for me the option name '--ignore-guids' looks a bit confusing, I would think that it is about and ports. Thoughts? Sasha From sashak at voltaire.com Fri Jun 27 09:22:57 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:22:57 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] osm_port_profile: Change some variable names for consistency In-Reply-To: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> References: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627162257.GF29456@sashak.voltaire.com> On 07:07 Thu 26 Jun , Hal Rosenstock wrote: > osm_port_profile: Change some variable names for consistency > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 27 09:23:45 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:23:45 +0300 Subject: [ofa-general] Re: [PATCH] opensm/man/opensm.8.in: Clarify ignore-guids file format In-Reply-To: <1214503214.13056.625.camel@hrosenstock-ws.xsigo.com> References: <1214503214.13056.625.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627162345.GG29456@sashak.voltaire.com> On 11:00 Thu 26 Jun , Hal Rosenstock wrote: > opensm/man/opensm.8.in: Clarify ignore-guids file format > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Jun 27 09:25:32 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:25:32 +0300 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080627091224.2b18659b.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <20080627150811.GC29456@sashak.voltaire.com> <20080627091224.2b18659b.weiny2@llnl.gov> Message-ID: <20080627162532.GH29456@sashak.voltaire.com> On 09:12 Fri 27 Jun , Ira Weiny wrote: > > Yea that looks like it would work. Looks more efficient as well... ;-) Ok. I will add this too. Sasha From podophryidae at blueskycoffeehouse.com Fri Jun 27 09:34:32 2008 From: podophryidae at blueskycoffeehouse.com (Spock Palmer) Date: Sat, 28 Jun 2008 01:34:32 +0900 Subject: [ofa-general] ~Contrlbute CS 3~ Message-ID: <000801c8d873$5c36e800$0100007f@sumnws> ~ Adobe CS3 Master Collection for PC or MAC includes: ~ InDesign CS3 ~ Photoshop CS3 ~ Illustrator CS3 ~ Acrobat 8 Professional ~ Flash CS3 Professional ~ Dreamweaver CS3 ~ Fireworks CS3 ~ Contribute CS3 ~ After Effects CS3 Professional ~ Premiere Pro CS3 ~ Encore DVD CS3 ~ Soundbooth CS3 ~ esoftnew . com in Internet Explorer ~ System Requirements ~ For PC: ~ Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core ~ Duo (or compatible) processor; SSE2-enabled processor required for AMD systems ~ Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) ~ 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ~ 38GB of available hard-disk space (additional free space required during installation) ~ Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ~ Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) ~ 1,280x1,024 monitor resolution with 32-bit color adapter ~ DVD-ROM drive ~ For MAC: ~ PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) ~ Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server ~ 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ~ 36GB of available hard-disk space (additional free space required during installation) ~ Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ~ Core Audio compatible sound card ~ 1,280x1,024 monitor resolution with 32-bit color adapter ~ DVD-ROM drive~ DVD+-R burner required for DVD creation After Long Exile, Afghan Returns to Aid U.S. Marines The safety checklist has 22 items spread out over the before, during and after stages of an operation. During a pilot program, hospitals initially said they were adhering to proven standards of care only a third of the time. That rate later improved to two-thirds of the time. From sashak at voltaire.com Fri Jun 27 09:35:04 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:35:04 +0300 Subject: [ofa-general] [RFC] OpenSM: port number format in log messages In-Reply-To: <1214579809.13056.688.camel@hrosenstock-ws.xsigo.com> References: <1214579809.13056.688.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627163504.GI29456@sashak.voltaire.com> On 08:16 Fri 27 Jun , Hal Rosenstock wrote: > Currently, port numbers are handled inconsistently in the OpenSM > logging. In some places they are decimal and others hex. It seems the > simplest thing here is decimal. Comments ? I agree - even unsigned decimal. Sasha From hrosenstock at xsigo.com Fri Jun 27 09:41:18 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 09:41:18 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] osm_port_profile: Change some variable names for consistency In-Reply-To: <20080627161951.GE29456@sashak.voltaire.com> References: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> <20080627161951.GE29456@sashak.voltaire.com> Message-ID: <1214584878.13056.692.camel@hrosenstock-ws.xsigo.com> Hi Sasha, On Fri, 2008-06-27 at 19:19 +0300, Sasha Khapyorsky wrote: > Hi Hal, > > On 07:07 Thu 26 Jun , Hal Rosenstock wrote: > > osm_port_profile: Change some variable names for consistency > > Interesting. Looking at the code I see that this file should list > switch's external ports where paths counting will not be done. If so > should this be clarified in manual page? I already submitted a patch for this. > Also for me the option name > '--ignore-guids' looks a bit confusing, I would think that it is about > and ports. Thoughts? Up to you. I agree the name could be improved. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hrosenstock at xsigo.com Fri Jun 27 09:44:13 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 09:44:13 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080627091224.2b18659b.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <20080627150811.GC29456@sashak.voltaire.com> <20080627091224.2b18659b.weiny2@llnl.gov> Message-ID: <1214585053.13056.695.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-27 at 09:12 -0700, Ira Weiny wrote: > > Also do you think that intermidiate buffer could be removed here > with > > something like below? > > Yea that looks like it would work. Is this code also tested ? Just wondering as comments like that leave that question open. -- Hal > Looks more efficient as well... ;-) From weiny2 at llnl.gov Fri Jun 27 09:49:51 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 09:49:51 -0700 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <1214585053.13056.695.camel@hrosenstock-ws.xsigo.com> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <20080627150811.GC29456@sashak.voltaire.com> <20080627091224.2b18659b.weiny2@llnl.gov> <1214585053.13056.695.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627094951.1ee777db.weiny2@llnl.gov> On Fri, 27 Jun 2008 09:44:13 -0700 Hal Rosenstock wrote: > On Fri, 2008-06-27 at 09:12 -0700, Ira Weiny wrote: > > > Also do you think that intermidiate buffer could be removed here > > with > > > something like below? > > > > Yea that looks like it would work. > > Is this code also tested ? Just wondering as comments like that leave > that question open. > No I did not test it, I think Sasha was going to try the patch. I don't think he just committed it. Ira From sashak at voltaire.com Fri Jun 27 09:48:12 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:48:12 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_mcast_mgr.c: Only send MCG delete notice if not previously done In-Reply-To: <1214516203.13056.654.camel@hrosenstock-ws.xsigo.com> References: <1214516203.13056.654.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627164812.GJ29456@sashak.voltaire.com> On 14:36 Thu 26 Jun , Hal Rosenstock wrote: > opensm/osm_mcast_mgr.c: Only send MCG delete notice if not previously > done > > This eliminates duplicated MCG delete notices when MCG is removed > > Signed-off-by: Hal Rosenstock Nice finding. Applied. Thanks. BTW do you remember what was the reason to not delete MCG on a last leave request? I know that spec allows this (but requires to post trap 67 on a last leave), but is there any practical reason for doing this in OpenSM? Sasha From sashak at voltaire.com Fri Jun 27 09:57:32 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 19:57:32 +0300 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080627094951.1ee777db.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> <20080627150811.GC29456@sashak.voltaire.com> <20080627091224.2b18659b.weiny2@llnl.gov> <1214585053.13056.695.camel@hrosenstock-ws.xsigo.com> <20080627094951.1ee777db.weiny2@llnl.gov> Message-ID: <20080627165732.GK29456@sashak.voltaire.com> On 09:49 Fri 27 Jun , Ira Weiny wrote: > > No I did not test it, I think Sasha was going to try the patch. I don't think > he just committed it. Yes, I tested it. Sasha From sashak at voltaire.com Fri Jun 27 10:04:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 20:04:39 +0300 Subject: [ofa-general] Re: [PATCH][TRIVIAL] osm_port_profile: Change some variable names for consistency In-Reply-To: <1214584878.13056.692.camel@hrosenstock-ws.xsigo.com> References: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> <20080627161951.GE29456@sashak.voltaire.com> <1214584878.13056.692.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627170439.GL29456@sashak.voltaire.com> On 09:41 Fri 27 Jun , Hal Rosenstock wrote: > > I already submitted a patch for this. I meant to mention also that it is for switch external ports only. > > Also for me the option name > > '--ignore-guids' looks a bit confusing, I would think that it is about > > and ports. Thoughts? > > Up to you. I agree the name could be improved. What about '--ignore_switch_ports'? Sasha From hrosenstock at xsigo.com Fri Jun 27 10:48:58 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 10:48:58 -0700 Subject: [ofa-general] [PATCH] opensm/osm_port_profile: Handle all possible ports Message-ID: <1214588938.13056.702.camel@hrosenstock-ws.xsigo.com> opensm/osm_port_profile: Handle all possible ports rather than artificial 32 port limit Also, improve error handling when dealing with "ignore guids" file Signed-off-by: Hal Rosenstock diff --git a/opensm/include/opensm/osm_port_profile.h b/opensm/include/opensm/osm_port_profile.h index 42a6561..c7c969c 100644 --- a/opensm/include/opensm/osm_port_profile.h +++ b/opensm/include/opensm/osm_port_profile.h @@ -2,6 +2,7 @@ * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -100,6 +101,26 @@ typedef struct osm_port_profile { * SEE ALSO *********/ +/****s* OpenSM: Switch/osm_port_mask_t +* NAME +* osm_port_mask_t +* +* DESCRIPTION +* The Port Mask object contains a port numbered bit mask +* for whether the port should be ignored by the link load +* equalization algorithm. +* +* SYNOPSIS +*/ +typedef long osm_port_mask_t[32 / sizeof(long)]; +/* +* FIELDS +* osm_port_mask_t +* Bit mask by port number +* +* SEE ALSO +*********/ + /****f* OpenSM: Port Profile/osm_port_prof_construct * NAME * osm_port_prof_construct @@ -198,13 +219,15 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, IN ib_net64_t node_guid, IN uint8_t port_num) { const cl_map_t *p_map = &p_subn->port_prof_ignore_guids; - const void *p_obj = cl_map_get(p_map, node_guid); - size_t res; + void *p_obj = cl_map_get(p_map, node_guid); + long mask, *addr; - // HACK: we currently support ignoring ports 0 - 31 if (p_obj != NULL) { - res = (size_t) p_obj & (size_t) (1 << port_num); - return (res != 0); + /* Test bit corresponding to port_num */ + addr = p_obj; + addr += port_num / (8 * sizeof(long)); + mask = 1L << (port_num % (8 * sizeof(long))); + return ((mask & *addr) != 0); } return FALSE; } @@ -217,7 +240,8 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, * [in] The node guid * * RETURN VALUE -* None. +* Returns TRUE if ignore port mask for requested port number is set. +* FALSE otherwise; * * NOTES * @@ -233,24 +257,36 @@ osm_port_prof_is_ignored_port(IN const osm_subn_t * p_subn, * * SYNOPSIS */ -static inline void +static inline boolean_t osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, IN ib_net64_t node_guid, IN uint8_t port_num) { cl_map_t *p_map = &p_subn->port_prof_ignore_guids; - const void *p_obj = cl_map_get(p_map, node_guid); - size_t value = 0; - - // HACK: we currently support ignoring ports 0 - 31 - CL_ASSERT(port_num < 32); + void *p_obj = cl_map_get(p_map, node_guid); + long mask, *addr; + int insert = 0; - if (p_obj != NULL) { - value = (size_t) p_obj; - cl_map_remove(p_map, node_guid); + if (!p_obj) { + p_obj = malloc(sizeof(osm_port_mask_t)); + if (!p_obj) + return FALSE; + memset(p_obj, 0, sizeof(osm_port_mask_t)); + insert = 1; } - value = value | (1 << port_num); - cl_map_insert(p_map, node_guid, (void *)value); + /* Set bit corresponding to port_num */ + addr = p_obj; + addr += port_num / (8 * sizeof(long)); + mask = 1L << (port_num % (8 * sizeof(long))); + *addr |= mask; + + if (insert) { + if (!cl_map_insert(p_map, node_guid, p_obj)) { + free(p_obj); + return FALSE; + } + } + return TRUE; } /* * PARAMETERS @@ -261,7 +297,8 @@ osm_port_prof_set_ignored_port(IN osm_subn_t * p_subn, * [in] The node guid * * RETURN VALUE -* None. +* Returns TRUE if the ignore port mask was properly updated. +* FALSE otherwise. * * NOTES * diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 7c7525b..0490227 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -504,13 +504,27 @@ parse_ignore_guids_file(IN char *guids_file_name, IN osm_opensm_t * p_osm) goto Exit; } + if (port_num > IB_NODE_NUM_PORTS_MAX) { + OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 0604: " + "Invalid PortNum: 0x%X for Node: 0x%" + PRIx64 "\n", port_num, node_guid); + status = IB_ERROR; + goto Exit; + } + /* ok insert it */ - osm_port_prof_set_ignored_port(&p_osm->subn, - cl_hton64(node_guid), port_num); - OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, - "Inserted Node: 0x%" PRIx64 - " PortNum: 0x%X into ignored guids list\n", node_guid, - port_num); + if (!osm_port_prof_set_ignored_port(&p_osm->subn, + cl_hton64(node_guid), + port_num)) + OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 0605: " + "osm_port_prof_set_ignored_port failed for " + "Node: 0x%" PRIx64 " PortNum: 0x%X\n", + node_guid, port_num); + else + OSM_LOG(&p_osm->log, OSM_LOG_DEBUG, + "Inserted Node: 0x%" PRIx64 + " PortNum: 0x%X into ignored guids list\n", + node_guid, port_num); } diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 73b476f..756495c 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -100,6 +100,8 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) osm_prtn_t *p_prtn, *p_next_prtn; osm_mgrp_t *p_mgrp; osm_infr_t *p_infr, *p_next_infr; + cl_map_iterator_t pmask_iter, next_pmask_iter; + osm_port_mask_t *p_port_mask; /* it might be a good idea to de-allocate all known objects */ p_next_node = (osm_node_t *) cl_qmap_head(&p_subn->node_guid_tbl); @@ -159,7 +161,15 @@ void osm_subn_destroy(IN osm_subn_t * const p_subn) cl_ptr_vector_destroy(&p_subn->port_lid_tbl); - cl_map_remove_all(&p_subn->port_prof_ignore_guids); + next_pmask_iter = cl_map_head(&p_subn->port_prof_ignore_guids); + while (next_pmask_iter != cl_map_end(&p_subn->port_prof_ignore_guids)) { + pmask_iter = next_pmask_iter; + next_pmask_iter = cl_map_next(next_pmask_iter); + p_port_mask = cl_map_obj(pmask_iter); + cl_map_remove_item(&p_subn->port_prof_ignore_guids, pmask_iter); + free(p_port_mask); + } + cl_map_destroy(&p_subn->port_prof_ignore_guids); osm_qos_policy_destroy(p_subn->p_qos_policy); From sashak at voltaire.com Fri Jun 27 10:50:24 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 20:50:24 +0300 Subject: [ofa-general] Re: [PATCH] opensm: add osm_version field to osm_opensm_t object In-Reply-To: <1214500256.13056.608.camel@hrosenstock-ws.xsigo.com> References: <20080625222135.GA28318@sashak.voltaire.com> <1214500256.13056.608.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627175024.GM29456@sashak.voltaire.com> On 10:10 Thu 26 Jun , Hal Rosenstock wrote: > On Thu, 2008-06-26 at 01:21 +0300, Sasha Khapyorsky wrote: > > This adds osm_version field to osm_opensm_t object. > > Is part of the intention here to use this in plugins ? It can be used in plugin for OpenSM version checking (like it is done in osmeventplugin in event plugin API v2 patch), but I didn't think to limit its usage by something specific. > Is this > versioning of osm_opensm_t for that (and other uses too) ? No, it is OpenSM version itself. > If so, that may be a little too granular IMO. Perhaps it should be more > like the libraries in terms of versioning. Yes, it could be useful to do so if it is about API/ABI versioning, but here it is just OpenSM version as is. Sasha From sashak at voltaire.com Fri Jun 27 10:59:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 20:59:39 +0300 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080626093926.65e6abcf.weiny2@llnl.gov> References: <20080625222331.GB28318@sashak.voltaire.com> <20080626093926.65e6abcf.weiny2@llnl.gov> Message-ID: <20080627175939.GN29456@sashak.voltaire.com> On 09:39 Thu 26 Jun , Ira Weiny wrote: > One comment, which I mentioned to you off-list, was that passing struct > osm_opensm without having that header public currently breaks my plugin and > will make it impossible for them to compile against the installed headers. Right, is is disadvantage. > I am really leaning toward including all the headers in the opensm-devel > package so plugin writers don't have to have a git tree to build. I think we can do it. Personally I will not build plugin against RPM package, IMHO plugin writing required OpenSM knowledge at a source code level. But others may want just to build plugin, so it can be useful. Want me to add this? Sasha From weiny2 at llnl.gov Fri Jun 27 11:05:15 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 11:05:15 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627175939.GN29456@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> <20080626093926.65e6abcf.weiny2@llnl.gov> <20080627175939.GN29456@sashak.voltaire.com> Message-ID: <20080627110515.6c120eed.weiny2@llnl.gov> On Fri, 27 Jun 2008 20:59:39 +0300 Sasha Khapyorsky wrote: > On 09:39 Thu 26 Jun , Ira Weiny wrote: > > One comment, which I mentioned to you off-list, was that passing struct > > osm_opensm without having that header public currently breaks my plugin and > > will make it impossible for them to compile against the installed headers. > > Right, is is disadvantage. > > > I am really leaning toward including all the headers in the opensm-devel > > package so plugin writers don't have to have a git tree to build. > > I think we can do it. Personally I will not build plugin against RPM > package, IMHO plugin writing required OpenSM knowledge at a source code > level. But others may want just to build plugin, so it can be useful. Exactly, many people will want to build plugin's without knowing anything. I also see plugins as being something many people add to an already included distro so... > > Want me to add this? > I will add it, as I work through it with my plugin. Ira From hrosenstock at xsigo.com Fri Jun 27 10:55:37 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Fri, 27 Jun 2008 10:55:37 -0700 Subject: [ofa-general] Re: [PATCH][TRIVIAL] osm_port_profile: Change some variable names for consistency In-Reply-To: <20080627170439.GL29456@sashak.voltaire.com> References: <1214489271.13056.600.camel@hrosenstock-ws.xsigo.com> <20080627161951.GE29456@sashak.voltaire.com> <1214584878.13056.692.camel@hrosenstock-ws.xsigo.com> <20080627170439.GL29456@sashak.voltaire.com> Message-ID: <1214589337.13056.706.camel@hrosenstock-ws.xsigo.com> On Fri, 2008-06-27 at 20:04 +0300, Sasha Khapyorsky wrote: > On 09:41 Fri 27 Jun , Hal Rosenstock wrote: > > > > I already submitted a patch for this. > > I meant to mention also that it is for switch external ports only. Sure; but that aspect is in the application of the profile rather than in the specification of the entries in the file. > > > Also for me the option name > > > '--ignore-guids' looks a bit confusing, I would think that it is about > > > and ports. Thoughts? > > > > Up to you. I agree the name could be improved. > > What about '--ignore_switch_ports'? Makes sense to me. Not sure whether this disturbs any existing users though. Yevgeny, do you have any idea about this ? -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Fri Jun 27 11:19:31 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 21:19:31 +0300 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080627181931.GO29456@sashak.voltaire.com> On 10:26 Thu 26 Jun , Hal Rosenstock wrote: > On Thu, 2008-06-26 at 01:23 +0300, Sasha Khapyorsky wrote: > > Main difference is that construct() method is renamed to create() and > > gets reference to osm_opensm_t object instead of just osm_log_t. This > > will provide to event plugin access to all OpenSM data structures and > > should help to create more generic plugins. > > This exposes almost all internal data structures! While that is > powerful, it also could be dangerous as plugins can now more readily > corrupt OpenSM. It's certainly the most flexible but leaves everything > totally open. Is this the best approach ? I think so. Having complicated interface instead is also error prone (but not flexible) implementation. I expect some level of OpenSM internals knowledge from plugin writers. Finally it is all in their hands. > If this goes ahead, versioning is an issue. Not sure osm_version is the > best barometer for everything. It may be for include files for building > a plugin but what about a coarser versioning (like library versioning) ? Then this will need to track every single data structure or API change. Plugin may require this or not, I prefer to leave this for a writer (at least now). > It would be nice if a single plugin can handle as many OpenSM versions > as possible (by something simpler than a range of osm_versions) rather > than needing a separate plugin per OpenSM version at the other end of > the spectrum. I'm not sure that this should be hard requirement (the point of plugin is to not make easy life for proprietary guys, but rather to add possibility to extend OpenSM functionality by not directly SM related and so pretty optional features). I think to rebuild plugin against used OpenSM is acceptable. Finally it is plugin writer decision how to use osm_version. Sasha From sashak at voltaire.com Fri Jun 27 11:20:34 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 27 Jun 2008 21:20:34 +0300 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627110515.6c120eed.weiny2@llnl.gov> References: <20080625222331.GB28318@sashak.voltaire.com> <20080626093926.65e6abcf.weiny2@llnl.gov> <20080627175939.GN29456@sashak.voltaire.com> <20080627110515.6c120eed.weiny2@llnl.gov> Message-ID: <20080627182034.GP29456@sashak.voltaire.com> On 11:05 Fri 27 Jun , Ira Weiny wrote: > > I will add it, as I work through it with my plugin. Great. Thanks! Sasha From weiny2 at llnl.gov Fri Jun 27 13:03:33 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 13:03:33 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627181931.GO29456@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> Message-ID: <20080627130333.08d783c6.weiny2@llnl.gov> On Fri, 27 Jun 2008 21:19:31 +0300 Sasha Khapyorsky wrote: > On 10:26 Thu 26 Jun , Hal Rosenstock wrote: > > On Thu, 2008-06-26 at 01:23 +0300, Sasha Khapyorsky wrote: > > > Main difference is that construct() method is renamed to create() and > > > gets reference to osm_opensm_t object instead of just osm_log_t. This > > > will provide to event plugin access to all OpenSM data structures and > > > should help to create more generic plugins. > > > > This exposes almost all internal data structures! While that is > > powerful, it also could be dangerous as plugins can now more readily > > corrupt OpenSM. It's certainly the most flexible but leaves everything > > totally open. Is this the best approach ? While I share your concerns, I have to admit this is a good compromise for Sasha. > > I think so. Having complicated interface instead is also error prone > (but not flexible) implementation. I expect some level of OpenSM > internals knowledge from plugin writers. Finally it is all in their hands. The inflexibility has already been demonstrated when I was trying to get node type through the interface. Sasha did not want to have to continue to rev the plugin interface for this. While this open interface would allow plugins to corrupt things any plugin could cause problems if they want to be malicious. > > > If this goes ahead, versioning is an issue. Not sure osm_version is the > > best barometer for everything. It may be for include files for building > > a plugin but what about a coarser versioning (like library versioning) ? > > Then this will need to track every single data structure or API change. > Plugin may require this or not, I prefer to leave this for a writer (at > least now). > > > It would be nice if a single plugin can handle as many OpenSM versions > > as possible (by something simpler than a range of osm_versions) rather > > than needing a separate plugin per OpenSM version at the other end of > > the spectrum. > > I'm not sure that this should be hard requirement (the point of plugin > is to not make easy life for proprietary guys, but rather to add > possibility to extend OpenSM functionality by not directly SM related and > so pretty optional features). I think to rebuild plugin against used > OpenSM is acceptable. Finally it is plugin writer decision how to use > osm_version. > I think I disagree with Sasha on this one. OpenSM should protect itself from plugins with the wrong version. Ira From jgunthorpe at obsidianresearch.com Fri Jun 27 13:37:45 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 27 Jun 2008 14:37:45 -0600 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627130333.08d783c6.weiny2@llnl.gov> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> Message-ID: <20080627203745.GA29198@obsidianresearch.com> On Fri, Jun 27, 2008 at 01:03:33PM -0700, Ira Weiny wrote: > > Then this will need to track every single data structure or API change. > > Plugin may require this or not, I prefer to leave this for a writer (at > > least now). So what is the point of plugins if: - The plugin has access to so much internal state that it must be recompiled every time you change virtually any struct in opensm - The plugin has no isolation from opensm so it isn't acting as a fault containment or simplified API type of thing - opensm is BSD licensed anyhow, so this clearly isn't for any reasons of GPL avoidance or what have you. How is this better than just patching opensm directly for people who want these kinds of things? You can do alot of good by making the internal APIs 'plug in like' so adding new things doesn't require touching lots of places without going down the whole messy road of actual dynamically loadable plug ins.. But if you can't identify a fixed, clean API for a dynamically loadable plugin then you almost certainly should not have them in an open source project... This is probably better long term as far as encouraging more contribution to opensm rather than encouraging contributions to be kept as plugins.. Jason From akstcafghanamnsdgs at afghana.com Fri Jun 27 13:50:33 2008 From: akstcafghanamnsdgs at afghana.com (Yvonne Grubbs) Date: Fri, 27 Jun 2008 21:50:33 +0100 Subject: [ofa-general] Exclusive Nârçotiç Alternative Message-ID: <01c8d89f$d32cfe00$93e9304e@akstcafghanamnsdgs> Elite Medicinal Phârm http://ademirmion38407.blogspot.com/ called "Rolf." From sashak at voltaire.com Fri Jun 27 14:01:02 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 00:01:02 +0300 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627130333.08d783c6.weiny2@llnl.gov> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> Message-ID: <20080627210102.GT29456@sashak.voltaire.com> On 13:03 Fri 27 Jun , Ira Weiny wrote: > > I think I disagree with Sasha on this one. OpenSM should protect itself from > plugins with the wrong version. Basically I'm fine with this approach too, but thought to move it to plugin side because some (hypothetical, I don't know such) can do something very simple (for instance to write trap delivery times in a file) and don't touch OpenSM at all, for such osm_version check (and rebuild requirement) would be overkill. This my concern is really very minor, so I'm ready to place osm_version check in OpenSM itself. Sasha From weiny2 at llnl.gov Fri Jun 27 14:37:24 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 14:37:24 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627203745.GA29198@obsidianresearch.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> <20080627203745.GA29198@obsidianresearch.com> Message-ID: <20080627143724.70df3654.weiny2@llnl.gov> On Fri, 27 Jun 2008 14:37:45 -0600 Jason Gunthorpe wrote: > On Fri, Jun 27, 2008 at 01:03:33PM -0700, Ira Weiny wrote: > > > Then this will need to track every single data structure or API change. > > > Plugin may require this or not, I prefer to leave this for a writer (at > > > least now). > > So what is the point of plugins if: > - The plugin has access to so much internal state that it must be > recompiled every time you change virtually any struct in opensm > - The plugin has no isolation from opensm so it isn't acting as a > fault containment or simplified API type of thing > - opensm is BSD licensed anyhow, so this clearly isn't for any > reasons of GPL avoidance or what have you. > > How is this better than just patching opensm directly for people who > want these kinds of things? There are 2 main reasons I have here at LLNL for wanting this: 1) (the original main reason) was because I wanted to connect OpenSM to MySQL. For many a dependency on MySQL for OpenSM is unreasonable. 2) There are concerns with how much OpenSM can/should do running on various types of hardware. Here are LLNL we have some pretty beefy nodes running OpenSM. (The latest has 16 cores). Having more monitoring threads (or other features) is not a big deal for us. For some, this might not be the case. While features can be turned on and off with config files, sometimes not including code is a better solution. For example an embedded OpenSM? So when we come up with some wild extension to OpenSM I can see some in the community not wanting it included. Having to maintain a forked OpenSM source base is a pain. Having a "plugin" which is compiled separately, but stand alone, is nice. > > You can do alot of good by making the internal APIs 'plug in like' so > adding new things doesn't require touching lots of places without > going down the whole messy road of actual dynamically loadable plug ins.. Unfortunately that is not the current state of things. We are working toward that, "plugin routing algo's" for example, are not really plugins. > > But if you can't identify a fixed, clean API for a dynamically > loadable plugin then you almost certainly should not have them in an > open source project... While I agree with you there are just somethings which will never get accepted to OpenSM. If a plugin becomes more mainstream than hopefully that writer will be able to integrate it, and I think Sasha would encourage it. > > This is probably better long term as far as encouraging more > contribution to opensm rather than encouraging contributions to be > kept as plugins.. > Yes this might cause some people not to contribute. However, I think many already are already not contributing and just maintaining forked versions. While my intentions with a plugin interface are not to make current "non-contributor" lives easier perhaps it will make future potential contributors lives easier and they can be brought into the mainstream? Does this help? Ira From rdreier at cisco.com Fri Jun 27 14:43:20 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Jun 2008 14:43:20 -0700 Subject: [ofa-general] [PATCH] IB/uverbs: BKL is not needed for ib_uverbs_open() In-Reply-To: <20080627095418.2353267d@bike.lwn.net> (Jonathan Corbet's message of "Fri, 27 Jun 2008 09:54:18 -0600") References: <20080627095418.2353267d@bike.lwn.net> Message-ID: Remove explicit lock_kernel() calls and document why the code is safe. Signed-off-by: Roland Dreier --- Hi Jon, Is this sort of patch the next step in the BKL-removal plan? Is now an appropriate time to send such things? If this looks good to you I will do a careful audit of the rest of drivers/infiniband/core and send you a few more similar patches. Thanks, Roland drivers/infiniband/core/uverbs_main.c | 20 +++++++++++++------- 1 files changed, 13 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c index f839dde..661b3f8 100644 --- a/drivers/infiniband/core/uverbs_main.c +++ b/drivers/infiniband/core/uverbs_main.c @@ -45,7 +45,6 @@ #include #include #include -#include #include @@ -611,23 +610,32 @@ static int ib_uverbs_mmap(struct file *filp, struct vm_area_struct *vma) return file->device->ib_dev->mmap(file->ucontext, vma); } +/* + * ib_uverbs_open() does not need the BKL: + * + * - dev_table[] accesses are protected by map_lock, the + * ib_uverbs_device structures are properly reference counted, and + * everything else is purely local to the file being created, so + * races against other open calls are not a proble; + * - there is no ioctl method to race against; + * - the device is added to dev_table[] as the last part of module + * initialization, the open method will either immediately run + * -ENXIO, or all required initialization will be done. + */ static int ib_uverbs_open(struct inode *inode, struct file *filp) { struct ib_uverbs_device *dev; struct ib_uverbs_file *file; int ret; - lock_kernel(); spin_lock(&map_lock); dev = dev_table[iminor(inode) - IB_UVERBS_BASE_MINOR]; if (dev) kref_get(&dev->ref); spin_unlock(&map_lock); - if (!dev) { - unlock_kernel(); + if (!dev) return -ENXIO; - } if (!try_module_get(dev->ib_dev->owner)) { ret = -ENODEV; @@ -648,7 +656,6 @@ static int ib_uverbs_open(struct inode *inode, struct file *filp) filp->private_data = file; - unlock_kernel(); return 0; err_module: @@ -656,7 +663,6 @@ err_module: err: kref_put(&dev->ref, ib_uverbs_release_dev); - unlock_kernel(); return ret; } -- 1.5.6 From rdreier at cisco.com Fri Jun 27 15:26:31 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Jun 2008 15:26:31 -0700 Subject: [ofa-general] Re: [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: <1214435300.30933.14.camel@dumpserver> (David J. Wilder's message of "Wed, 25 Jun 2008 16:08:19 -0700") References: <1214435300.30933.14.camel@dumpserver> Message-ID: I just added this patch to my tree, which I hope should fix this problem without adding a huge stack space allocation (which might cause other issues under load later). It's a little more complicated but I feel better about fixing things this way. Can you guys review/test this and let me know if it looks OK? Thanks, Roland >From 93c47a64b0bf01b69e711698c3bc1b10deb1ca6f Mon Sep 17 00:00:00 2001 From: Roland Dreier Date: Fri, 27 Jun 2008 15:24:15 -0700 Subject: [PATCH] IPoIB/cm: Fix racy use of receive WR/SGL in ipoib_cm_post_receive_nonsrq() For devices that don't support SRQs, ipoib_cm_post_receive_nonsrq() is called from both ipoib_cm_handle_rx_wc() and ipoib_cm_nonsrq_init_rx(), and these two callers are not synchronized against each other. However, ipoib_cm_post_receive_nonsrq() always reuses the same receive work request and scatter list structures, so multiple callers can end up stepping on each other, which leads to posting garbled work requests. Fix this by having the caller pass in the ib_recv_wr and ib_sge structures to use, and allocating new local structures in ipoib_cm_nonsrq_init_rx(). Based on a patch by Pradeep Satyanarayana and David Wilder , with debugging help from Hoang-Nam Nguyen . Signed-off-by: Roland Dreier --- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 63 +++++++++++++++++++++++-------- 1 files changed, 47 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 6223fc3..37bf67b 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -111,18 +111,20 @@ static int ipoib_cm_post_receive_srq(struct net_device *dev, int id) } static int ipoib_cm_post_receive_nonsrq(struct net_device *dev, - struct ipoib_cm_rx *rx, int id) + struct ipoib_cm_rx *rx, + struct ib_recv_wr *wr, + struct ib_sge *sge, int id) { struct ipoib_dev_priv *priv = netdev_priv(dev); struct ib_recv_wr *bad_wr; int i, ret; - priv->cm.rx_wr.wr_id = id | IPOIB_OP_CM | IPOIB_OP_RECV; + wr->wr_id = id | IPOIB_OP_CM | IPOIB_OP_RECV; for (i = 0; i < IPOIB_CM_RX_SG; ++i) - priv->cm.rx_sge[i].addr = rx->rx_ring[id].mapping[i]; + sge[i].addr = rx->rx_ring[id].mapping[i]; - ret = ib_post_recv(rx->qp, &priv->cm.rx_wr, &bad_wr); + ret = ib_post_recv(rx->qp, wr, &bad_wr); if (unlikely(ret)) { ipoib_warn(priv, "post recv failed for buf %d (%d)\n", id, ret); ipoib_cm_dma_unmap_rx(priv, IPOIB_CM_RX_SG - 1, @@ -320,10 +322,33 @@ static int ipoib_cm_modify_rx_qp(struct net_device *dev, return 0; } +static void ipoib_cm_init_rx_wr(struct net_device *dev, + struct ib_recv_wr *wr, + struct ib_sge *sge) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + int i; + + for (i = 0; i < priv->cm.num_frags; ++i) + sge[i].lkey = priv->mr->lkey; + + sge[0].length = IPOIB_CM_HEAD_SIZE; + for (i = 1; i < priv->cm.num_frags; ++i) + sge[i].length = PAGE_SIZE; + + wr->next = NULL; + wr->sg_list = priv->cm.rx_sge; + wr->num_sge = priv->cm.num_frags; +} + static int ipoib_cm_nonsrq_init_rx(struct net_device *dev, struct ib_cm_id *cm_id, struct ipoib_cm_rx *rx) { struct ipoib_dev_priv *priv = netdev_priv(dev); + struct { + struct ib_recv_wr wr; + struct ib_sge sge[IPOIB_CM_RX_SG]; + } *t; int ret; int i; @@ -331,6 +356,14 @@ static int ipoib_cm_nonsrq_init_rx(struct net_device *dev, struct ib_cm_id *cm_i if (!rx->rx_ring) return -ENOMEM; + t = kmalloc(sizeof *t, GFP_KERNEL); + if (!t) { + ret = -ENOMEM; + goto err_free; + } + + ipoib_cm_init_rx_wr(dev, &t->wr, t->sge); + spin_lock_irq(&priv->lock); if (priv->cm.nonsrq_conn_qp >= ipoib_max_conn_qp) { @@ -349,8 +382,8 @@ static int ipoib_cm_nonsrq_init_rx(struct net_device *dev, struct ib_cm_id *cm_i ipoib_warn(priv, "failed to allocate receive buffer %d\n", i); ret = -ENOMEM; goto err_count; - } - ret = ipoib_cm_post_receive_nonsrq(dev, rx, i); + } + ret = ipoib_cm_post_receive_nonsrq(dev, rx, &t->wr, t->sge, i); if (ret) { ipoib_warn(priv, "ipoib_cm_post_receive_nonsrq " "failed for buf %d\n", i); @@ -361,6 +394,8 @@ static int ipoib_cm_nonsrq_init_rx(struct net_device *dev, struct ib_cm_id *cm_i rx->recv_count = ipoib_recvq_size; + kfree(t); + return 0; err_count: @@ -369,6 +404,7 @@ err_count: spin_unlock_irq(&priv->lock); err_free: + kfree(t); ipoib_cm_free_rx_ring(dev, rx->rx_ring); return ret; @@ -637,7 +673,10 @@ repost: ipoib_warn(priv, "ipoib_cm_post_receive_srq failed " "for buf %d\n", wr_id); } else { - if (unlikely(ipoib_cm_post_receive_nonsrq(dev, p, wr_id))) { + if (unlikely(ipoib_cm_post_receive_nonsrq(dev, p, + &priv->cm.rx_wr, + priv->cm.rx_sge, + wr_id))) { --p->recv_count; ipoib_warn(priv, "ipoib_cm_post_receive_nonsrq failed " "for buf %d\n", wr_id); @@ -1502,15 +1541,7 @@ int ipoib_cm_dev_init(struct net_device *dev) priv->cm.num_frags = IPOIB_CM_RX_SG; } - for (i = 0; i < priv->cm.num_frags; ++i) - priv->cm.rx_sge[i].lkey = priv->mr->lkey; - - priv->cm.rx_sge[0].length = IPOIB_CM_HEAD_SIZE; - for (i = 1; i < priv->cm.num_frags; ++i) - priv->cm.rx_sge[i].length = PAGE_SIZE; - priv->cm.rx_wr.next = NULL; - priv->cm.rx_wr.sg_list = priv->cm.rx_sge; - priv->cm.rx_wr.num_sge = priv->cm.num_frags; + ipoib_cm_init_rx_wr(dev, &priv->cm.rx_wr, priv->cm.rx_sge); if (ipoib_cm_has_srq(dev)) { for (i = 0; i < ipoib_recvq_size; ++i) { -- 1.5.6 From rdreier at cisco.com Fri Jun 27 15:27:37 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Jun 2008 15:27:37 -0700 Subject: [ofa-general] Re: [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: (Hoang-Nam Nguyen's message of "Thu, 26 Jun 2008 12:24:16 +0200") References: Message-ID: > - enhance ipoib_cm_post_receive_nonsrq() with wr and sge passed to by > the caller, ie ipoib_poll() -> ipoib_cm_handle_rx_wc() resp. > ipoib_cm_nonsrq_init_rx() > - ipoib_cm_handle_rx_wc() passes rx_wr and rx_sge as already defined > in struct ipoib_cm_dev_priv to ipoib_cm_post_receive_nonsrq() > since we use one cq, there should not be a race in this path > - ipoib_cm_nonsrq_init_rx() allocates wr and sge on the heap or slabcache, > passes them to ipoib_cm_post_receive_nonsrq() and frees them thereafter I just wrote a patch before reading your email, and this is exactly what I did. Thanks, Roland From jgunthorpe at obsidianresearch.com Fri Jun 27 15:31:18 2008 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 27 Jun 2008 16:31:18 -0600 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627143724.70df3654.weiny2@llnl.gov> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> <20080627203745.GA29198@obsidianresearch.com> <20080627143724.70df3654.weiny2@llnl.gov> Message-ID: <20080627223118.GB29198@obsidianresearch.com> On Fri, Jun 27, 2008 at 02:37:24PM -0700, Ira Weiny wrote: > > How is this better than just patching opensm directly for people who > > want these kinds of things? > > There are 2 main reasons I have here at LLNL for wanting this: > > 1) (the original main reason) was because I wanted to connect OpenSM to > MySQL. For many a dependency on MySQL for OpenSM is unreasonable. Well, if you look out there, you see alot of projects that bring everything into the same source tree, including stuff like this. It is a reasonable thing to do if the feature is broadly useful. Exim for instance links to a jillion libraries, including mysql - but it does have compile time options to strip stuff out if you really care about smallness. These days linking to some common library is not a big problem.. The distribution has it packaged seperately and you can just install it. > community not wanting it included. Having to maintain a forked > OpenSM source base is a pain. Having a "plugin" which is > compiled separately, but stand alone, is nice. How so? git makes a fork pretty easy to keep track of, and if the plugin is tied tightly to the opensm codebase you still have to deal with most of the same issues, like fixing 'merge conflicts' and recompiling. The other path many projects take is instead of adding plugins they add a scripting language. There are many script languages that can be embedded. Exim for instance includes perl. I'm not sure this is much better since maintaining the script hooks is a fair amount of work as well, but at least you get quite alot of return for your efforts.. I've often thought that a nice python mad processing library would be hugely better than the mismash of C utilities and perl post-processing we have today, if integrated with opensm it might just be quite the fantastic thing. But a big project :) > > This is probably better long term as far as encouraging more > > contribution to opensm rather than encouraging contributions to be > > kept as plugins.. > Yes this might cause some people not to contribute. However, I > think many already are already not contributing and just maintaining > forked versions. While my intentions with a plugin interface are > not to make current "non-contributor" lives easier perhaps it will > make future potential contributors lives easier and they can be > brought into the mainstream? There is also a nasty trap here, once the plugin is written there is almost no motivation to get it into the main line. Lots of projects seem to end up with a bundle of plugins with poor code quality ... Anyhow, the huge thing to avoid is a fear of changing opensm because it would break all those out of tree plugins people are using. If things ever get to that point then adding plugin support was a huge mistake :) Jason From obovoid at sitecontrol2.com Fri Jun 27 17:46:53 2008 From: obovoid at sitecontrol2.com (Israel Anderson) Date: Fri, 27 Jun 2008 18:46:53 -0600 Subject: [ofa-general] +Photo shop CS 3+ Message-ID: <000701c8d8a6$70480d00$0100007f@tkwsg> + Adobe CS3 Master Collection for PC or MAC includes: + InDesign CS3 + Photoshop CS3 + Illustrator CS3 + Acrobat 8 Professional + Flash CS3 Professional + Dreamweaver CS3 + Fireworks CS3 + Contribute CS3 + After Effects CS3 Professional + Premiere Pro CS3 + Encore DVD CS3 + Soundbooth CS3 + esoftnew .com in Internet browser + System Requirements + For PC: + Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core + Duo (or compatible) processor; SSE2-enabled processor required for AMD systems + Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) + 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components + 38GB of available hard-disk space (additional free space required during installation) + Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred + Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) + 1,280x1,024 monitor resolution with 32-bit color adapter + DVD-ROM drive + For MAC: + PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) + Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server + 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components + 36GB of available hard-disk space (additional free space required during installation) + Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred + Core Audio compatible sound card + 1,280x1,024 monitor resolution with 32-bit color adapter + DVD-ROM drive+ DVD+-R burner required for DVD creation Disabled Groups Say Little has Changed in China Turkish religious scholar Fethullah Gulen ranks first in Foreign Policy's survey of public intellectuals. He's followed by nine Muslim thinkers from predominantly Muslim nations. Editor Kate Palmer says the results reflect a wave of campaigning for top slots. From weiny2 at llnl.gov Fri Jun 27 16:02:19 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 16:02:19 -0700 Subject: [ofa-general] Re: [PATCH RFC v3] opensm/event_plugin: plugin API version 3 In-Reply-To: <20080625222331.GB28318@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> Message-ID: <20080627160219.6ac21033.weiny2@llnl.gov> New version which includes all the headers in the devel package. As a side note it was not hard to convert the plugin over. My vote is to go for this. Ira >From 66ddb2808fc20b12afe87c72ab41584c3a0f387e Mon Sep 17 00:00:00 2001 From: Sasha Khapyorsky Date: Thu, 26 Jun 2008 01:23:31 +0300 Subject: [PATCH] opensm/event_plugin: plugin API version 3 Main difference is that construct() method is renamed to create() and gets reference to osm_opensm_t object instead of just osm_log_t. This will provide to event plugin access to all OpenSM data structures and should help to create more generic plugins. For consistency with above destroy() method is renamed to delete(). Event plugin interface version is increased and osmeventplugin example updated accordingly. Modified from Version 2 to include all the headers in the "devel" package. Signed-off-by: Sasha Khapyorsky Signed-off-by: Ira K. Weiny --- opensm/include/opensm/osm_event_plugin.h | 12 ++++++------ opensm/opensm/Makefile.am | 9 +-------- opensm/opensm/osm_event_plugin.c | 23 +++++++++++------------ opensm/opensm/osm_opensm.c | 2 +- opensm/osmeventplugin/src/osmeventplugin.c | 18 ++++++++++++------ 5 files changed, 31 insertions(+), 33 deletions(-) diff --git a/opensm/include/opensm/osm_event_plugin.h b/opensm/include/opensm/osm_event_plugin.h index d9e546c..fa2a8b8 100644 --- a/opensm/include/opensm/osm_event_plugin.h +++ b/opensm/include/opensm/osm_event_plugin.h @@ -37,7 +37,6 @@ #include #include #include -#include #ifdef __cplusplus # define BEGIN_C_DECLS extern "C" { @@ -60,6 +59,8 @@ BEGIN_C_DECLS *********/ #define OSM_EPI_NODE_NAME_LEN (128) + +struct osm_opensm; /** ========================================================================= * Event types */ @@ -144,11 +145,11 @@ typedef struct osm_epi_trap_event { * The version should be set to OSM_EVENT_PLUGIN_INTERFACE_VER */ #define OSM_EVENT_PLUGIN_IMPL_NAME "osm_event_plugin" -#define OSM_EVENT_PLUGIN_INTERFACE_VER (1) +#define OSM_EVENT_PLUGIN_INTERFACE_VER 2 typedef struct osm_event_plugin { int interface_version; - void *(*construct) (osm_log_t * osm_log); - void (*destroy) (void *plugin_data); + void *(*create) (struct osm_opensm *osm); + void (*delete) (void *plugin_data); void (*report) (void *plugin_data, osm_epi_event_id_t event_id, void *event_data); } osm_event_plugin_t; @@ -161,14 +162,13 @@ typedef struct osm_epi_plugin { void *handle; osm_event_plugin_t *impl; void *plugin_data; - osm_log_t *p_log; char *plugin_name; } osm_epi_plugin_t; /** * functions */ -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name); +osm_epi_plugin_t *osm_epi_construct(struct osm_opensm *osm, char *plugin_name); void osm_epi_destroy(osm_epi_plugin_t * plugin); /** ========================================================================= diff --git a/opensm/opensm/Makefile.am b/opensm/opensm/Makefile.am index acd0b1d..a10e947 100644 --- a/opensm/opensm/Makefile.am +++ b/opensm/opensm/Makefile.am @@ -93,14 +93,7 @@ opensm_LDFLAGS = -Wl,--rpath -Wl,$(libdir) -lpthread -ldl opensmincludedir = $(includedir)/infiniband/opensm -opensminclude_HEADERS = $(srcdir)/../include/opensm/osm_base.h \ - $(srcdir)/../include/opensm/osm_log.h \ - $(srcdir)/../include/opensm/osm_path.h \ - $(srcdir)/../include/opensm/osm_madw.h \ - $(srcdir)/../include/opensm/osm_mad_pool.h \ - $(srcdir)/../include/opensm/osm_msgdef.h \ - $(srcdir)/../include/opensm/osm_helper.h \ - $(srcdir)/../include/opensm/osm_event_plugin.h +opensminclude_HEADERS = $(srcdir)/../include/opensm/*.h BUILT_SOURCES = osm_version osm_version: diff --git a/opensm/opensm/osm_event_plugin.c b/opensm/opensm/osm_event_plugin.c index e579fad..98d5302 100644 --- a/opensm/opensm/osm_event_plugin.c +++ b/opensm/opensm/osm_event_plugin.c @@ -44,8 +44,8 @@ #include #include - #include +#include #if defined(PATH_MAX) #define OSM_PATH_MAX (PATH_MAX + 1) @@ -58,7 +58,7 @@ /** * functions */ -osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) +osm_epi_plugin_t *osm_epi_construct(osm_opensm_t *osm, char *plugin_name) { char lib_name[OSM_PATH_MAX]; osm_epi_plugin_t *rc = NULL; @@ -75,7 +75,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) rc->handle = dlopen(lib_name, RTLD_LAZY); if (!rc->handle) { - OSM_LOG(p_log, OSM_LOG_ERROR, + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Failed to open event plugin \"%s\" : \"%s\"\n", lib_name, dlerror()); goto DLOPENFAIL; @@ -85,7 +85,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) (osm_event_plugin_t *) dlsym(rc->handle, OSM_EVENT_PLUGIN_IMPL_NAME); if (!rc->impl) { - OSM_LOG(p_log, OSM_LOG_ERROR, + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Failed to find \"%s\" symbol in \"%s\" : \"%s\"\n", OSM_EVENT_PLUGIN_IMPL_NAME, lib_name, dlerror()); goto Exit; @@ -93,7 +93,7 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) /* Check the version to make sure this module will work with us */ if (rc->impl->interface_version != OSM_EVENT_PLUGIN_INTERFACE_VER) { - OSM_LOG(p_log, OSM_LOG_ERROR, + OSM_LOG(&osm->log, OSM_LOG_ERROR, "Error opening %s: " "%s symbol is the wrong version %d != %d\n", plugin_name, @@ -103,19 +103,18 @@ osm_epi_plugin_t *osm_epi_construct(osm_log_t * p_log, char *plugin_name) goto Exit; } - if (!rc->impl->construct) { - OSM_LOG(p_log, OSM_LOG_ERROR, - "%s symbol has no construct function\n", + if (!rc->impl->create) { + OSM_LOG(&osm->log, OSM_LOG_ERROR, + "%s symbol has no create() function\n", OSM_EVENT_PLUGIN_IMPL_NAME); goto Exit; } - rc->plugin_data = rc->impl->construct(p_log); + rc->plugin_data = rc->impl->create(osm); if (!rc->plugin_data) goto Exit; - rc->p_log = p_log; rc->plugin_name = strdup(plugin_name); return (rc); @@ -129,8 +128,8 @@ DLOPENFAIL: void osm_epi_destroy(osm_epi_plugin_t * plugin) { if (plugin) { - if (plugin->impl->destroy) - plugin->impl->destroy(plugin->plugin_data); + if (plugin->impl->delete) + plugin->impl->delete(plugin->plugin_data); dlclose(plugin->handle); free(plugin->plugin_name); free(plugin); diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index adacd93..a6dda21 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -250,7 +250,7 @@ static void load_plugins(osm_opensm_t *osm, char *plugin_names) name = strtok_r(plugin_names, " \t\n", &p); while (name && *name) { - epi = osm_epi_construct(&osm->log, name); + epi = osm_epi_construct(osm, name); if (!epi) osm_log(&osm->log, OSM_LOG_ERROR, "cannot load plugin \'%s\'\n", name); diff --git a/opensm/osmeventplugin/src/osmeventplugin.c b/opensm/osmeventplugin/src/osmeventplugin.c index 6cc4c70..a56d3c3 100644 --- a/opensm/osmeventplugin/src/osmeventplugin.c +++ b/opensm/osmeventplugin/src/osmeventplugin.c @@ -38,12 +38,15 @@ #include #include #include +#include #include #include #include -#include #include #include +#include +#include +#include /** ========================================================================= * This is a simple example plugin which logs some of the events the OSM @@ -57,8 +60,11 @@ typedef struct _log_events { /** ========================================================================= */ -static void *construct(osm_log_t * osmlog) +static void *construct(osm_opensm_t *osm) { + if (strcmp(osm->osm_version, OSM_VERSION)) + return NULL; + _log_events_t *log = malloc(sizeof(*log)); if (!log) return (NULL); @@ -66,14 +72,14 @@ static void *construct(osm_log_t * osmlog) log->log_file = fopen(SAMPLE_PLUGIN_OUTPUT_FILE, "a+"); if (!(log->log_file)) { - osm_log(osmlog, OSM_LOG_ERROR, + osm_log(&osm->log, OSM_LOG_ERROR, "Sample Event Plugin: Failed to open output file \"%s\"\n", SAMPLE_PLUGIN_OUTPUT_FILE); free(log); return (NULL); } - log->osmlog = osmlog; + log->osmlog = &osm->log; return ((void *)log); } @@ -173,7 +179,7 @@ static void report(void *_log, osm_epi_event_id_t event_id, void *event_data) */ osm_event_plugin_t osm_event_plugin = { interface_version:OSM_EVENT_PLUGIN_INTERFACE_VER, - construct:construct, - destroy:destroy, + create:construct, + delete:destroy, report:report }; -- 1.5.4.5 From sashak at voltaire.com Fri Jun 27 16:03:39 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 02:03:39 +0300 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627203745.GA29198@obsidianresearch.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> <20080627203745.GA29198@obsidianresearch.com> Message-ID: <20080627230339.GV29456@sashak.voltaire.com> Hi Jason, On 14:37 Fri 27 Jun , Jason Gunthorpe wrote: > > How is this better than just patching opensm directly for people who > want these kinds of things? It is not better, it is different. Some things can be unrelated to "pure" SM/SA functionality and not be part of OpenSM project, but sometimes we may want to have it as part of OpenSM process, examples are existing plugin 'opensmskumme' (it loads performance counters to MySQL db), we wanted to have IB diagnostics plugin and management related plugin. > You can do alot of good by making the internal APIs 'plug in like' so > adding new things doesn't require touching lots of places without > going down the whole messy road of actual dynamically loadable plug ins.. > > But if you can't identify a fixed, clean API for a dynamically > loadable plugin then you almost certainly should not have them in an > open source project... > > This is probably better long term as far as encouraging more > contribution to opensm rather than encouraging contributions to be > kept as plugins.. GPL is some sort of an issue - OFA requires dual licensing, so GPL only code cannot enter OpenSM. Sasha From weiny2 at llnl.gov Fri Jun 27 16:09:52 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 27 Jun 2008 16:09:52 -0700 Subject: [ofa-general] [PATCH RFC] Create a new library "libibnetdisc" Message-ID: <20080627160952.600be0e2.weiny2@llnl.gov> I have been wanted to do this for some time and ever since mentioning it to Sasha at Sonoma have worked on it in my "spare" time... ;-) This creates an ibnetdiscover library. It seems as time goes on that more and more people have tried to "print" something from ibnetdiscover to get that data out with some parser. I figure it should be available from a library. I have only written test programs "ibnetdisctest" and "iblinkinfotest" to prototype performance. Also the interface is a bit rought around the edges but all the data is there. At this point I have some pretty good results from iblinkinfotest (~80% faster!) So I wanted to get opinions on where this should go. Thanks, Ira >From 0c66cf5e0687bc1c1c88c93f8c39f719dd2674e5 Mon Sep 17 00:00:00 2001 From: Ira K. Weiny Date: Thu, 17 Apr 2008 15:24:21 -0700 Subject: [PATCH] Create a new library libibnetdisc This encompasses the functionality of ibnetdiscover in a C library. It returns a single "ibnd_fabric_t" object which represents the data found during the scan. The NodeInfo, PortInfo, and SwitchInfo are preserved from the queries made on the fabric to be used by the calling function as they see fit. This greatly benefits some diags like iblinkinfo.pl. This diag in particular was re-written using this library in C and has shown an 85% speed up on a ~1000 node cluster. Previous iblinkinfo.pl real 3m35.876s user 0m13.210s sys 1m1.046s New iblinkinfotest real 0m32.869s user 0m0.067s sys 0m0.140s Signed-off-by: Ira K. Weiny --- libibnetdisc/AUTHORS | 1 + libibnetdisc/COPYING | 384 +++++++++++++ libibnetdisc/ChangeLog | 4 + libibnetdisc/Makefile.am | 54 ++ libibnetdisc/autogen.sh | 11 + libibnetdisc/configure.in | 64 +++ libibnetdisc/include/infiniband/ibnetdisc.h | 294 ++++++++++ libibnetdisc/libibnetdisc.spec.in | 71 +++ libibnetdisc/libibnetdisc.ver | 9 + libibnetdisc/src/chassis.c | 806 +++++++++++++++++++++++++++ libibnetdisc/src/chassis.h | 82 +++ libibnetdisc/src/iblinkinfotest.c | 388 +++++++++++++ libibnetdisc/src/ibnetdisc.c | 792 ++++++++++++++++++++++++++ libibnetdisc/src/ibnetdisctest.c | 636 +++++++++++++++++++++ libibnetdisc/src/libibnetdisc.map | 24 + 15 files changed, 3620 insertions(+), 0 deletions(-) create mode 100644 libibnetdisc/AUTHORS create mode 100644 libibnetdisc/COPYING create mode 100644 libibnetdisc/ChangeLog create mode 100644 libibnetdisc/Makefile.am create mode 100755 libibnetdisc/autogen.sh create mode 100644 libibnetdisc/configure.in create mode 100644 libibnetdisc/include/infiniband/ibnetdisc.h create mode 100644 libibnetdisc/libibnetdisc.spec.in create mode 100644 libibnetdisc/libibnetdisc.ver create mode 100644 libibnetdisc/src/chassis.c create mode 100644 libibnetdisc/src/chassis.h create mode 100644 libibnetdisc/src/iblinkinfotest.c create mode 100644 libibnetdisc/src/ibnetdisc.c create mode 100644 libibnetdisc/src/ibnetdisctest.c create mode 100644 libibnetdisc/src/libibnetdisc.map diff --git a/libibnetdisc/AUTHORS b/libibnetdisc/AUTHORS new file mode 100644 index 0000000..d7211f9 --- /dev/null +++ b/libibnetdisc/AUTHORS @@ -0,0 +1 @@ +Ira Weiny diff --git a/libibnetdisc/COPYING b/libibnetdisc/COPYING new file mode 100644 index 0000000..a017728 --- /dev/null +++ b/libibnetdisc/COPYING @@ -0,0 +1,384 @@ +This software with the exception of OpenSM is available to you +under a choice of one of two licenses. You may chose to be +licensed under the terms of the the OpenIB.org BSD license or +the GNU General Public License (GPL) Version 2, both included +below. + +OpenSM is licensed under either GNU General Public License (GPL) +Version 2, or Intel BSD + Patent license. See OpenSM for the +specific language for the latter licensing terms. + + +Copyright (c) 2004, 2005 Voltaire, Inc. All rights reserved. + +================================================================== + + OpenIB.org BSD license + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +================================================================== + + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. diff --git a/libibnetdisc/ChangeLog b/libibnetdisc/ChangeLog new file mode 100644 index 0000000..d74037e --- /dev/null +++ b/libibnetdisc/ChangeLog @@ -0,0 +1,4 @@ + +2008-04-09 Ira Weiny + + * Added to git tree diff --git a/libibnetdisc/Makefile.am b/libibnetdisc/Makefile.am new file mode 100644 index 0000000..0f1b683 --- /dev/null +++ b/libibnetdisc/Makefile.am @@ -0,0 +1,54 @@ + +SUBDIRS = . + +INCLUDES = -I$(srcdir)/include -I$(includedir) -I$(includedir)/infiniband + +lib_LTLIBRARIES = libibnetdisc.la +sbin_PROGRAMS = + +if ENABLE_TEST_UTILS +sbin_PROGRAMS += src/ibnetdisctest \ + src/iblinkinfotest +endif + +DBGFLAGS = -g + +if HAVE_LD_VERSION_SCRIPT +libibnetdisc_version_script = -Wl,--version-script=$(srcdir)/src/libibnetdisc.map +else +libibnetdisc_version_script = +endif + +libibnetdisc_la_SOURCES = src/ibnetdisc.c src/chassis.c +libibnetdisc_la_CFLAGS = -Wall $(DBGFLAGS) +libibnetdisc_la_LDFLAGS = -version-info $(ibnetdisc_api_version) \ + -export-dynamic $(libibnetdisc_version_script) \ + -losmcomp +libibnetdisc_la_DEPENDENCIES = $(srcdir)/src/libibnetdisc.map + +libibnetdiscincludedir = $(includedir)/infiniband + +src_ibnetdisctest_SOURCES = src/ibnetdisctest.c +src_ibnetdisctest_CFLAGS = -Wall $(DBGFLAGS) +src_ibnetdisctest_LDFLAGS = -Wl,--rpath -Wl,$(libdir) \ + -libcommon -libmad -libnetdisc -losmcomp + +src_iblinkinfotest_SOURCES = src/iblinkinfotest.c +src_iblinkinfotest_CFLAGS = -Wall $(DBGFLAGS) +src_iblinkinfotest_LDFLAGS = -Wl,--rpath -Wl,$(libdir) \ + -libcommon -libmad -libnetdisc -losmcomp + +libibnetdiscinclude_HEADERS = $(srcdir)/include/infiniband/ibnetdisc.h + +EXTRA_DIST = $(srcdir)/include/infiniband/common.h \ + libibnetdisc.spec.in libibnetdisc.spec \ + $(srcdir)/src/libibnetdisc.map libibnetdisc.ver autogen.sh + +dist-hook: + if [ -x $(top_srcdir)/../gen_chlog.sh ] ; then \ + $(top_srcdir)/../gen_chlog.sh $(PACKAGE) > $(distdir)/ChangeLog ; \ + fi + if [ -x $(top_srcdir)/../gen_ver.sh ] ; then \ + ver=`$(top_srcdir)/../gen_ver.sh $(PACKAGE)` ; \ + sed -e '/AC_INIT/s/$(PACKAGE), .*,/$(PACKAGE), '$$ver',/' $(top_srcdir)/configure.in > $(distdir)/configure.in ; \ + fi diff --git a/libibnetdisc/autogen.sh b/libibnetdisc/autogen.sh new file mode 100755 index 0000000..4827884 --- /dev/null +++ b/libibnetdisc/autogen.sh @@ -0,0 +1,11 @@ +#! /bin/sh + +# create config dir if not exist +test -d config || mkdir config + +set -x +aclocal -I config +libtoolize --force --copy +autoheader +automake --foreign --add-missing --copy +autoconf diff --git a/libibnetdisc/configure.in b/libibnetdisc/configure.in new file mode 100644 index 0000000..2c17bea --- /dev/null +++ b/libibnetdisc/configure.in @@ -0,0 +1,64 @@ +dnl Process this file with autoconf to produce a configure script. + +AC_PREREQ(2.57) +AC_INIT(libibnetdisc, 1.0.0, general at lists.openfabrics.org) +dnl AC_CONFIG_SRCDIR([src/stack.c]) +AC_CONFIG_AUX_DIR(config) +AM_CONFIG_HEADER(config.h) +AM_INIT_AUTOMAKE + +AC_SUBST(RELEASE, ${RELEASE:-unknown}) +AC_SUBST(TARBALL, ${TARBALL:-${PACKAGE}-${VERSION}.tar.gz}) + +dnl the library version info is available in the file: libibnetdisc.ver +ibnetdisc_api_version=`grep LIBVERSION $srcdir/libibnetdisc.ver | sed 's/LIBVERSION=//'` +if test -z $ibnetdisc_api_version; then + ibnetdisc_api_version=1:0:0 +fi +AC_SUBST(ibnetdisc_api_version) + +dnl Checks for programs +AC_PROG_CC +AC_PROG_CPP +AC_PROG_INSTALL +AC_PROG_LN_S +AC_PROG_MAKE_SET +AM_PROG_LIBTOOL + +dnl Checks for header files. +AC_HEADER_STDC +AC_CHECK_HEADERS([stdint.h stdlib.h string.h syslog.h unistd.h]) + +dnl Checks for library functions +AC_TYPE_SIGNAL +AC_FUNC_VPRINTF +AC_CHECK_FUNCS([strrchr strtoul strtoull]) + +dnl Checks for typedefs, structures, and compiler characteristics. +AC_C_CONST +AC_C_INLINE +AC_STRUCT_TM + +AC_CACHE_CHECK(whether ld accepts --version-script, ac_cv_version_script, + if test -n "`$LD --help < /dev/null 2>/dev/null | grep version-script`"; then + ac_cv_version_script=yes + else + ac_cv_version_script=no + fi) + +AM_CONDITIONAL(HAVE_LD_VERSION_SCRIPT, test "$ac_cv_version_script" = "yes") + +dnl Check if we should include test utilities +AC_MSG_CHECKING(for --enable-test-utils) +AC_ARG_ENABLE(test-utils, +[ --enable-test-utils build additional test utilities], +[case "${enableval}" in + yes) tutils=yes ;; + no) tutils=no ;; + *) AC_MSG_ERROR(bad value ${enableval} for --enable-test-utils) ;; +esac],[tutils=no]) +AM_CONDITIONAL(ENABLE_TEST_UTILS, test x$tutils = xyes) +AC_MSG_RESULT(${tutils=no}) + +AC_CONFIG_FILES([Makefile libibnetdisc.spec]) +AC_OUTPUT diff --git a/libibnetdisc/include/infiniband/ibnetdisc.h b/libibnetdisc/include/infiniband/ibnetdisc.h new file mode 100644 index 0000000..53f0cc5 --- /dev/null +++ b/libibnetdisc/include/infiniband/ibnetdisc.h @@ -0,0 +1,294 @@ +/* + * Copyright (c) 2008 Lawrence Livermore National Lab. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#ifndef _IBNETDISC_H_ +#define _IBNETDISC_H_ + +#include + +#define MAXHOPS 63 + +/* HASH table defines */ +#define HASHGUID(guid) ((uint32_t)(((uint32_t)(guid) * 101) ^ ((uint32_t)((guid) >> 32) * 103))) +#define HTSZ 137 + +typedef enum { + IBND_CA_NODE = 1, + IBND_SWITCH_NODE = 2, + IBND_ROUTER_NODE = 3 +} ibnd_node_type_t; + +typedef enum { + IBND_LINK_DOWN = 1, + IBND_LINK_INIT = 2, + IBND_LINK_ARMED = 3, + IBND_LINK_ACTIVE = 4 +} ibnd_link_state_t; + +/** ========================================================================= + * Node + */ +typedef struct switch_info { + int smaenhsp0; +} ibnd_switch_info_t; + +typedef struct node_info { + int base_ver; + int class_ver; + int type; + int numports; + uint64_t sysimgguid; + uint64_t nodeguid; + uint64_t nodeportguid; + uint16_t partition_cap; + uint32_t devid; + uint32_t revision; + int localport; + uint32_t vendid; +} ibnd_node_info_t; + +struct port; +struct ib_fabric; +struct chassis_record; +typedef struct node { + struct node *next; /* all node list in fabric */ + struct node *htnext; /* store node in guid hash table */ + struct node *dnext; /* store node in nodesdist table */ + struct node *type_next; /* store node in "type" list (ca|switch|router) */ + + struct ib_fabric *fabric; /* the fabric node belongs to */ + + ib_portid_t path_portid; /* path from "from_node" */ + int dist; /* num of hops from "from_node" */ + + int smalid; + int smalmc; + ibnd_switch_info_t sw_info; + ibnd_node_info_t info; + + char nodedesc[64]; + + struct port *ports; /* NULL term link list of ports */ + struct port **portary; /* 1 based, in order array of port pointers above */ + /* MAY BE NULL! */ + + /* chassis info */ + struct node *chassis_next; /* store node in "chassis" list */ + struct chassis_record *chrecord; + + void *user_data; /* users can store data here */ +} ibnd_node_t; + +/** ========================================================================= + * Port + */ +typedef struct port_info { + int lid; + int smlid; + int link_speed_supported; + int link_speed_enabled; + int link_speed_active; + int link_state; + int phys_state; + int link_down_def_state; + int mkey_prot_bits; + int lmc; + int neighbor_mtu; + int smsl; + int init_type; + int vl_capability; + int vl_high_limit; + int vl_arb_high_cap; + int vl_arb_low_cap; + int init_reply; + int mtu_cap; + int vl_stall_count; + int hoq_lifetime; + int oper_vls; + int partition_enforce_in; + int partition_enforce_out; + int filter_raw_in; + int filter_raw_out; + int mkey_violations; + int pkey_violations; + int qkey_violations; + int guid_capabilities; + int client_rereg; + int subnet_timeout; + int response_time_val; + int local_phys_error; + int overrun_error; + int max_credit_hint; + uint32_t link_round_trip; + int local_port; + int link_width_supported; + int link_width_enabled; + int link_width_active; + int diag_code; + int mkey_lease; + uint32_t capability_mask; + uint64_t mkey; + uint64_t gid_prefix; +} ibnd_port_info_t; + +typedef struct port { + struct port *next; + uint64_t guid; + int portnum; + int ext_portnum; /* optional (!= 0) external port num */ + ibnd_node_t *node; + struct port *remoteport; /* null if SMA, or does not exist */ + ibnd_port_info_t info; + void *user_data; /* users can store data here */ +} ibnd_port_t; + + +/** ========================================================================= + * Chassis data + */ +typedef struct chassis_record { + struct chassis_record *next; + unsigned char chassisnum; + unsigned char chassistype; + unsigned char anafanum; + unsigned char slotnum; + unsigned char chassisslot; +} ibnd_chassis_record_t; + +#define SPINES_MAX_NUM 12 +#define LINES_MAX_NUM 36 + +typedef struct chassis_list { + struct chassis_list *next; + uint64_t chassisguid; + int chassisnum; + int chassistype; + + /* generic grouping by SystemImageGUID */ + int nodecount; + ibnd_node_t *nodes; + + /* specific to voltaire type nodes */ + ibnd_node_t *spinenode[SPINES_MAX_NUM + 1]; + ibnd_node_t *linenode[LINES_MAX_NUM + 1]; +} ibnd_chassis_list_t; + +/** ========================================================================= + * Fabric + * Main fabric object which is returned and represents the data discovered + */ +typedef struct ib_fabric { + /* the node which you requested to start on + * "from" parameter in ibnd_discover. + */ + ibnd_node_t *from_node; + + /* list of all nodes in the system */ + ibnd_node_t *nodes; + + /* NULL terminated lists of node types */ + ibnd_node_t *switches; + ibnd_node_t *ch_adapters; + ibnd_node_t *routers; + + /* list of all chassis found in the fabric */ + ibnd_chassis_list_t *chassis; + + /* the following are for internal use */ + ibnd_node_t *nodestbl[HTSZ]; + int maxhops_discovered; + ibnd_node_t *nodesdist[MAXHOPS+1]; + ibnd_chassis_list_t *first_chassis; + ibnd_chassis_list_t *current_chassis; +} ibnd_fabric_t; + + +/** ========================================================================= + * Initialization + */ +int ibnd_init(char *dev_name, int dev_port); +void ibnd_debug(int i); + +/** ========================================================================= + * discover (allocates fabric data) and free + */ +ibnd_fabric_t *ibnd_discover(int timeout_ms, ib_portid_t *from, int hops); + /** + * timeout_ms: (required) and gives the timeout for a _single_ query on + * the fabric. So if there are mutiple nodes not + * responding this may result in a lengthy delay. + * from: (optional) specify the node to start scanning from. + * If NULL start from the node we are running on. + * hops: (optional) Specify how much of the fabric to traverse. + * negative value == scan entire fabric + */ +void ibnd_destroy_fabric(ibnd_fabric_t *fabric); + +/** ========================================================================= + * Node operations + */ +typedef void (*ibnd_iter_func_t)(ibnd_node_t *node, void *user_data); + +ibnd_node_t *ibnd_find_node_guid(ibnd_fabric_t *fabric, uint64_t guid); +ibnd_node_t *ibnd_find_node_dr(ibnd_fabric_t *fabric, char *dr_str); +ibnd_node_t *ibnd_update_node(ibnd_node_t *node); + +void ibnd_iter_nodes(ibnd_fabric_t *fabric, + ibnd_iter_func_t func, + void *user_data); +void ibnd_iter_nodes_type(ibnd_fabric_t *fabric, + ibnd_iter_func_t func, + ibnd_node_type_t node_type, + void *user_data); + +/** ========================================================================= + * Str convert functions + */ +char *ibnd_linkwidth_str(int link_width); +char *ibnd_linkspeed_str(int link_speed); +char *ibnd_linkstate_str(int link_state); +char *ibnd_physstate_str(int phys_state); +const char *ibnd_node_type_str(ibnd_node_t *node); + +/** ========================================================================= + * Chassis queries + */ +uint64_t ibnd_get_chassis_guid(ibnd_fabric_t *fabric, unsigned char chassisnum); +char *ibnd_get_chassis_type(ibnd_node_t *node); +char *ibnd_get_chassis_slot_str(ibnd_node_t *node, char *str, size_t size); + +int ibnd_is_xsigo_guid(uint64_t guid); +int ibnd_is_xsigo_tca(uint64_t guid); +int ibnd_is_xsigo_hca(uint64_t guid); + +#endif /* _IBNETDISC_H_ */ diff --git a/libibnetdisc/libibnetdisc.spec.in b/libibnetdisc/libibnetdisc.spec.in new file mode 100644 index 0000000..5fe55c4 --- /dev/null +++ b/libibnetdisc/libibnetdisc.spec.in @@ -0,0 +1,71 @@ + +%define RELEASE @RELEASE@ +%define rel %{?CUSTOM_RELEASE} %{!?CUSTOM_RELEASE:%RELEASE} + +Summary: OpenFabrics Alliance InfiniBand MAD library +Name: libibnetdisc +Version: @VERSION@ +Release: %rel%{?dist} +License: GPLv2 or BSD +Group: System Environment/Libraries +BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) +Source: http://www.openfabrics.org/downloads/management/@TARBALL@ +Url: http://openfabrics.org/ +BuildRequires: libibunetdisc-devel, libtool +Requires(post): /sbin/ldconfig +Requires(postun): /sbin/ldconfig + +%description +libibnetdisc provides a higer level C interface to scaning an IB fabric. + +%package devel +Summary: Development files for the libibnetdisc library +Group: System Environment/Libraries +Requires: %{name} = %{version}-%{release} libibunetdisc-devel +Requires(post): /sbin/ldconfig +Requires(postun): /sbin/ldconfig + +%description devel +Development files for the libibnetdisc library. + +%package static +Summary: Static version of the libibnetdisc library +Group: System Environment/Libraries +Requires: %{name} = %{version}-%{release} + +%description static +Static version of the libibnetdisc library + +%prep +%setup -q + +%build +%configure +make %{?_smp_mflags} + +%install +make DESTDIR=${RPM_BUILD_ROOT} install +# remove unpackaged files from the buildroot +rm -f $RPM_BUILD_ROOT%{_libdir}/*.la + +%clean +rm -rf $RPM_BUILD_ROOT + +%post -p /sbin/ldconfig +%postun -p /sbin/ldconfig +%post devel -p /sbin/ldconfig +%postun devel -p /sbin/ldconfig + +%files +%defattr(-,root,root) +%{_libdir}/libibnetdisc*.so.* +%doc AUTHORS COPYING ChangeLog + +%files devel +%defattr(-,root,root) +%{_libdir}/libibnetdisc.so +%{_includedir}/infiniband/*.h + +%files static +%defattr(-,root,root) +%{_libdir}/libibnetdisc.a diff --git a/libibnetdisc/libibnetdisc.ver b/libibnetdisc/libibnetdisc.ver new file mode 100644 index 0000000..a0a5f3c --- /dev/null +++ b/libibnetdisc/libibnetdisc.ver @@ -0,0 +1,9 @@ +# In this file we track the current API version +# of the IB net discover interface (and libraries) +# The version is built of the following +# tree numbers: +# API_REV:RUNNING_REV:AGE +# API_REV - advance on any added API +# RUNNING_REV - advance any change to the vendor files +# AGE - number of backward versions the API still supports +LIBVERSION=1:0:0 diff --git a/libibnetdisc/src/chassis.c b/libibnetdisc/src/chassis.c new file mode 100644 index 0000000..6b79c49 --- /dev/null +++ b/libibnetdisc/src/chassis.c @@ -0,0 +1,806 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2008 Lawrence Livermore National Lab. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +/*========================================================*/ +/* FABRIC SCANNER SPECIFIC DATA */ +/*========================================================*/ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include + +#include +#include + +#include "chassis.h" + +static char *ChassisTypeStr[5] = { "", "ISR9288", "ISR9096", "ISR2012", "ISR2004" }; +static char *ChassisSlotTypeStr[4] = { "", "Line", "Spine", "SRBD" }; + +char *ibnd_get_chassis_type(ibnd_node_t *node) +{ + /* Currently, only if Voltaire chassis */ + if (node->info.vendid != VTR_VENDOR_ID) + return (NULL); + if (!node->chrecord) + return (NULL); + if (node->chrecord->chassistype == UNRESOLVED_CT + || node->chrecord->chassistype > ISR2004_CT) + return (NULL); + return ChassisTypeStr[node->chrecord->chassistype]; +} + +char *ibnd_get_chassis_slot_str(ibnd_node_t *node, char *str, size_t size) +{ + /* Currently, only if Voltaire chassis */ + if (node->info.vendid != VTR_VENDOR_ID) + return (NULL); + if (!node->chrecord) + return (NULL); + if (node->chrecord->chassisslot == UNRESOLVED_CS + || node->chrecord->chassisslot > SRBD_CS) + return (NULL); + if (!str) + return (NULL); + snprintf(str, size, "%s %d Chip %d", + ChassisSlotTypeStr[node->chrecord->chassisslot], + node->chrecord->slotnum, + node->chrecord->anafanum); + return (str); +} + +static ibnd_chassis_list_t *find_chassisnum(ibnd_fabric_t *fabric, unsigned char chassisnum) +{ + ibnd_chassis_list_t *current; + + for (current = fabric->first_chassis; current; current = current->next) { + if (current->chassisnum == chassisnum) + return current; + } + + return NULL; +} + +static uint64_t topspin_chassisguid(uint64_t guid) +{ + /* Byte 3 in system image GUID is chassis type, and */ + /* Byte 4 is location ID (slot) so just mask off byte 4 */ + return guid & 0xffffffff00ffffffULL; +} + +int ibnd_is_xsigo_guid(uint64_t guid) +{ + if ((guid & 0xffffff0000000000ULL) == 0x0013970000000000ULL) + return 1; + else + return 0; +} + +static int is_xsigo_leafone(uint64_t guid) +{ + if ((guid & 0xffffffffff000000ULL) == 0x0013970102000000ULL) + return 1; + else + return 0; +} + +int ibnd_is_xsigo_hca(uint64_t guid) +{ + /* NodeType 2 is HCA */ + if ((guid & 0xffffffff00000000ULL) == 0x0013970200000000ULL) + return 1; + else + return 0; +} + +int ibnd_is_xsigo_tca(uint64_t guid) +{ + /* NodeType 3 is TCA */ + if ((guid & 0xffffffff00000000ULL) == 0x0013970300000000ULL) + return 1; + else + return 0; +} + +static int is_xsigo_ca(uint64_t guid) +{ + if (ibnd_is_xsigo_hca(guid) || ibnd_is_xsigo_tca(guid)) + return 1; + else + return 0; +} + +static int is_xsigo_switch(uint64_t guid) +{ + if ((guid & 0xffffffff00000000ULL) == 0x0013970100000000ULL) + return 1; + else + return 0; +} + +static uint64_t xsigo_chassisguid(ibnd_node_t *node) +{ + if (!is_xsigo_ca(node->info.sysimgguid)) { + /* Byte 3 is NodeType and byte 4 is PortType */ + /* If NodeType is 1 (switch), PortType is masked */ + if (is_xsigo_switch(node->info.sysimgguid)) + return node->info.sysimgguid & 0xffffffff00ffffffULL; + else + return node->info.sysimgguid; + } else { + /* Is there a peer port ? */ + if (!node->ports->remoteport) + return node->info.sysimgguid; + + /* If peer port is Leaf 1, use its chassis GUID */ + if (is_xsigo_leafone(node->ports->remoteport->node->info.sysimgguid)) + return node->ports->remoteport->node->info.sysimgguid & + 0xffffffff00ffffffULL; + else + return node->info.sysimgguid; + } +} + +static uint64_t get_chassisguid(ibnd_node_t *node) +{ + if (node->info.vendid == TS_VENDOR_ID || node->info.vendid == SS_VENDOR_ID) + return topspin_chassisguid(node->info.sysimgguid); + else if (node->info.vendid == XS_VENDOR_ID || ibnd_is_xsigo_guid(node->info.sysimgguid)) + return xsigo_chassisguid(node); + else + return node->info.sysimgguid; +} + +static ibnd_chassis_list_t *find_chassisguid(ibnd_node_t *node) +{ + ibnd_chassis_list_t *current; + uint64_t chguid; + + chguid = get_chassisguid(node); + for (current = node->fabric->first_chassis; current; current = current->next) { + if (current->chassisguid == chguid) + return current; + } + + return NULL; +} + +uint64_t ibnd_get_chassis_guid(ibnd_fabric_t *fabric, unsigned char chassisnum) +{ + ibnd_chassis_list_t *chassis; + + chassis = find_chassisnum(fabric, chassisnum); + if (chassis) + return chassis->chassisguid; + else + return 0; +} + +static int is_router(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_IB_FC_ROUTER || + node->info.devid == VTR_DEVID_IB_IP_ROUTER); +} + +static int is_spine_9096(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SFB4 || + node->info.devid == VTR_DEVID_SFB4_DDR); +} + +static int is_spine_9288(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SFB12 || + node->info.devid == VTR_DEVID_SFB12_DDR); +} + +static int is_spine_2004(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SFB2004); +} + +static int is_spine_2012(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SFB2012); +} + +static int is_spine(ibnd_node_t *node) +{ + return (is_spine_9096(node) || is_spine_9288(node) || + is_spine_2004(node) || is_spine_2012(node)); +} + +static int is_line_24(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SLB24 || + node->info.devid == VTR_DEVID_SLB24_DDR); +} + +static int is_line_8(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SLB8); +} + +static int is_line_2024(ibnd_node_t *node) +{ + return (node->info.devid == VTR_DEVID_SLB2024); +} + +static int is_line(ibnd_node_t *node) +{ + return (is_line_24(node) || is_line_8(node) || is_line_2024(node)); +} + +int is_chassis_switch(ibnd_node_t *node) +{ + return (is_spine(node) || is_line(node)); +} + +/* these structs help find Line (Anafa) slot number while using spine portnum */ +int line_slot_2_sfb4[25] = { 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4 }; +int anafa_line_slot_2_sfb4[25] = { 0, 1, 1, 1, 2, 2, 2, 1, 1, 1, 2, 2, 2, 1, 1, 1, 2, 2, 2, 1, 1, 1, 2, 2, 2 }; +int line_slot_2_sfb12[25] = { 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9,10, 10, 11, 11, 12, 12 }; +int anafa_line_slot_2_sfb12[25] = { 0, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2 }; + +/* IPR FCR modules connectivity while using sFB4 port as reference */ +int ipr_slot_2_sfb4_port[25] = { 0, 3, 2, 1, 3, 2, 1, 3, 2, 1, 3, 2, 1, 3, 2, 1, 3, 2, 1, 3, 2, 1, 3, 2, 1 }; + +/* these structs help find Spine (Anafa) slot number while using spine portnum */ +int spine12_slot_2_slb[25] = { 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }; +int anafa_spine12_slot_2_slb[25]= { 0, 1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }; +int spine4_slot_2_slb[25] = { 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }; +int anafa_spine4_slot_2_slb[25] = { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }; +/* reference { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 }; */ + +static void get_sfb_slot(ibnd_node_t *node, ibnd_port_t *lineport) +{ + ibnd_chassis_record_t *ch = node->chrecord; + + ch->chassisslot = SPINE_CS; + if (is_spine_9096(node)) { + ch->chassistype = ISR9096_CT; + ch->slotnum = spine4_slot_2_slb[lineport->portnum]; + ch->anafanum = anafa_spine4_slot_2_slb[lineport->portnum]; + } else if (is_spine_9288(node)) { + ch->chassistype = ISR9288_CT; + ch->slotnum = spine12_slot_2_slb[lineport->portnum]; + ch->anafanum = anafa_spine12_slot_2_slb[lineport->portnum]; + } else if (is_spine_2012(node)) { + ch->chassistype = ISR2012_CT; + ch->slotnum = spine12_slot_2_slb[lineport->portnum]; + ch->anafanum = anafa_spine12_slot_2_slb[lineport->portnum]; + } else if (is_spine_2004(node)) { + ch->chassistype = ISR2004_CT; + ch->slotnum = spine4_slot_2_slb[lineport->portnum]; + ch->anafanum = anafa_spine4_slot_2_slb[lineport->portnum]; + } else { + IBPANIC("Unexpected node found: guid 0x%016" PRIx64, + node->info.nodeguid); + } +} + +static void get_router_slot(ibnd_node_t *node, ibnd_port_t *spineport) +{ + ibnd_chassis_record_t *ch = node->chrecord; + int guessnum = 0; + + if (!ch) { + if (!(node->chrecord = calloc(1, sizeof(ibnd_chassis_record_t)))) + IBPANIC("out of mem"); + ch = node->chrecord; + } + + ch->chassisslot = SRBD_CS; + if (is_spine_9096(spineport->node)) { + ch->chassistype = ISR9096_CT; + ch->slotnum = line_slot_2_sfb4[spineport->portnum]; + ch->anafanum = ipr_slot_2_sfb4_port[spineport->portnum]; + } else if (is_spine_9288(spineport->node)) { + ch->chassistype = ISR9288_CT; + ch->slotnum = line_slot_2_sfb12[spineport->portnum]; + /* this is a smart guess based on nodeguids order on sFB-12 module */ + guessnum = spineport->node->info.nodeguid % 4; + /* module 1 <--> remote anafa 3 */ + /* module 2 <--> remote anafa 2 */ + /* module 3 <--> remote anafa 1 */ + ch->anafanum = (guessnum == 3 ? 1 : (guessnum == 1 ? 3 : 2)); + } else if (is_spine_2012(spineport->node)) { + ch->chassistype = ISR2012_CT; + ch->slotnum = line_slot_2_sfb12[spineport->portnum]; + /* this is a smart guess based on nodeguids order on sFB-12 module */ + guessnum = spineport->node->info.nodeguid % 4; + // module 1 <--> remote anafa 3 + // module 2 <--> remote anafa 2 + // module 3 <--> remote anafa 1 + ch->anafanum = (guessnum == 3? 1 : (guessnum == 1 ? 3 : 2)); + } else if (is_spine_2004(spineport->node)) { + ch->chassistype = ISR2004_CT; + ch->slotnum = line_slot_2_sfb4[spineport->portnum]; + ch->anafanum = ipr_slot_2_sfb4_port[spineport->portnum]; + } else { + IBPANIC("Unexpected node found: guid 0x%016" PRIx64, + spineport->node->info.nodeguid); + } +} + +static void get_slb_slot(ibnd_chassis_record_t *ch, ibnd_port_t *spineport) +{ + ch->chassisslot = LINE_CS; + if (is_spine_9096(spineport->node)) { + ch->chassistype = ISR9096_CT; + ch->slotnum = line_slot_2_sfb4[spineport->portnum]; + ch->anafanum = anafa_line_slot_2_sfb4[spineport->portnum]; + } else if (is_spine_9288(spineport->node)) { + ch->chassistype = ISR9288_CT; + ch->slotnum = line_slot_2_sfb12[spineport->portnum]; + ch->anafanum = anafa_line_slot_2_sfb12[spineport->portnum]; + } else if (is_spine_2012(spineport->node)) { + ch->chassistype = ISR2012_CT; + ch->slotnum = line_slot_2_sfb12[spineport->portnum]; + ch->anafanum = anafa_line_slot_2_sfb12[spineport->portnum]; + } else if (is_spine_2004(spineport->node)) { + ch->chassistype = ISR2004_CT; + ch->slotnum = line_slot_2_sfb4[spineport->portnum]; + ch->anafanum = anafa_line_slot_2_sfb4[spineport->portnum]; + } else { + IBPANIC("Unexpected node found: guid 0x%016" PRIx64, + spineport->node->info.nodeguid); + } +} + +/* forward declare this */ +static void voltaire_portmap(ibnd_port_t *port); +/* + This function called for every Voltaire node in fabric + It could be optimized so, but time overhead is very small + and its only diag.util +*/ +static void fill_voltaire_chassis_record(ibnd_node_t *node) +{ + ibnd_port_t *port; + ibnd_node_t *remnode = 0; + ibnd_chassis_record_t *ch = 0; + + if (node->chrecord) /* somehow this node has already been passed */ + return; + + if (!(node->chrecord = calloc(1, sizeof(ibnd_chassis_record_t)))) + IBPANIC("out of mem"); + + ch = node->chrecord; + + /* node is router only in case of using unique lid */ + /* (which is lid of chassis router port) */ + /* in such case node->ports is actually a requested port... */ + if (is_router(node) && is_spine(node->ports->remoteport->node)) + get_router_slot(node, node->ports->remoteport); + else if (is_spine(node)) { + for (port = node->ports; port; port = port->next) { + if (!port->remoteport) + continue; + remnode = port->remoteport->node; + if (remnode->info.type != IBND_SWITCH_NODE) { + if (!remnode->chrecord) + get_router_slot(remnode, port); + continue; + } + if (!ch->chassistype) + /* we assume here that remoteport belongs to line */ + get_sfb_slot(node, port->remoteport); + + /* we could break here, but need to find if more routers connected */ + } + + } else if (is_line(node)) { + for (port = node->ports; port; port = port->next) { + if (port->portnum > 12) + continue; + if (!port->remoteport) + continue; + /* we assume here that remoteport belongs to spine */ + get_slb_slot(ch, port->remoteport); + break; + } + } + + /* for each port of this node, map external ports */ + for (port = node->ports; port; port = port->next) { + voltaire_portmap(port); + } + + return; +} + +static int get_line_index(ibnd_node_t *node) +{ + int retval = 3 * (node->chrecord->slotnum - 1) + node->chrecord->anafanum; + + if (retval > LINES_MAX_NUM || retval < 1) + IBPANIC("Internal error"); + return retval; +} + +static int get_spine_index(ibnd_node_t *node) +{ + int retval; + + if (is_spine_9288(node) || is_spine_2012(node)) + retval = 3 * (node->chrecord->slotnum - 1) + node->chrecord->anafanum; + else + retval = node->chrecord->slotnum; + + if (retval > SPINES_MAX_NUM || retval < 1) + IBPANIC("Internal error"); + return retval; +} + +static void insert_line_router(ibnd_node_t *node, ibnd_chassis_list_t *chassislist) +{ + int i = get_line_index(node); + + if (chassislist->linenode[i]) + return; /* already filled slot */ + + chassislist->linenode[i] = node; + node->chrecord->chassisnum = chassislist->chassisnum; +} + +static void insert_spine(ibnd_node_t *node, ibnd_chassis_list_t *chassislist) +{ + int i = get_spine_index(node); + + if (chassislist->spinenode[i]) + return; /* already filled slot */ + + chassislist->spinenode[i] = node; + node->chrecord->chassisnum = chassislist->chassisnum; +} + +static void pass_on_lines_catch_spines(ibnd_chassis_list_t *chassislist) +{ + ibnd_node_t *node, *remnode; + ibnd_port_t *port; + int i; + + for (i = 1; i <= LINES_MAX_NUM; i++) { + node = chassislist->linenode[i]; + + if (!(node && is_line(node))) + continue; /* empty slot or router */ + + for (port = node->ports; port; port = port->next) { + if (port->portnum > 12) + continue; + + if (!port->remoteport) + continue; + remnode = port->remoteport->node; + + if (!remnode->chrecord) + continue; /* some error - spine not initialized ? FIXME */ + insert_spine(remnode, chassislist); + } + } +} + +static void pass_on_spines_catch_lines(ibnd_chassis_list_t *chassislist) +{ + ibnd_node_t *node, *remnode; + ibnd_port_t *port; + int i; + + for (i = 1; i <= SPINES_MAX_NUM; i++) { + node = chassislist->spinenode[i]; + if (!node) + continue; /* empty slot */ + for (port = node->ports; port; port = port->next) { + if (!port->remoteport) + continue; + remnode = port->remoteport->node; + + if (!remnode->chrecord) + continue; /* some error - line/router not initialized ? FIXME */ + insert_line_router(remnode, chassislist); + } + } +} + +/* + Stupid interpolation algorithm... + But nothing to do - have to be compliant with VoltaireSM/NMS +*/ +static void pass_on_spines_interpolate_chguid(ibnd_chassis_list_t *chassislist) +{ + ibnd_node_t *node; + int i; + + for (i = 1; i <= SPINES_MAX_NUM; i++) { + node = chassislist->spinenode[i]; + if (!node) + continue; /* skip the empty slots */ + + /* take first guid minus one to be consistent with SM */ + chassislist->chassisguid = node->info.nodeguid - 1; + break; + } +} + +/* + This function fills chassislist structure with all nodes + in that chassis + chassislist structure = structure of one standalone chassis +*/ +static void build_chassis(ibnd_node_t *node, ibnd_chassis_list_t *chassislist) +{ + ibnd_node_t *remnode = 0; + ibnd_port_t *port = 0; + + /* we get here with node = chassis_spine */ + chassislist->chassistype = node->chrecord->chassistype; + insert_spine(node, chassislist); + + /* loop: pass on all ports of node */ + for (port = node->ports; port; port = port->next) { + if (!port->remoteport) + continue; + remnode = port->remoteport->node; + + if (!remnode->chrecord) + continue; /* some error - line or router not initialized ? FIXME */ + + insert_line_router(remnode, chassislist); + } + + pass_on_lines_catch_spines(chassislist); + /* this pass needed for to catch routers, since routers connected only */ + /* to spines in slot 1 or 4 and we could miss them first time */ + pass_on_spines_catch_lines(chassislist); + + /* additional 2 passes needed for to overcome a problem of pure "in-chassis" */ + /* connectivity - extra pass to ensure that all related chips/modules */ + /* inserted into the chassislist */ + pass_on_lines_catch_spines(chassislist); + pass_on_spines_catch_lines(chassislist); + pass_on_spines_interpolate_chguid(chassislist); +} + +/*========================================================*/ +/* INTERNAL TO EXTERNAL PORT MAPPING */ +/*========================================================*/ + +/* +Description : On ISR9288/9096 external ports indexing + is not matching the internal ( anafa ) port + indexes. Use this MAP to translate the data you get from + the OpenIB diagnostics (smpquery, ibroute, ibtracert, etc.) + + +Module : sLB-24 + anafa 1 anafa 2 +ext port | 13 14 15 16 17 18 | 19 20 21 22 23 24 +int port | 22 23 24 18 17 16 | 22 23 24 18 17 16 +ext port | 1 2 3 4 5 6 | 7 8 9 10 11 12 +int port | 19 20 21 15 14 13 | 19 20 21 15 14 13 +------------------------------------------------ + +Module : sLB-8 + anafa 1 anafa 2 +ext port | 13 14 15 16 17 18 | 19 20 21 22 23 24 +int port | 24 23 22 18 17 16 | 24 23 22 18 17 16 +ext port | 1 2 3 4 5 6 | 7 8 9 10 11 12 +int port | 21 20 19 15 14 13 | 21 20 19 15 14 13 + +-----------> + anafa 1 anafa 2 +ext port | - - 5 - - 6 | - - 7 - - 8 +int port | 24 23 22 18 17 16 | 24 23 22 18 17 16 +ext port | - - 1 - - 2 | - - 3 - - 4 +int port | 21 20 19 15 14 13 | 21 20 19 15 14 13 +------------------------------------------------ + +Module : sLB-2024 + +ext port | 13 14 15 16 17 18 19 20 21 22 23 24 +A1 int port| 13 14 15 16 17 18 19 20 21 22 23 24 +ext port | 1 2 3 4 5 6 7 8 9 10 11 12 +A2 int port| 13 14 15 16 17 18 19 20 21 22 23 24 +--------------------------------------------------- + +*/ + +int int2ext_map_slb24[2][25] = { + { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 5, 4, 18, 17, 16, 1, 2, 3, 13, 14, 15 }, + { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 11, 10, 24, 23, 22, 7, 8, 9, 19, 20, 21 } + }; +int int2ext_map_slb8[2][25] = { + { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 6, 6, 6, 1, 1, 1, 5, 5, 5 }, + { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 4, 8, 8, 8, 3, 3, 3, 7, 7, 7 } + }; +int int2ext_map_slb2024[2][25] = { + { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 }, + { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 } + }; +/* reference { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 }; */ + +/* map internal ports to external ports if appropriate */ +static void +voltaire_portmap(ibnd_port_t *port) +{ + ibnd_chassis_record_t *ch = port->node->chrecord; + int portnum = port->portnum; + int chipnum = 0; + ibnd_node_t *node = port->node; + + if (!ch || !is_line(node) || (portnum < 13 || portnum > 24)) { + port->ext_portnum = 0; + return; + } + + if (ch->anafanum < 1 || ch->anafanum > 2) { + port->ext_portnum = 0; + return; + } + + chipnum = ch->anafanum - 1; + + if (is_line_24(node)) + port->ext_portnum = int2ext_map_slb24[chipnum][portnum]; + else if (is_line_2024(node)) + port->ext_portnum = int2ext_map_slb2024[chipnum][portnum]; + else + port->ext_portnum = int2ext_map_slb8[chipnum][portnum]; +} + +static void add_chassislist(ibnd_fabric_t *fabric) +{ + if (!(fabric->current_chassis = calloc(1, sizeof(ibnd_chassis_list_t)))) + IBPANIC("out of mem"); + + if (fabric->first_chassis == NULL) { + fabric->first_chassis = fabric->current_chassis; + } else { + fabric->current_chassis->next = NULL; + } +} + +static void +add_node_to_chassis(ibnd_chassis_list_t *chassis, ibnd_node_t *node) +{ + node->chassis_next = chassis->nodes; + if (chassis->nodes) + chassis->nodes->chassis_next = node; + else + chassis->nodes = node; +} + +/* + Main grouping function + Algorithm: + 1. pass on every Voltaire node + 2. catch spine chip for every Voltaire node + 2.1 build/interpolate chassis around this chip + 2.2 go to 1. + 3. pass on non Voltaire nodes (SystemImageGUID based grouping) + 4. now group non Voltaire nodes by SystemImageGUID +*/ +ibnd_chassis_list_t *group_nodes(ibnd_fabric_t *fabric) +{ + ibnd_node_t *node; + int dist; + int chassisnum = 0; + ibnd_chassis_list_t *chassis; + + fabric->first_chassis = NULL; + fabric->current_chassis = NULL; + + /* first pass on switches and build for every Voltaire node */ + /* an appropriate chassis record (slotnum and position) */ + /* according to internal connectivity */ + /* not very efficient but clear code so... */ + for (dist = 0; dist <= fabric->maxhops_discovered; dist++) { + for (node = fabric->nodesdist[dist]; node; node = node->dnext) { + if (node->info.vendid == VTR_VENDOR_ID) + fill_voltaire_chassis_record(node); + } + } + + /* separate every Voltaire chassis from each other and build linked list of them */ + /* algorithm: catch spine and find all surrounding nodes */ + for (dist = 0; dist <= fabric->maxhops_discovered; dist++) { + for (node = fabric->nodesdist[dist]; node; node = node->dnext) { + if (node->info.vendid != VTR_VENDOR_ID) + continue; + if (!node->chrecord || node->chrecord->chassisnum || !is_spine(node)) + continue; + add_chassislist(fabric); + fabric->current_chassis->chassisnum = ++chassisnum; + build_chassis(node, fabric->current_chassis); + } + } + + /* now make pass on nodes for chassis which are not Voltaire */ + /* grouped by common SystemImageGUID */ + for (dist = 0; dist <= fabric->maxhops_discovered; dist++) { + for (node = fabric->nodesdist[dist]; node; node = node->dnext) { + if (node->info.vendid == VTR_VENDOR_ID) + continue; + if (node->info.sysimgguid) { + chassis = find_chassisguid(node); + if (chassis) + chassis->nodecount++; + else { + /* Possible new chassis */ + add_chassislist(fabric); + fabric->current_chassis->chassisguid = get_chassisguid(node); + fabric->current_chassis->nodecount = 1; + } + } + } + } + + /* now, make another pass to see which nodes are part of chassis */ + /* (defined as chassis->nodecount > 1) */ + for (dist = 0; dist <= MAXHOPS; ) { + for (node = fabric->nodesdist[dist]; node; node = node->dnext) { + if (node->info.vendid == VTR_VENDOR_ID) + continue; + if (node->info.sysimgguid) { + chassis = find_chassisguid(node); + if (chassis && chassis->nodecount > 1) { + if (!chassis->chassisnum) + chassis->chassisnum = ++chassisnum; + if (!node->chrecord) { + if (!(node->chrecord = + calloc(1, + sizeof(ibnd_chassis_record_t)))) + IBPANIC("out of mem"); + node->chrecord->chassisnum = chassis->chassisnum; + add_node_to_chassis(chassis, node); + } + } + } + } + if (dist == fabric->maxhops_discovered) + dist = MAXHOPS; /* skip to CAs */ + else + dist++; + } + + return (fabric->first_chassis); +} diff --git a/libibnetdisc/src/chassis.h b/libibnetdisc/src/chassis.h new file mode 100644 index 0000000..ea271d0 --- /dev/null +++ b/libibnetdisc/src/chassis.h @@ -0,0 +1,82 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#ifndef _CHASSIS_H_ +#define _CHASSIS_H_ + +#include + +/*========================================================*/ +/* CHASSIS RECOGNITION SPECIFIC DATA */ +/*========================================================*/ + +/* Device IDs */ +#define VTR_DEVID_IB_FC_ROUTER 0x5a00 +#define VTR_DEVID_IB_IP_ROUTER 0x5a01 +#define VTR_DEVID_ISR9600_SPINE 0x5a02 +#define VTR_DEVID_ISR9600_LEAF 0x5a03 +#define VTR_DEVID_HCA1 0x5a04 +#define VTR_DEVID_HCA2 0x5a44 +#define VTR_DEVID_HCA3 0x6278 +#define VTR_DEVID_SW_6IB4 0x5a05 +#define VTR_DEVID_ISR9024 0x5a06 +#define VTR_DEVID_ISR9288 0x5a07 +#define VTR_DEVID_SLB24 0x5a09 +#define VTR_DEVID_SFB12 0x5a08 +#define VTR_DEVID_SFB4 0x5a0b +#define VTR_DEVID_ISR9024_12 0x5a0c +#define VTR_DEVID_SLB8 0x5a0d +#define VTR_DEVID_RLX_SWITCH_BLADE 0x5a20 +#define VTR_DEVID_ISR9024_DDR 0x5a31 +#define VTR_DEVID_SFB12_DDR 0x5a32 +#define VTR_DEVID_SFB4_DDR 0x5a33 +#define VTR_DEVID_SLB24_DDR 0x5a34 +#define VTR_DEVID_SFB2012 0x5a37 +#define VTR_DEVID_SLB2024 0x5a38 +#define VTR_DEVID_ISR2012 0x5a39 +#define VTR_DEVID_SFB2004 0x5a40 +#define VTR_DEVID_ISR2004 0x5a41 + +/* Vendor IDs (for chassis based systems) */ +#define VTR_VENDOR_ID 0x8f1 /* Voltaire */ +#define TS_VENDOR_ID 0x5ad /* Cisco */ +#define SS_VENDOR_ID 0x66a /* InfiniCon */ +#define XS_VENDOR_ID 0x1397 /* Xsigo */ + +enum ibnd_chassis_type { UNRESOLVED_CT, ISR9288_CT, ISR9096_CT, ISR2012_CT, ISR2004_CT }; +enum ibnd_chassis_slot_type { UNRESOLVED_CS, LINE_CS, SPINE_CS, SRBD_CS }; + +ibnd_chassis_list_t *group_nodes(ibnd_fabric_t *fabric); + +#endif /* _CHASSIS_H_ */ diff --git a/libibnetdisc/src/iblinkinfotest.c b/libibnetdisc/src/iblinkinfotest.c new file mode 100644 index 0000000..2721870 --- /dev/null +++ b/libibnetdisc/src/iblinkinfotest.c @@ -0,0 +1,388 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2008 Lawrence Livermore National Lab. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "chassis.h" + +char *argv0 = "iblinkinfotest"; +static FILE *f; + +static char *node_name_map_file = NULL; +static nn_map_t *node_name_map = NULL; + +static int timeout_ms = 2000; + +static int debug = 0; +#define DEBUG(str, args...) \ + if (debug) fprintf(stderr, str, ##args) + +static int down_links_only = 0; +static int line_mode = 0; +static int add_sw_settings = 0; +static int print_port_guids = 0; + +static unsigned int +get_max(unsigned int num) +{ + unsigned int v = num; // 32-bit word to find the log base 2 of + unsigned r = 0; // r will be lg(v) + + while (v >>= 1) // unroll for more speed... + { + r++; + } + + return (1 << r); +} + +void +get_msg(char *width_msg, char *speed_msg, int msg_size, ibnd_port_t *port) +{ + int max_speed = 0; + + int max_width = get_max(port->info.link_width_supported + & port->remoteport->info.link_width_supported); + if ((max_width & port->info.link_width_active) == 0) { + // we are not at the max supported width + // print what we could be at. + snprintf(width_msg, msg_size, "Could be %s", + ibnd_linkwidth_str(max_width)); + } + + max_speed = get_max(port->info.link_speed_supported + & port->remoteport->info.link_speed_supported); + if ((max_speed & port->info.link_speed_active) == 0) { + // we are not at the max supported speed + // print what we could be at. + snprintf(speed_msg, msg_size, "Could be %s", + ibnd_linkspeed_str(max_speed)); + } +} + +void +print_port(ibnd_node_t *node, ibnd_port_t *port) +{ + char remote_guid_str[256]; + char remote_str[256]; + char link_str[256]; + char width_msg[256]; + char speed_msg[256]; + char ext_port_str[256]; + + if (!port) + return; + + remote_guid_str[0] = '\0'; + remote_str[0] = '\0'; + link_str[0] = '\0'; + width_msg[0] = '\0'; + speed_msg[0] = '\0'; + + if (port->remoteport) { + if (port->remoteport->ext_portnum) + snprintf(ext_port_str, 256, "%d", port->remoteport->ext_portnum); + else + ext_port_str[0] = '\0'; + + get_msg(width_msg, speed_msg, 256, port); + if (line_mode) { + if (print_port_guids) { + snprintf(remote_guid_str, 256, + "0x%016lx ", + port->remoteport->guid); + } else { + snprintf(remote_guid_str, 256, + "0x%016lx ", + port->remoteport->node->info.nodeguid); + } + } + + snprintf(remote_str, 256, + "%s%6d %4d[%2s] \"%s\" (%s %s)\n", + remote_guid_str, + port->remoteport->info.lid ? + port->remoteport->info.lid : + port->remoteport->node->smalid, + port->remoteport->portnum, + ext_port_str, + port->remoteport->node->nodedesc, + width_msg, + speed_msg + ); + } else { + snprintf(remote_str, 256, + "%6s %4s[%2s] \"\" ( )\n", "", "", ""); + } + + if (add_sw_settings) { + snprintf(link_str, 256, + "(%3s %s %6s/%8s) (HOQ:%d VL_Stall:%d)", + ibnd_linkwidth_str(port->info.link_width_active), + ibnd_linkspeed_str(port->info.link_speed_active), + ibnd_linkstate_str(port->info.link_state), + ibnd_physstate_str(port->info.phys_state), + port->info.hoq_lifetime, + port->info.vl_stall_count + ); + } else { + snprintf(link_str, 256, + "(%3s %s %6s/%8s)", + ibnd_linkwidth_str(port->info.link_width_active), + ibnd_linkspeed_str(port->info.link_speed_active), + ibnd_linkstate_str(port->info.link_state), + ibnd_physstate_str(port->info.phys_state) + ); + } + + if (port->ext_portnum) + snprintf(ext_port_str, 256, "%d", port->ext_portnum); + else + ext_port_str[0] = '\0'; + + if (line_mode) { + printf("0x%016lx \"%30s\" %6d %4d[%2s] ==%s==> %s", + node->info.nodeguid, + node->nodedesc, + node->smalid, port->portnum, + ext_port_str, + link_str, + remote_str + ); + } else { + printf(" %6d %4d[%2s] ==%s==> %s", + node->smalid, port->portnum, + ext_port_str, + link_str, + remote_str + ); + } +} + +void +print_switch(ibnd_node_t *node, void *user_data) +{ + int i = 0; + + if (!line_mode) { + printf("Switch 0x%016lx %s:\n", + node->info.nodeguid, + node->nodedesc); + } + + for (i = 1; i <= node->info.numports; i++) { + ibnd_port_t *port = node->portary[i]; + if (!port) + continue; + if (!down_links_only || port->info.link_state == IBND_LINK_DOWN) { + print_port(node, port); + } + } +} + +void +usage(void) +{ + fprintf(stderr, + "Usage: %s [-hclp -S -D -C -P ]\n" + " Report link speed and connection for each port of each switch which is active\n" + " -h This help message\n" + " -S output only the node specified by guid\n" + " -D start query from node specifed by \n" + " -f specify node to start \"from\"\n" + " -n Number of hops to include away from specified node\n" + " -d print only down links\n" + " -l (line mode) print all information for each link on each line\n" + " -p print additional switch settings (PktLifeTime,HoqLife,VLStallCount)\n" + + + " -t timeout for any single fabric query\n" + " -s show errors\n" + " --node-name-map use specified node name map\n" + + " -C use selected Channel Adaptor name for queries\n" + " -P use selected channel adaptor port for queries\n" + " -g print port guids instead of node guids\n" + , + argv0); + exit(-1); +} + +int +main(int argc, char **argv) +{ + char *ca = 0; + int ca_port = 0; + ibnd_fabric_t *fabric = NULL; + uint64_t guid = 0; + char *dr_path = NULL; + char *from = NULL; + int hops = 0; + ib_portid_t port_id; + + static char const str_opts[] = "S:D:n:C:P:t:sldgphuf:"; + static const struct option long_opts[] = { + { "S", 1, 0, 'S'}, + { "D", 1, 0, 'D'}, + { "num-hops", 1, 0, 'n'}, + { "down-links-only", 0, 0, 'd'}, + { "line-mode", 0, 0, 'l'}, + { "ca-name", 1, 0, 'C'}, + { "ca-port", 1, 0, 'P'}, + { "timeout", 1, 0, 't'}, + { "show", 0, 0, 's'}, + { "print-port-guids", 0, 0, 'g'}, + { "print-additional", 0, 0, 'p'}, + { "help", 0, 0, 'h'}, + { "usage", 0, 0, 'u'}, + { "node-name-map", 1, 0, 1}, + { "debug", 0, 0, 2}, + { "from", 1, 0, 'f'}, + { } + }; + + f = stdout; + + argv0 = argv[0]; + + while (1) { + int ch = getopt_long(argc, argv, str_opts, long_opts, NULL); + if ( ch == -1 ) + break; + switch(ch) { + case 1: + node_name_map_file = strdup(optarg); + break; + case 2: + debug = 1; + ibnd_debug(1); + break; + case 'f': + from = strdup(optarg); + break; + case 'C': + ca = strdup(optarg); + break; + case 'P': + ca_port = strtoul(optarg, 0, 0); + break; + case 'D': + dr_path = strdup(optarg); + break; + case 'n': + hops = (int)strtol(optarg, NULL, 0); + break; + case 'd': + down_links_only = 1; + break; + case 'l': + line_mode = 1; + break; + case 't': + timeout_ms = strtoul(optarg, 0, 0); + break; + case 'g': + print_port_guids = 1; + break; + case 'S': + guid = (uint64_t)strtoull(optarg, 0, 0); + break; + case 'p': + add_sw_settings = 1; + break; + default: + usage(); + break; + } + } + argc -= optind; + argv += optind; + + if (argc && !(f = fopen(argv[0], "w"))) + fprintf(stderr, "can't open file %s for writing", argv[0]); + + node_name_map = open_node_name_map(node_name_map_file); + + if (ibnd_init(ca, ca_port)) { + fprintf(stderr, "ERROR failed to init libibnetdisc"); + exit(1); + } + + if (from) { + /* only scan part of the fabric */ + str2drpath(&(port_id.drpath), from, 0, 0); + if ((fabric = ibnd_discover(timeout_ms, &port_id, hops)) == NULL) { + fprintf(stderr, "discover failed\n"); + exit(1); + } + guid = 0; + } else { + if ((fabric = ibnd_discover(timeout_ms, NULL, -1)) == NULL) { + fprintf(stderr, "discover failed\n"); + exit(1); + } + } + + if (guid) { + ibnd_node_t *sw = ibnd_find_node_guid(fabric, guid); + print_switch(sw, NULL); + } else if (dr_path) { + ibnd_node_t *sw = ibnd_find_node_dr(fabric, dr_path); + print_switch(sw, NULL); + } else { + ibnd_iter_nodes_type(fabric, print_switch, IBND_SWITCH_NODE, NULL); + } + + ibnd_destroy_fabric(fabric); + + close_node_name_map(node_name_map); + exit(0); +} diff --git a/libibnetdisc/src/ibnetdisc.c b/libibnetdisc/src/ibnetdisc.c new file mode 100644 index 0000000..cd298e1 --- /dev/null +++ b/libibnetdisc/src/ibnetdisc.c @@ -0,0 +1,792 @@ +/* + * Copyright (c) 2008 Lawrence Livermore National Laboratory + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +#include "chassis.h" + +static int timeout_ms = 2000; + +#define IBND_DEBUG(str, args...) \ + if (ibdebug) printf("%s:%d; "str, __FILE__, __LINE__, ##args) + +static char *linkwidth_str[] = { + "??", + "1x", + "4x", + "??", + "8x", + "??", + "??", + "??", + "12x" +}; + +static char *linkspeed_str[] = { + "???", + "SDR", + "DDR", + "???", + "QDR" +}; + +static char *linkstate_str[] = { + "No State", + "Down", + "Init", + "Armed", + "Active" +}; + +static char *physstate_str[] = { + "No State", + "Sleep", + "Polling", + "Disabled", + "PortConfigTraining", + "LinkUp", + "LinkErrorRecovery", + "Phy Test" +}; + +char * +ibnd_linkwidth_str(int link_width) +{ + if (link_width > 8) + return linkwidth_str[0]; + else + return linkwidth_str[link_width]; +} + +char * +ibnd_linkspeed_str(int link_speed) +{ + if (link_speed > 4) + return linkspeed_str[0]; + else + return linkspeed_str[link_speed]; +} +char * +ibnd_linkstate_str(int link_state) +{ + if (link_state > 4) + return linkstate_str[0]; + else + return linkstate_str[link_state]; +} + +char * +ibnd_physstate_str(int phys_state) +{ + if (phys_state > 7) + return physstate_str[0]; + else + return physstate_str[phys_state]; +} + +void +decode_port_info(void * rcv_buf, ibnd_port_info_t *pi) +{ + mad_decode_field(rcv_buf, IB_PORT_LID_F, &pi->lid); + mad_decode_field(rcv_buf, IB_PORT_SMLID_F, &pi->smlid); + + mad_decode_field(rcv_buf, IB_PORT_LINK_SPEED_SUPPORTED_F, &pi->link_speed_supported); + mad_decode_field(rcv_buf, IB_PORT_LINK_SPEED_ENABLED_F, &pi->link_speed_enabled); + mad_decode_field(rcv_buf, IB_PORT_LINK_SPEED_ACTIVE_F, &pi->link_speed_active); + + mad_decode_field(rcv_buf, IB_PORT_LOCAL_PORT_F, &pi->local_port); + mad_decode_field(rcv_buf, IB_PORT_LINK_WIDTH_SUPPORTED_F, &pi->link_width_supported); + mad_decode_field(rcv_buf, IB_PORT_LINK_WIDTH_ENABLED_F, &pi->link_width_enabled); + + mad_decode_field(rcv_buf, IB_PORT_LINK_WIDTH_ACTIVE_F, &pi->link_width_active); + + mad_decode_field(rcv_buf, IB_PORT_DIAG_F, &pi->diag_code); + mad_decode_field(rcv_buf, IB_PORT_MKEY_LEASE_F, &pi->mkey_lease); + mad_decode_field(rcv_buf, IB_PORT_CAPMASK_F, &pi->capability_mask); + mad_decode_field(rcv_buf, IB_PORT_MKEY_F, &pi->mkey); + mad_decode_field(rcv_buf, IB_PORT_GID_PREFIX_F, &pi->gid_prefix); + + mad_decode_field(rcv_buf, IB_PORT_STATE_F, &pi->link_state); + mad_decode_field(rcv_buf, IB_PORT_PHYS_STATE_F, &pi->phys_state); + + mad_decode_field(rcv_buf, IB_PORT_LINK_DOWN_DEF_F, &pi->link_down_def_state); + mad_decode_field(rcv_buf, IB_PORT_MKEY_PROT_BITS_F, &pi->mkey_prot_bits); + + mad_decode_field(rcv_buf, IB_PORT_LMC_F, &pi->lmc); + mad_decode_field(rcv_buf, IB_PORT_NEIGHBOR_MTU_F, &pi->neighbor_mtu); + mad_decode_field(rcv_buf, IB_PORT_SMSL_F, &pi->smsl); + mad_decode_field(rcv_buf, IB_PORT_INIT_TYPE_F, &pi->init_type); + + mad_decode_field(rcv_buf, IB_PORT_VL_CAP_F, &pi->vl_capability); + mad_decode_field(rcv_buf, IB_PORT_VL_HIGH_LIMIT_F, &pi->vl_high_limit); + mad_decode_field(rcv_buf, IB_PORT_VL_ARBITRATION_HIGH_CAP_F, &pi->vl_arb_high_cap); + mad_decode_field(rcv_buf, IB_PORT_VL_ARBITRATION_LOW_CAP_F, &pi->vl_arb_low_cap); + + mad_decode_field(rcv_buf, IB_PORT_INIT_TYPE_REPLY_F, &pi->init_reply); + mad_decode_field(rcv_buf, IB_PORT_MTU_CAP_F, &pi->mtu_cap); + mad_decode_field(rcv_buf, IB_PORT_VL_STALL_COUNT_F, &pi->vl_stall_count); + mad_decode_field(rcv_buf, IB_PORT_HOQ_LIFE_F, &pi->hoq_lifetime); + mad_decode_field(rcv_buf, IB_PORT_OPER_VLS_F, &pi->oper_vls); + mad_decode_field(rcv_buf, IB_PORT_PART_EN_INB_F, &pi->partition_enforce_in); + mad_decode_field(rcv_buf, IB_PORT_PART_EN_OUTB_F, &pi->partition_enforce_out); + mad_decode_field(rcv_buf, IB_PORT_FILTER_RAW_INB_F, &pi->filter_raw_in); + mad_decode_field(rcv_buf, IB_PORT_FILTER_RAW_OUTB_F, &pi->filter_raw_out); + mad_decode_field(rcv_buf, IB_PORT_MKEY_VIOL_F, &pi->mkey_violations); + mad_decode_field(rcv_buf, IB_PORT_PKEY_VIOL_F, &pi->pkey_violations); + mad_decode_field(rcv_buf, IB_PORT_QKEY_VIOL_F, &pi->qkey_violations); + + mad_decode_field(rcv_buf, IB_PORT_GUID_CAP_F, &pi->guid_capabilities); + + mad_decode_field(rcv_buf, IB_PORT_CLIENT_REREG_F, &pi->client_rereg); + mad_decode_field(rcv_buf, IB_PORT_SUBN_TIMEOUT_F, &pi->subnet_timeout); + mad_decode_field(rcv_buf, IB_PORT_RESP_TIME_VAL_F, &pi->response_time_val); + mad_decode_field(rcv_buf, IB_PORT_LOCAL_PHYS_ERR_F, &pi->local_phys_error); + mad_decode_field(rcv_buf, IB_PORT_OVERRUN_ERR_F, &pi->overrun_error); + mad_decode_field(rcv_buf, IB_PORT_MAX_CREDIT_HINT_F, &pi->max_credit_hint); + mad_decode_field(rcv_buf, IB_PORT_LINK_ROUND_TRIP_F, &pi->link_round_trip); +} + +int +get_port(ibnd_port_t *port, int portnum, ib_portid_t *portid) +{ + char portinfo[64]; + void *pi = portinfo; + + port->portnum = portnum; + + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout_ms)) + return -1; + + decode_port_info(pi, &port->info); + + IBND_DEBUG("portid %s portnum %d: lid %d state %d physstate %d %s %s\n", + portid2str(portid), portnum, port->info.lid, port->info.link_state, + port->info.phys_state, ibnd_linkwidth_str(port->info.link_width_active), + ibnd_linkspeed_str(port->info.link_speed_active)); + return 1; +} + +void +decode_node_info(void * rcv_buf, ibnd_node_info_t *ni) +{ + mad_decode_field(rcv_buf, IB_NODE_BASE_VERS_F, &ni->base_ver); + mad_decode_field(rcv_buf, IB_NODE_CLASS_VERS_F, &ni->class_ver); + mad_decode_field(rcv_buf, IB_NODE_TYPE_F, &ni->type); + mad_decode_field(rcv_buf, IB_NODE_NPORTS_F, &ni->numports); + mad_decode_field(rcv_buf, IB_NODE_SYSTEM_GUID_F, &ni->sysimgguid); + mad_decode_field(rcv_buf, IB_NODE_GUID_F, &ni->nodeguid); + mad_decode_field(rcv_buf, IB_NODE_PORT_GUID_F, &ni->nodeportguid); + mad_decode_field(rcv_buf, IB_NODE_PARTITION_CAP_F, &ni->partition_cap); + mad_decode_field(rcv_buf, IB_NODE_DEVID_F, &ni->devid); + mad_decode_field(rcv_buf, IB_NODE_REVISION_F, &ni->revision); + mad_decode_field(rcv_buf, IB_NODE_LOCAL_PORT_F, &ni->localport); + mad_decode_field(rcv_buf, IB_NODE_VENDORID_F, &ni->vendid); +} + +/* + * Returns -1 if error. + */ +int +query_node_info(ibnd_node_t *node, ib_portid_t *portid) +{ + char nodeinfo[64]; + void *ni = nodeinfo; + if (!smp_query(ni, portid, IB_ATTR_NODE_INFO, 0, timeout_ms)) + return -1; + decode_node_info(ni, &(node->info)); + return (0); +} + +/* + * Returns 0 if non switch node is found, 1 if switch is found, -1 if error. + */ +int +query_node(ibnd_node_t *node, ibnd_port_t *port, ib_portid_t *portid) +{ + char portinfo[64]; + void *pi = portinfo; + char switchinfo[64]; + void *si = switchinfo; + void *nd = node->nodedesc; + + if (query_node_info(node, portid)) + return -1; + + port->portnum = node->info.localport; + port->guid = node->info.nodeportguid; + + if (!smp_query(nd, portid, IB_ATTR_NODE_DESC, 0, timeout_ms)) + return -1; + + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, 0, timeout_ms)) + return -1; + decode_port_info(pi, &port->info); + + if (node->info.type != IBND_SWITCH_NODE) + return 0; + + node->smalid = port->info.lid; + node->smalmc = port->info.lmc; + + /* after we have the sma information find out the real PortInfo for this port */ + if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, node->info.localport, timeout_ms)) + return -1; + decode_port_info(pi, &port->info); + + if (!smp_query(si, portid, IB_ATTR_SWITCH_INFO, 0, timeout_ms)) + node->sw_info.smaenhsp0 = 0; /* assume base SP0 */ + else + mad_decode_field(si, IB_SW_ENHANCED_PORT0_F, &node->sw_info.smaenhsp0); + + IBND_DEBUG("portid %s: got switch node %" PRIx64 " '%s'\n", + portid2str(portid), node->info.nodeguid, node->nodedesc); + return 1; +} + +static int +add_port_to_dpath(ib_dr_path_t *path, int nextport) +{ + if (path->cnt+2 >= sizeof(path->p)) + return -1; + ++path->cnt; + path->p[path->cnt] = nextport; + return path->cnt; +} + +static int +extend_dpath(ibnd_fabric_t *fabric, ib_dr_path_t *path, int nextport) +{ + int rc = add_port_to_dpath(path, nextport); + if ((rc != -1) && (path->cnt > fabric->maxhops_discovered)) + fabric->maxhops_discovered = path->cnt; + return (rc); +} + +static ibnd_node_t * +find_existing_node(ibnd_fabric_t *fabric, ibnd_node_t *new) +{ + int hash = HASHGUID(new->info.nodeguid) % HTSZ; + ibnd_node_t *node; + + for (node = fabric->nodestbl[hash]; node; node = node->htnext) + if (node->info.nodeguid == new->info.nodeguid) + return node; + + return NULL; +} + +ibnd_node_t * +ibnd_find_node_guid(ibnd_fabric_t *fabric, uint64_t guid) +{ + int hash = HASHGUID(guid) % HTSZ; + ibnd_node_t *node; + + for (node = fabric->nodestbl[hash]; node; node = node->htnext) + if (node->info.nodeguid == guid) + return node; + + return NULL; +} + +ibnd_node_t * +ibnd_update_node(ibnd_node_t *node) +{ + char portinfo[64]; + void *pi = portinfo; + ibnd_port_info_t port0_info; + char switchinfo[64]; + void *si = switchinfo; + ibnd_port_t *port = NULL; + void *nd = node->nodedesc; + + if (query_node_info(node, &(node->path_portid))) + return (NULL); + + if (!smp_query(nd, &(node->path_portid), IB_ATTR_NODE_DESC, 0, timeout_ms)) + return (NULL); + + /* update all the ports */ + for (port = node->ports; port; port = port->next) { + get_port(port, port->portnum, &(node->path_portid)); + } + + if (node->info.type != IBND_SWITCH_NODE) + goto done; + + if (!smp_query(pi, &(node->path_portid), IB_ATTR_PORT_INFO, 0, timeout_ms)) + return (NULL); + decode_port_info(pi, &port0_info); + + node->smalid = port0_info.lid; + node->smalmc = port0_info.lmc; + + if (!smp_query(si, &(node->path_portid), IB_ATTR_SWITCH_INFO, 0, timeout_ms)) + node->sw_info.smaenhsp0 = 0; /* assume base SP0 */ + else + mad_decode_field(si, IB_SW_ENHANCED_PORT0_F, &node->sw_info.smaenhsp0); + +done: + return (node); +} + +ibnd_node_t * +ibnd_find_node_dr(ibnd_fabric_t *fabric, char *dr_str) +{ + int i = 0; + ibnd_node_t *rc = fabric->from_node; + ib_dr_path_t path; + + if (str2drpath(&path, dr_str, 0, 0) == -1) { + return (NULL); + } + + for (i = 0; i <= path.cnt; i++) { + ibnd_port_t *remote_port = NULL; + if (path.p[i] == 0) + continue; + if (!rc->portary) + return (NULL); + + remote_port = rc->portary[path.p[i]]->remoteport; + if (!remote_port) + return (NULL); + + rc = remote_port->node; + } + + return (rc); +} + +static void +add_to_guid_hash(ibnd_node_t *node, ibnd_node_t *hash[]) +{ + int hash_idx = HASHGUID(node->info.nodeguid) % HTSZ; + + node->htnext = hash[hash_idx]; + hash[hash_idx] = node; +} + +static void +add_to_ch_adapters(ibnd_node_t *node, ibnd_fabric_t *fabric) +{ + node->type_next = fabric->ch_adapters; + fabric->ch_adapters = node; +} + +static void +add_to_switches(ibnd_node_t *node, ibnd_fabric_t *fabric) +{ + node->type_next = fabric->switches; + fabric->switches = node; +} + +static void +add_to_routers(ibnd_node_t *node, ibnd_fabric_t *fabric) +{ + node->type_next = fabric->routers; + fabric->routers = node; +} + +static ibnd_node_t * +create_node(ibnd_fabric_t *fabric, ibnd_node_t *temp, ib_portid_t *path, int dist) +{ + ibnd_node_t *node; + + node = malloc(sizeof(*node)); + if (!node) { + IBPANIC("OOM: node creation failed\n"); + return NULL; + } + + memcpy(node, temp, sizeof(*node)); + node->dist = dist; + node->path_portid = *path; + node->fabric = fabric; + + add_to_guid_hash(node, fabric->nodestbl); + + /* add this to the all nodes list */ + node->next = fabric->nodes; + fabric->nodes = node; + + if (node->info.type != IBND_SWITCH_NODE) + dist = MAXHOPS; /* special Ca list */ + + switch (node->info.type) { + case IBND_CA_NODE: + add_to_ch_adapters(node, fabric); + break; + case IBND_SWITCH_NODE: + add_to_switches(node, fabric); + break; + case IBND_ROUTER_NODE: + add_to_routers(node, fabric); + break; + } + + node->dnext = fabric->nodesdist[dist]; + fabric->nodesdist[dist] = node; + + return node; +} + +static ibnd_port_t * +find_existing_port(ibnd_node_t *node, ibnd_port_t *port) +{ + ibnd_port_t *old; + + for (old = node->ports; old; old = old->next) + if (old->portnum == port->portnum) + return old; + + return NULL; +} + +static ibnd_port_t * +add_port(ibnd_node_t *node, ibnd_port_t *temp) +{ + ibnd_port_t *port; + + port = malloc(sizeof(*port)); + if (!port) + return NULL; + + memcpy(port, temp, sizeof(*port)); + port->node = node; + port->next = node->ports; + node->ports = port; + + if (node->portary == NULL) { + node->portary = calloc(sizeof(*node->portary), node->info.numports + 1); + IBND_DEBUG("Failed to allocate the portary array\n"); + } + + if (node->portary) + node->portary[temp->portnum] = port; + + return port; +} + +static void +link_ports(ibnd_node_t *node, ibnd_port_t *port, ibnd_node_t *remotenode, ibnd_port_t *remoteport) +{ + IBND_DEBUG("linking: 0x%" PRIx64 " %p->%p:%u and 0x%" PRIx64 " %p->%p:%u\n", + node->info.nodeguid, node, port, port->portnum, + remotenode->info.nodeguid, remotenode, remoteport, remoteport->portnum); + if (port->remoteport) + port->remoteport->remoteport = NULL; + if (remoteport->remoteport) + remoteport->remoteport->remoteport = NULL; + port->remoteport = remoteport; + remoteport->remoteport = port; +} + +static int +get_remote_node(ibnd_fabric_t *fabric, ibnd_node_t *node, ibnd_port_t *port, ib_portid_t *path, + int portnum, int dist) +{ + ibnd_node_t node_buf; + ibnd_port_t port_buf; + ibnd_node_t *remotenode, *oldnode; + ibnd_port_t *remoteport, *oldport; + + memset(&node_buf, 0, sizeof(node_buf)); + memset(&port_buf, 0, sizeof(port_buf)); + + IBND_DEBUG("handle node %p port %p:%d dist %d\n", node, port, portnum, dist); + if (port->info.phys_state != 5) /* LinkUp */ + return -1; + + if (extend_dpath(fabric, &path->drpath, portnum) < 0) + return -1; + + if (query_node(&node_buf, &port_buf, path) < 0) { + IBWARN("NodeInfo on %s failed, skipping port", + portid2str(path)); + path->drpath.cnt--; /* restore path */ + return -1; + } + + oldnode = find_existing_node(fabric, &node_buf); + if (oldnode) + remotenode = oldnode; + else if (!(remotenode = create_node(fabric, &node_buf, path, dist + 1))) + IBPANIC("no memory"); + + oldport = find_existing_port(remotenode, &port_buf); + if (oldport) { + remoteport = oldport; + } else if (!(remoteport = add_port(remotenode, &port_buf))) + IBPANIC("no memory"); + + link_ports(node, port, remotenode, remoteport); + + path->drpath.cnt--; /* restore path */ + return 0; +} + +int +ibnd_init(char *dev_name, int dev_port) +{ + int mgmt_classes[2] = {IB_SMI_CLASS, IB_SMI_DIRECT_CLASS}; + + /* Crank up the mad lib */ + madrpc_init(dev_name, dev_port, mgmt_classes, 2); + return (0); +} + +/* + */ +ibnd_fabric_t * +ibnd_discover(int timeout_ms, ib_portid_t *from, int hops) +{ + ib_portid_t my_portid = {0}; + ibnd_node_t node_buf; + ibnd_port_t port_buf; + ibnd_node_t *node; + ibnd_port_t *port; + int i; + int dist = 0; + ib_portid_t *path; + int max_hops = MAXHOPS-1; /* default find everything */ + + /* if not everything how much? */ + if (hops >= 0) { + max_hops = hops; + } + + /* If not specified start from "my" port */ + if (!from) { + from = &my_portid; + } + + ibnd_fabric_t *fabric = malloc(sizeof(*fabric)); + + if (!fabric) { + IBPANIC("OOM: failed to malloc ibnd_fabric_t\n"); + return (NULL); + } + + memset(fabric, 0, sizeof(*fabric)); + + IBND_DEBUG("from %s\n", portid2str(from)); + + memset(&node_buf, 0, sizeof(node_buf)); + memset(&port_buf, 0, sizeof(port_buf)); + + if (query_node(&node_buf, &port_buf, from) < 0) { + IBWARN("can't reach node %s\n", portid2str(from)); + return (NULL); + } + + node = create_node(fabric, &node_buf, from, 0); + if (!node) + return (NULL); + + fabric->from_node = node; + + port = add_port(node, &port_buf); + if (!port) + IBPANIC("out of memory"); + + if (node->info.type != IBND_SWITCH_NODE && + get_remote_node(fabric, node, port, from, node->info.localport, 0) < 0) + return fabric; + + for (dist = 0; dist <= max_hops; dist++) { + + for (node = fabric->nodesdist[dist]; node; node = node->dnext) { + + path = &node->path_portid; + + IBND_DEBUG("dist %d node %p\n", dist, node); + /* dump_endnode(path, "processing", node, port); */ + + for (i = 1; i <= node->info.numports; i++) { + if (i == node->info.localport) + continue; + + if (get_port(&port_buf, i, path) < 0) { + IBWARN("can't reach node %s port %d", portid2str(path), i); + return 0; + } + + port = find_existing_port(node, &port_buf); + if (port) + continue; + + port = add_port(node, &port_buf); + if (!port) + IBPANIC("out of memory"); + + /* If switch, set port GUID to node port GUID */ + if (node->info.type == IBND_SWITCH_NODE) + port->guid = node->info.nodeportguid; + + get_remote_node(fabric, node, port, path, i, dist); + } + } + } + + fabric->chassis = group_nodes(fabric); + + return fabric; +} + +static void +destroy_node(ibnd_node_t *node) +{ + ibnd_port_t *port = node->ports; + ibnd_port_t *next = NULL; + + free(node->portary); + while (port) { + next = port->next; + free(port); + port = next; + } + if (node->chrecord) + free(node->chrecord); + free(node); +} + +void +ibnd_destroy_fabric(ibnd_fabric_t *fabric) +{ + int dist = 0; + ibnd_node_t *node = NULL; + ibnd_node_t *next = NULL; + + for (dist = 0; dist <= MAXHOPS; dist++) { + node = fabric->nodesdist[dist]; + while (node) { + next = node->dnext; + destroy_node(node); + node = next; + } + } + free(fabric); +} + +void +ibnd_debug(int i) +{ + if (i) { + ibdebug++; + madrpc_show_errors(1); + umad_debug(i); + } else { + ibdebug = 0; + madrpc_show_errors(0); + umad_debug(i); + } +} + +const char* +ibnd_node_type_str(ibnd_node_t *node) +{ + switch(node->info.type) { + case IBND_CA_NODE: return "Ca"; + case IBND_SWITCH_NODE: return "Switch"; + case IBND_ROUTER_NODE: return "Router"; + } + return "??"; +} + +void +ibnd_iter_nodes(ibnd_fabric_t *fabric, + ibnd_iter_func_t func, + void *user_data) +{ + int i = 0; + ibnd_node_t *list = NULL; + ibnd_node_t *cur = NULL; + + for (i = 0; i < HTSZ; i++) { + list = fabric->nodestbl[i]; + for (cur = list; cur; cur = cur->type_next) { + func(cur, user_data); + } + } +} + + +void +ibnd_iter_nodes_type(ibnd_fabric_t *fabric, + ibnd_iter_func_t func, + ibnd_node_type_t node_type, + void *user_data) +{ + ibnd_node_t *list = NULL; + ibnd_node_t *cur = NULL; + + switch (node_type) { + case IBND_SWITCH_NODE: + list = fabric->switches; + break; + case IBND_CA_NODE: + list = fabric->ch_adapters; + break; + case IBND_ROUTER_NODE: + list = fabric->routers; + break; + default: + IBND_DEBUG("Invalid node_type specified %d\n", node_type); + break; + } + + for (cur = list; cur; cur = cur->type_next) { + func(cur, user_data); + } +} + diff --git a/libibnetdisc/src/ibnetdisctest.c b/libibnetdisc/src/ibnetdisctest.c new file mode 100644 index 0000000..be825dc --- /dev/null +++ b/libibnetdisc/src/ibnetdisctest.c @@ -0,0 +1,636 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2008 Lawrence Livermore National Lab. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +//#include "chassis.h" + +#define LIST_CA_NODE (1 << IBND_CA_NODE) +#define LIST_SWITCH_NODE (1 << IBND_SWITCH_NODE) +#define LIST_ROUTER_NODE (1 << IBND_ROUTER_NODE) + +char *argv0 = "ibnetdiscover"; +static FILE *f; + +static char *node_name_map_file = NULL; +static nn_map_t *node_name_map = NULL; + +static int timeout_ms = 2000; +static int dumplevel = 0; + +static int debug = 0; +#define DEBUG(str, args...) \ + if (debug) fprintf(stderr, str, ##args) + +char * +node_name(ibnd_node_t *node) +{ + static char buf[256]; + + switch(node->info.type) { + case IBND_CA_NODE: + sprintf(buf, "\"%s", "H"); + break; + case IBND_SWITCH_NODE: + sprintf(buf, "\"%s", "S"); + break; + case IBND_ROUTER_NODE: + sprintf(buf, "\"%s", "R"); + break; + default: + sprintf(buf, "\"%s", "?"); + break; + } + sprintf(buf+2, "-%016" PRIx64 "\"", node->info.nodeguid); + + return buf; +} + +void +list_node(ibnd_node_t *node, void *user_data) +{ + char *nodename = remap_node_name(node_name_map, node->info.nodeguid, + node->nodedesc); + + fprintf(f, "%s\t : 0x%016" PRIx64 " ports %d devid 0x%x vendid 0x%x \"%s\"\n", + ibnd_node_type_str(node), + node->info.nodeguid, node->info.numports, node->info.devid, + node->info.vendid, + nodename); + + free(nodename); +} + +void +list_nodes(ibnd_fabric_t *fabric, int list) +{ + if (list & LIST_CA_NODE) { + ibnd_iter_nodes_type(fabric, list_node, IBND_CA_NODE, NULL); + } + if (list & LIST_SWITCH_NODE) { + ibnd_iter_nodes_type(fabric, list_node, IBND_SWITCH_NODE, NULL); + } + if (list & LIST_ROUTER_NODE) { + ibnd_iter_nodes_type(fabric, list_node, IBND_ROUTER_NODE, NULL); + } +} + +void +out_ids(ibnd_node_t *node, int group, char *chname) +{ + fprintf(f, "\nvendid=0x%x\ndevid=0x%x\n", node->info.vendid, node->info.devid); + if (node->info.sysimgguid) + fprintf(f, "sysimgguid=0x%" PRIx64, node->info.sysimgguid); + if (group + && node->chrecord && node->chrecord->chassisnum) { + fprintf(f, "\t\t# Chassis %d", node->chrecord->chassisnum); + if (chname) + fprintf(f, " (%s)", clean_nodedesc(chname)); + if (ibnd_is_xsigo_tca(node->info.nodeguid) && node->ports->remoteport) + fprintf(f, " slot %d", node->ports->remoteport->portnum); + } + fprintf(f, "\n"); +} + + +uint64_t +out_chassis(ibnd_fabric_t *fabric, int chassisnum) +{ + uint64_t guid; + + fprintf(f, "\nChassis %d", chassisnum); + guid = ibnd_get_chassis_guid(fabric, chassisnum); + if (guid) + fprintf(f, " (guid 0x%" PRIx64 ")", guid); + fprintf(f, "\n"); + return guid; +} + +void +out_switch(ibnd_node_t *node, int group, char *chname) +{ + char *str; + char str2[256]; + char *nodename = NULL; + + out_ids(node, group, chname); + fprintf(f, "switchguid=0x%" PRIx64, node->info.nodeguid); + fprintf(f, "(%" PRIx64 ")", node->info.nodeportguid); + if (group) { + str = ibnd_get_chassis_type(node); + if (str) + fprintf(f, "%s ", str); + str = ibnd_get_chassis_slot_str(node, str2, 256); + if (str) + fprintf(f, "%s ", str); + } + + nodename = remap_node_name(node_name_map, node->info.nodeguid, + node->nodedesc); + + fprintf(f, "\nSwitch\t%d %s\t\t# \"%s\" %s port 0 lid %d lmc %d\n", + node->info.numports, node_name(node), + nodename, + node->sw_info.smaenhsp0 ? "enhanced" : "base", + node->smalid, node->smalmc); + + free(nodename); +} + +void +out_ca(ibnd_node_t *node, int group, char *chname) +{ + char *node_type; + char *node_type2; + + out_ids(node, group, chname); + switch(node->info.type) { + case IBND_CA_NODE: + node_type = "ca"; + node_type2 = "Ca"; + break; + case IBND_ROUTER_NODE: + node_type = "rt"; + node_type2 = "Rt"; + break; + default: + node_type = "???"; + node_type2 = "???"; + break; + } + + fprintf(f, "%sguid=0x%" PRIx64 "\n", node_type, node->info.nodeguid); + fprintf(f, "%s\t%d %s\t\t# \"%s\"", + node_type2, node->info.numports, node_name(node), + clean_nodedesc(node->nodedesc)); + if (group && ibnd_is_xsigo_hca(node->info.nodeguid)) + fprintf(f, " (scp)"); + fprintf(f, "\n"); +} + +#define OUT_BUFFER_SIZE 16 +static char * +out_ext_port(ibnd_port_t *port, int group) +{ + static char mapping[OUT_BUFFER_SIZE]; + + if (group && port->ext_portnum != 0) { + snprintf(mapping, OUT_BUFFER_SIZE, + "[ext %d]", port->ext_portnum); + } + + return (mapping); +} + +void +out_switch_port(ibnd_port_t *port, int group) +{ + char *ext_port_str = NULL; + char *rem_nodename = NULL; + + DEBUG("port %p:%d remoteport %p\n", port, port->portnum, port->remoteport); + fprintf(f, "[%d]", port->portnum); + + ext_port_str = out_ext_port(port, group); + if (ext_port_str) + fprintf(f, "%s", ext_port_str); + + rem_nodename = remap_node_name(node_name_map, + port->remoteport->node->info.nodeguid, + port->remoteport->node->nodedesc); + + ext_port_str = out_ext_port(port->remoteport, group); + fprintf(f, "\t%s[%d]%s", + node_name(port->remoteport->node), + port->remoteport->portnum, + ext_port_str ? ext_port_str : ""); + if (port->remoteport->node->info.type != IBND_SWITCH_NODE) + fprintf(f, "(%" PRIx64 ") ", port->remoteport->guid); + fprintf(f, "\t\t# \"%s\" lid %d %s%s", + rem_nodename, + port->remoteport->node->info.type == IBND_SWITCH_NODE ? port->remoteport->node->smalid : port->remoteport->info.lid, + ibnd_linkwidth_str(port->info.link_width_active), + ibnd_linkspeed_str(port->info.link_speed_active)); + + if (ibnd_is_xsigo_tca(port->remoteport->guid)) + fprintf(f, " slot %d", port->portnum); + else if (ibnd_is_xsigo_hca(port->remoteport->guid)) + fprintf(f, " (scp)"); + fprintf(f, "\n"); + + free(rem_nodename); +} + +void +out_ca_port(ibnd_port_t *port, int group) +{ + char *str = NULL; + char *rem_nodename = NULL; + + fprintf(f, "[%d]", port->portnum); + if (port->node->info.type != IBND_SWITCH_NODE) + fprintf(f, "(%" PRIx64 ") ", port->guid); + fprintf(f, "\t%s[%d]", + node_name(port->remoteport->node), + port->remoteport->portnum); + str = out_ext_port(port->remoteport, group); + if (str) + fprintf(f, "%s", str); + if (port->remoteport->node->info.type != IBND_SWITCH_NODE) + fprintf(f, " (%" PRIx64 ") ", port->remoteport->guid); + + rem_nodename = remap_node_name(node_name_map, + port->remoteport->node->info.nodeguid, + port->remoteport->node->nodedesc); + + fprintf(f, "\t\t# lid %d lmc %d \"%s\" lid %d %s%s\n", + port->info.lid, port->info.lmc, rem_nodename, + port->remoteport->node->info.type == IBND_SWITCH_NODE ? port->remoteport->node->smalid : port->remoteport->info.lid, + ibnd_linkwidth_str(port->info.link_width_active), + ibnd_linkspeed_str(port->info.link_speed_active)); + + free(rem_nodename); +} + +int +dump_topology(int group, ibnd_fabric_t *fabric) +{ + ibnd_node_t *node; + ibnd_port_t *port; + int i = 0, dist = 0; + time_t t = time(0); + uint64_t chguid; + char *chname = NULL; + + fprintf(f, "#\n# Topology file: generated on %s#\n", ctime(&t)); + fprintf(f, "# Max of %d hops discovered\n", fabric->maxhops_discovered); + fprintf(f, "# Initiated from node %016" PRIx64 " port %016" PRIx64 "\n", + fabric->from_node->info.nodeguid, fabric->from_node->info.nodeportguid); + + /* Make pass on switches */ + if (group) { + ibnd_chassis_list_t *ch = NULL; + + /* Chassis based switches first */ + for (ch = fabric->chassis; ch; ch = ch->next) { + int n = 0; + + if (!ch->chassisnum) + continue; + chguid = out_chassis(fabric, ch->chassisnum); + + chname = NULL; +/** + * Hal will this work for Xsigo? + */ + if (ibnd_is_xsigo_guid(chguid)) { + for (node = ch->nodes; node; node = node->chassis_next) { + if (ibnd_is_xsigo_hca(node->info.nodeguid)) { + chname = node->nodedesc; + fprintf(f, "Hostname: %s\n", clean_nodedesc(node->nodedesc)); + } + } + +#if 0 + for (node = fabric->nodesdist[MAXHOPS]; node; node = node->dnext) { + if (!node->chrecord || + !node->chrecord->chassisnum) + continue; + + if (node->chrecord->chassisnum != ch->chassisnum) + continue; + + if (ibnd_is_xsigo_hca(node->nodeguid)) { + chname = node->nodedesc; + fprintf(f, "Hostname: %s\n", clean_nodedesc(node->nodedesc)); + } + } +#endif + } + + fprintf(f, "\n# Spine Nodes"); + for (n = 1; n <= (SPINES_MAX_NUM+1); n++) { + if (ch->spinenode[n]) { + out_switch(ch->spinenode[n], group, chname); + for (port = ch->spinenode[n]->ports; port; port = port->next, i++) + if (port->remoteport) + out_switch_port(port, group); + } + } + fprintf(f, "\n# Line Nodes"); + for (n = 1; n <= (LINES_MAX_NUM+1); n++) { + if (ch->linenode[n]) { + out_switch(ch->linenode[n], group, chname); + for (port = ch->linenode[n]->ports; port; port = port->next, i++) + if (port->remoteport) + out_switch_port(port, group); + } + } + + fprintf(f, "\n# Chassis Switches"); + for (node = ch->nodes; node; node = node->chassis_next) { + if (node->info.type == IBND_SWITCH_NODE) { + out_switch(node, group, chname); + for (port = node->ports; port; port = port->next, i++) + if (port->remoteport) + out_switch_port(port, group); + } + } + + fprintf(f, "\n# Chassis CAs"); + for (node = ch->nodes; node; node = node->chassis_next) { + if (node->info.type == IBND_CA_NODE) { + out_ca(node, group, chname); + for (port = node->ports; port; port = port->next, i++) + if (port->remoteport) + out_ca_port(port, group); + } + } + + } + + } else { /* !group */ + for (node = fabric->switches; node; node = node->type_next) { + DEBUG("SWITCH: dist %d node %p\n", dist, node); + out_switch(node, group, chname); + for (port = node->ports; port; port = port->next, i++) + if (port->remoteport) + out_switch_port(port, group); + } + } + + chname = NULL; + if (group) { + fprintf(f, "\nNon-Chassis Nodes\n"); + for (node = fabric->switches; node; node = node->type_next) { + DEBUG("SWITCH: dist %d node %p\n", dist, node); + /* Now, skip chassis based switches */ + if (node->chrecord && + node->chrecord->chassisnum) + continue; + out_switch(node, group, chname); + + for (port = node->ports; port; port = port->next, i++) + if (port->remoteport) + out_switch_port(port, group); + } + + } + + /* Make pass on CAs */ + for (node = fabric->ch_adapters; node; node = node->type_next) { + DEBUG("CA: dist %d node %p\n", dist, node); + /* Now, skip chassis based CAs */ + if (group && node->chrecord && + node->chrecord->chassisnum) + continue; + out_ca(node, group, chname); + + for (port = node->ports; port; port = port->next, i++) + if (port->remoteport) + out_ca_port(port, group); + } + + /* make pass on routers */ + for (node = fabric->routers; node; node = node->type_next) { + DEBUG("RT: dist %d node %p\n", dist, node); + /* Now, skip chassis based CAs */ + if (group && node->chrecord && + node->chrecord->chassisnum) + continue; + out_ca(node, group, chname); + for (port = node->ports; port; port = port->next, i++) + if (port->remoteport) + out_ca_port(port, group); + } + + return i; +} + + +void dump_ports_report (ibnd_fabric_t *fabric) +{ + int b, p; + ibnd_node_t *node; + ibnd_port_t *port; + +#if 0 + /* I am not quite sure why this is here. + * I don't think it is appropriate for this code to change the fabric + * objects which are returned. + * So for now I am going to leave it out. + */ + // If switch and LID == 0, search of other switch ports with + // valid LID and assign it to all ports of that switch + for (b = 0; b <= MAXHOPS; b++) + for (node = fabric->nodesdist[b]; node; node = node->dnext) + if (node->type == SWITCH_NODE) { + int swlid = 0; + for (p = 0, port = node->ports; + p < node->numports && port && !swlid; + port = port->next) + if (port->pi.lid != 0) + swlid = port->pi.lid; + for (p = 0, port = node->ports; + p < node->numports && port; + port = port->next) + port->pi.lid = swlid; + } +#endif + + for (b = 0; b <= MAXHOPS; b++) + for (node = fabric->nodesdist[b]; node; node = node->dnext) { + for (p = 0, port = node->ports; + p < node->info.numports && port; + p++, port = port->next) { + fprintf(stdout, + "%2s %5d %2d 0x%016" PRIx64 " %s %s", + ibnd_node_type_str(port->node), port->info.lid, + port->portnum, + port->guid, + ibnd_linkwidth_str(port->info.link_width_active), + ibnd_linkspeed_str(port->info.link_speed_active)); + if (port->remoteport) + fprintf(stdout, + " - %2s %5d %2d 0x%016" PRIx64 + " ( '%s' - '%s' )\n", + ibnd_node_type_str(port->remoteport->node), + port->remoteport->info.lid, + port->remoteport->portnum, + port->remoteport->guid, + port->node->nodedesc, + port->remoteport->node->nodedesc); + else + fprintf(stdout, "%36s'%s'\n", "", + port->node->nodedesc); + } + } +} + +void +usage(void) +{ + fprintf(stderr, "Usage: %s [-d(ebug)] -s(how) -l(ist) -g(rouping) -H(ca_list) -S(witch_list) -R(outer_list) -V(ersion) -C ca_name -P ca_port " + "-t(imeout) timeout_ms --node-name-map node-name-map] -p(orts) []\n", + argv0); + fprintf(stderr, " --node-name-map specify a node name map file\n"); + exit(-1); +} + +int +main(int argc, char **argv) +{ + int list = 0; + char *ca = 0; + int ca_port = 0; + int group = 0; + int ports_report = 0; + ibnd_fabric_t *fabric = NULL; + + static char const str_opts[] = "C:P:t:devslgHSRpVhu"; + static const struct option long_opts[] = { + { "C", 1, 0, 'C'}, + { "P", 1, 0, 'P'}, + { "debug", 0, 0, 'd'}, + { "show", 0, 0, 's'}, + { "list", 0, 0, 'l'}, + { "grouping", 0, 0, 'g'}, + { "Hca_list", 0, 0, 'H'}, + { "Switch_list", 0, 0, 'S'}, + { "Router_list", 0, 0, 'R'}, + { "timeout", 1, 0, 't'}, + { "node-name-map", 1, 0, 1}, + { "ports", 0, 0, 'p'}, + { "help", 0, 0, 'h'}, + { "usage", 0, 0, 'u'}, + { } + }; + + f = stdout; + + argv0 = argv[0]; + + while (1) { + int ch = getopt_long(argc, argv, str_opts, long_opts, NULL); + if ( ch == -1 ) + break; + switch(ch) { + case 1: + node_name_map_file = strdup(optarg); + break; + case 'C': + ca = optarg; + break; + case 'P': + ca_port = strtoul(optarg, 0, 0); + break; + case 'd': + debug = 1; + ibnd_debug(1); + break; + case 't': + timeout_ms = strtoul(optarg, 0, 0); + break; + case 's': + dumplevel = 1; + break; + case 'l': + list = LIST_CA_NODE | LIST_SWITCH_NODE | LIST_ROUTER_NODE; + break; + case 'g': + group = 1; + break; + case 'S': + list |= LIST_SWITCH_NODE; + break; + case 'H': + list |= LIST_CA_NODE; + break; + case 'R': + list |= LIST_ROUTER_NODE; + break; + case 'p': + ports_report = 1; + break; + default: + usage(); + break; + } + } + argc -= optind; + argv += optind; + + if (argc && !(f = fopen(argv[0], "w"))) + fprintf(stderr, "can't open file %s for writing", argv[0]); + + node_name_map = open_node_name_map(node_name_map_file); + + if (ibnd_init(ca, ca_port)) { + fprintf(stderr, "ERROR failed to init libibnetdisc"); + exit(1); + } + + if ((fabric = ibnd_discover(timeout_ms, NULL, -1)) == NULL) { + fprintf(stderr, "discover failed\n"); + exit(1); + } + + if (ports_report) + dump_ports_report(fabric); + else if (list) + list_nodes(fabric, list); + else + dump_topology(group, fabric); + + ibnd_destroy_fabric(fabric); + close_node_name_map(node_name_map); + exit(0); +} diff --git a/libibnetdisc/src/libibnetdisc.map b/libibnetdisc/src/libibnetdisc.map new file mode 100644 index 0000000..88218b4 --- /dev/null +++ b/libibnetdisc/src/libibnetdisc.map @@ -0,0 +1,24 @@ +IBNETDISC_1.0 { + global: + ibnd_init; + ibnd_debug; + ibnd_discover; + ibnd_destroy_fabric; + ibnd_find_node_guid; + ibnd_update_node; + ibnd_find_node_dr; + ibnd_linkwidth_str; + ibnd_linkspeed_str; + ibnd_node_type_str; + ibnd_is_xsigo_guid; + ibnd_is_xsigo_tca; + ibnd_is_xsigo_hca; + ibnd_get_chassis_guid; + ibnd_get_chassis_type; + ibnd_get_chassis_slot_str; + ibnd_linkstate_str; + ibnd_physstate_str; + ibnd_iter_nodes; + ibnd_iter_nodes_type; + local: *; +}; -- From sashak at voltaire.com Fri Jun 27 16:34:09 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 02:34:09 +0300 Subject: [ofa-general] Re: [PATCH RFC] Create a new library "libibnetdisc" In-Reply-To: <20080627160952.600be0e2.weiny2@llnl.gov> References: <20080627160952.600be0e2.weiny2@llnl.gov> Message-ID: <20080627233409.GW29456@sashak.voltaire.com> Hi Ira, On 16:09 Fri 27 Jun , Ira Weiny wrote: > I have been wanted to do this for some time and ever since mentioning it to > Sasha at Sonoma have worked on it in my "spare" time... ;-) > > This creates an ibnetdiscover library. It seems as time goes on that more and > more people have tried to "print" something from ibnetdiscover to get that data > out with some parser. I figure it should be available from a library. I have > only written test programs "ibnetdisctest" and "iblinkinfotest" to prototype > performance. Also the interface is a bit rought around the edges but all the > data is there. > > At this point I have some pretty good results from iblinkinfotest (~80% > faster!) > > So I wanted to get opinions on where this should go. Do you think it should be a separate package? I thought rather about having it in infiniband-diags so the tools there will use it. Sasha From sashak at voltaire.com Fri Jun 27 17:49:46 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 03:49:46 +0300 Subject: [ofa-general] [PATCH] opensm: speedup and improve ipv6 snm handling Message-ID: <20080628004946.GY29456@sashak.voltaire.com> When requested MGID value matchs IPv6 SNM signature the search will be performed for already truncated value. And so we can move the MGID matching check to upper layer and it will be done once per mgrp by mgid search and not in an each match iteration. Signed-off-by: Sasha Khapyorsky --- Only compilation was tested yet. opensm/opensm/osm_sa_mcmember_record.c | 77 ++++++++++++++----------------- 1 files changed, 35 insertions(+), 42 deletions(-) diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index 086f3a5..b0a242e 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -968,7 +968,7 @@ Exit: typedef struct osm_sa_pr_mcmr_search_ctxt { - ib_gid_t *p_mgid; + ib_gid_t mgid; osm_mgrp_t *p_mgrp; osm_sa_t *sa; } osm_sa_pr_mcmr_search_ctxt_t; @@ -978,13 +978,8 @@ typedef struct osm_sa_pr_mcmr_search_ctxt { static void __search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) { - osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = - (osm_sa_pr_mcmr_search_ctxt_t *) context; - const ib_gid_t *p_recvd_mgid; - osm_sa_t *sa; - - p_recvd_mgid = p_ctxt->p_mgid; - sa = p_ctxt->sa; + osm_sa_pr_mcmr_search_ctxt_t *p_ctxt = context; + osm_sa_t *sa = p_ctxt->sa; /* ignore groups marked for deletion */ if (p_mgrp->to_be_deleted) @@ -992,39 +987,8 @@ __search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) /* compare entire MGID so different scope will not sneak in for the same MGID */ - if (memcmp(&p_mgrp->mcmember_rec.mgid, p_recvd_mgid, sizeof(ib_gid_t))) { - - if (sa->p_subn->opt.consolidate_ipv6_snm_req) { - /* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ - /* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ - /* Where Z is the scope, XXXX is the P_Key, and - * YYYYYY is the last 24 bits of the port guid */ -#define PREFIX_NOPKEY_MASK (0xff10ffff0000ffffULL) -#define PREFIX_MASK (0xff10ffffffffffffULL) -#define PREFIX_SIGNATURE (0xff10601b00000000ULL) -#define INT_ID_MASK (0xfffffff1ff000000ULL) -#define INT_ID_SIGNATURE (0x00000001ff000000ULL) - uint64_t g_prefix = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.prefix); - uint64_t g_interface_id = cl_ntoh64(p_mgrp->mcmember_rec.mgid.unicast.interface_id); - uint64_t rcv_prefix = cl_ntoh64(p_recvd_mgid->unicast.prefix); - uint64_t rcv_interface_id = cl_ntoh64(p_recvd_mgid->unicast.interface_id); - - if ((rcv_prefix & PREFIX_NOPKEY_MASK) == PREFIX_SIGNATURE && - (rcv_interface_id & INT_ID_MASK) == INT_ID_SIGNATURE && - (g_prefix & PREFIX_MASK) == - (rcv_prefix & PREFIX_MASK) && - (g_interface_id & INT_ID_MASK) == - (rcv_interface_id & INT_ID_MASK)) { - OSM_LOG(sa->p_log, OSM_LOG_INFO, - "Special Case Solicited Node Mcast " - "Join for MGID 0x%016" PRIx64 - " : 0x%016" PRIx64 "\n", - rcv_prefix, rcv_interface_id); - } else - return; - } else - return; - } + if (memcmp(&p_mgrp->mcmember_rec.mgid, &p_ctxt->mgid, sizeof(ib_gid_t))) + return; if (p_ctxt->p_mgrp) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B30: " @@ -1039,6 +1003,26 @@ __search_mgrp_by_mgid(IN osm_mgrp_t * const p_mgrp, IN void *context) /********************************************************************** **********************************************************************/ +#define PREFIX_MASK CL_HTON64(0xff10ffff0000ffffULL) +#define PREFIX_SIGNATURE CL_HTON64(0xff10601b00000000ULL) +#define INT_ID_MASK CL_HTON64(0xfffffff1ff000000ULL) +#define INT_ID_SIGNATURE CL_HTON64(0x00000001ff000000ULL) + +/* Special Case IPv6 Solicited Node Multicast (SNM) addresses */ +/* 0xff1Z601bXXXX0000 : 0x00000001ffYYYYYY */ +/* Where Z is the scope, XXXX is the P_Key, and + * YYYYYY is the last 24 bits of the port guid */ +static unsigned match_and_update_ipv6_snm_mgid(ib_gid_t *mgid) +{ + if ((mgid->unicast.prefix & PREFIX_MASK) == PREFIX_SIGNATURE && + (mgid->unicast.interface_id & INT_ID_MASK) == INT_ID_SIGNATURE) { + mgid->unicast.prefix &= PREFIX_MASK; + mgid->unicast.interface_id &= INT_ID_MASK; + return 1; + } + return 0; +} + ib_api_status_t osm_get_mgrp_by_mgid(IN osm_sa_t *sa, IN ib_gid_t *p_mgid, @@ -1048,10 +1032,19 @@ osm_get_mgrp_by_mgid(IN osm_sa_t *sa, osm_mgrp_t *p_mgrp; int i; - mcmr_search_context.p_mgid = p_mgid; + memcpy(&mcmr_search_context.mgid, p_mgid, sizeof(*p_mgid)); mcmr_search_context.sa = sa; mcmr_search_context.p_mgrp = NULL; + if (sa->p_subn->opt.consolidate_ipv6_snm_req && + match_and_update_ipv6_snm_mgid(&mcmr_search_context.mgid)) { + OSM_LOG(sa->p_log, OSM_LOG_DEBUG, + "Special Case Solicited Node Mcast Join for MGID" + " 0x%016" PRIx64 " : 0x%016" PRIx64 "\n", + cl_ntoh64(p_mgid->unicast.prefix), + cl_ntoh64(p_mgid->unicast.interface_id)); + } + for (i = 0; i <= sa->p_subn->max_mcast_lid_ho - IB_LID_MCAST_START_HO; i++) { p_mgrp = sa->p_subn->mgroups[i]; -- 1.5.5.1.178.g1f811 From davem at davemloft.net Fri Jun 27 20:09:14 2008 From: davem at davemloft.net (David Miller) Date: Fri, 27 Jun 2008 20:09:14 -0700 (PDT) Subject: [ofa-general] Re: [PATCH v2] net/inet_lro: remove setting skb->ip_summed when not LRO-able In-Reply-To: <1214407752.23583.69.camel@mtls03> References: <1214407752.23583.69.camel@mtls03> Message-ID: <20080627.200914.146689237.davem@davemloft.net> From: Eli Cohen Date: Wed, 25 Jun 2008 18:29:12 +0300 > When an SKB cannot be chained to a session, the current code attempts > to "restore" its ip_summed field from lro_mgr->ip_summed. However, > lro_mgr->ip_summed does not hold the original value; in fact, we'd > better not touch skb->ip_summed since it is not modified by the code > in the path leading to a failure to chain it. > Also use a cleaer comment to the describe the ip_summed field of > struct net_lro_mgr. > > Issue raised by Or Gerlitz > > Signed-off-by: Eli Cohen Patch applied, thanks everyone. From rdreier at cisco.com Fri Jun 27 21:39:28 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Jun 2008 21:39:28 -0700 Subject: [ofa-general] Re: [RFC][PATCH] last recv race patch In-Reply-To: (Shirley Ma's message of "Wed, 25 Jun 2008 15:16:16 -0700") References: Message-ID: > The race is between the list is taken off from reap_list for destory > because of the timeout and readd to reap_list from this condition. So we > need a condition check here. I'm sorry, I still don't see what the bug is. What timeout removes things from the rx_reap_list? And how could any context get added to the rx_reap_list more than once? The code you are changing is: > if (!--p->recv_count) { > spin_lock_irqsave(&priv->lock, flags); > - list_move(&p->list, &priv->cm.rx_reap_list); > - spin_unlock_irqrestore(&priv->lock, flags); > - queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); > + if (p->state == IPOIB_CM_RX_LIVE) { > + list_move(&p->list, &priv->cm.rx_reap_list); > + spin_unlock_irqrestore(&priv->lock, flags); > + queue_work(ipoib_workqueue, &priv->cm.rx_reap_task); > + } else > + spin_unlock_irqrestore(&priv->lock, flags); and I don't see how recv_count could ever be 0 twice: we never increment it. In any case I don't see how testing against the LIVE state could be correct, since for example we want to reap stale QPs that we moved to IPOIB_CM_RX_ERROR after all the pending receives have completed. - R. From rdreier at cisco.com Fri Jun 27 21:41:18 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Jun 2008 21:41:18 -0700 Subject: [ofa-general] ***SPAM*** [PATCH 3/3 v3] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: <1214492150.23583.116.camel@mtls03> (Eli Cohen's message of "Thu, 26 Jun 2008 17:55:50 +0300") References: <1214492150.23583.116.camel@mtls03> Message-ID: > > + .create_flags = 0 > Why do you need to set create_flags explicitly? right, that's redundant, since missing initializer fields are always cleared to 0. > > - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; > > + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; > and change this too? this doesn't seem that bad a change to me. It is defensive against later changes adding something else to create_flags earlier on in the function, and it is also nice to be uniform with the block loopback flag. So I'll leave this in. - R. From rdreier at cisco.com Fri Jun 27 21:43:23 2008 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Jun 2008 21:43:23 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <20080626225945.27989.73122.stgit@dell3.ogc.int> (Steve Wise's message of "Thu, 26 Jun 2008 17:59:45 -0500") References: <20080626225945.27989.73122.stgit@dell3.ogc.int> Message-ID: > 2) an individual SGE length cannot exceed 128MB for a zero-stag SGE. > This is ok since its not possible to allocate such a large chunk of > pinned contiguous dma-coherent memory. I guess this is OK, although conceivably one could get a huge region through chicanery like reserving memory at boot or something like that. > TODO: > > Bump the required fw major number since the driver/fw interface has changed. So I assume this patch isn't really baked enough to merge yet? - R. From vlad at lists.openfabrics.org Sat Jun 28 02:37:46 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 28 Jun 2008 02:37:46 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080628-0200 daily build status Message-ID: <20080628093746.E7974E60D5F@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080628-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From sashak at voltaire.com Sat Jun 28 05:35:36 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 15:35:36 +0300 Subject: [ofa-general] snprintf() usage bug [was: Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria] In-Reply-To: <48566664.1010004@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> Message-ID: <20080628123536.GC29456@sashak.voltaire.com> Hi Yevgeny, On 16:11 Mon 16 Jun , Yevgeny Kliteynik wrote: > @@ -621,6 +627,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( > list_iterator = cl_list_next(list_iterator); > continue; > } > + > + if (n) > + n += snprintf(buff + n, sizeof(buff) - n, > + ", "); > + n += snprintf(buff + n, sizeof(buff) - n, > + "S_Port (0x%016" PRIx64 ")", > + cl_ntoh64(osm_physp_get_port_guid( > + p_src_physp))); I tested this patch (from performance impacts perspective) and found that such usage of snprintf() is just wrong and leads to buffer overflow. snprintf man page ('Return value' section) states: The functions snprintf() and vsnprintf() do not write more than size bytes (including the trailing '\0'). If the output was truncated due to this limit then the return value is the number of characters (not including the trailing '\0') which would have been written to the final string if enough space had been available. Thus, a return value of size or more means that the output was truncated. This effectively invalidates repeated usage of things like: n += snprintf(buf + n, sizeof(buf) - n, ...) , when it is done without actual 'n' value checks. And ugh, I guess we have a lot of such in current management tree. :( Sasha From sashak at voltaire.com Sat Jun 28 06:16:19 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 16:16:19 +0300 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria In-Reply-To: <485DF17B.2050500@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> <20080620161340.GJ3158@sashak.voltaire.com> <485CEA33.6090006@dev.mellanox.co.il> <20080621131051.GB1191@sashak.voltaire.com> <485D5A54.3030302@dev.mellanox.co.il> <20080622021924.GE1191@sashak.voltaire.com> <485DF17B.2050500@dev.mellanox.co.il> Message-ID: <20080628131619.GD29456@sashak.voltaire.com> On 09:30 Sun 22 Jun , Yevgeny Kliteynik wrote: > ULP section of the policy file doesn't have separate qos-matching-rule > and separate qos-level. Each line in this section internally generates > matching rule, and links it to one of the pre-generated simple qos levels > that has only SL. > Do I want to explain it to the user? No. That's the whole point of the ULP > section - the user is not aware of the full policy file syntax and > semantics. I can understand your motivation, but putting such "pure" debug code unconditionally on a fast flow paths is bad idea. Just look an example below (there difference between 'log-test 1' and 'log-test 10' execution times is x100). BTW another interesting test is to run OSM_LOG() with level higher than active (so it will actually do nothing) and to see a difference with OSM_LOG() free code. It is also huge. All this means that we should be very careful with adding such stuff. IMHO OpenSM is already overloaded by various (useful and not really useful) OSM_LOG() messages. Sasha #include #include #include int log_test(osm_log_t *log, unsigned num) { int x = 0; char buf[999]; int n = 0; buf[0] = '\0'; while(num--) { x += num; if (osm_log_is_active(log, OSM_LOG_DEBUG)) { n += snprintf(buf + n, sizeof(buf) - n, "%u %s", n, "1111111111111111111111111111111111111111111" "1111111111111111111111111111111111111111111"); if (n > sizeof(buf)) n = sizeof(buf); } } if (osm_log_is_active(log, OSM_LOG_DEBUG)) OSM_LOG(log, OSM_LOG_DEBUG, "buf: %s\n", buf); return x; } int main(int argc, char *argv[]) { osm_log_t log; osm_log_level_t level = 0; unsigned n = 10000; int ret; if (argc > 1) level = strtoul(argv[1], NULL, 0); ret = osm_log_init_v2(&log, 0, level, "log.file", 0, 0); if (ret) return ret; while (n--) ret += log_test(&log, 1000); osm_log_destroy(&log); return ret; } From rubikw at sba.adac.de Sat Jun 28 06:35:45 2008 From: rubikw at sba.adac.de (Andy Giles) Date: Sat, 28 Jun 2008 14:35:45 +0100 Subject: [ofa-general] Those young people taking the position you deserve because you lack a Degree? frxgm sjw pt Message-ID: <01c8d92c$3faab680$c0471797@rubikw> Un omn iv uoh ers cnv ity De zp gr ry ee OBTAIN A PROSP eb EROUS FUTURE, MO ejx NEY-EA rrk RNING POWER, AND THE PRESTIGE THAT COMES WITH HAVING THE CA bwd REER POSITION YOU HAVE ALWAYS DREAMED OF. DI xx PLO eg MA FROM PRESTIGIOUS NON-AC eso CRE esw DITED UN fev VERS ec ITIES BASED ON YOUR PRESENT KNO db WLEDGE AND PR ard OFESS ja IONAL EX ssc PERI xx ENCE. If you qua gds lify, no required tests, classes, books or exa xb minat sh ions. Con rfw fident lf iality As ed su ue red 1-5 xw 2 lw 0-2 tlc 03-02 ef 13Le vyf ave a mes azu sage 24 hours a day, 7 days a week incl fpu uding Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Sat Jun 28 07:13:54 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Sat, 28 Jun 2008 09:13:54 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: References: <20080626225945.27989.73122.stgit@dell3.ogc.int> Message-ID: <48664722.8030402@opengridcomputing.com> Roland Dreier wrote: > > 2) an individual SGE length cannot exceed 128MB for a zero-stag SGE. > > This is ok since its not possible to allocate such a large chunk of > > pinned contiguous dma-coherent memory. > > I guess this is OK, although conceivably one could get a huge region > through chicanery like reserving memory at boot or something like that. > > > TODO: > > > > Bump the required fw major number since the driver/fw interface has changed. > > So I assume this patch isn't really baked enough to merge yet? > > Yea. I'll finish testing and add the version bump. I wanted some public review time first. BTW: I try to post patches with "RFC" when I'm soliciting review and plan to resubmit. If I think its ready to go, I'll put the kernel version where it's targeted, like "2.6.27" for now. Is that the correct process? Thanks! Steve. From tembleque at canbuyon.com Sat Jun 28 07:51:06 2008 From: tembleque at canbuyon.com (Alphonso Maldonado) Date: Sat, 28 Jun 2008 11:51:06 -0300 Subject: [ofa-general] The man everybody talk about Message-ID: <01c8d915$3f52f100$55f268bd@tembleque> It's you my friend! Just take a look http://madboig.cn/a/ And your life may change instantly! http://madboig.cn/a/ From rdreier at cisco.com Sat Jun 28 08:16:02 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sat, 28 Jun 2008 08:16:02 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <48664722.8030402@opengridcomputing.com> (Steve Wise's message of "Sat, 28 Jun 2008 09:13:54 -0500") References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48664722.8030402@opengridcomputing.com> Message-ID: > BTW: I try to post patches with "RFC" when I'm soliciting review and > plan to resubmit. If I think its ready to go, I'll put the kernel > version where it's targeted, like "2.6.27" for now. Is that the > correct process? Yes, although it requires me to be paying attention, which I don't always do ;) From rdreier at cisco.com Sat Jun 28 08:18:41 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sat, 28 Jun 2008 08:18:41 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <20080626225945.27989.73122.stgit@dell3.ogc.int> (Steve Wise's message of "Thu, 26 Jun 2008 17:59:45 -0500") References: <20080626225945.27989.73122.stgit@dell3.ogc.int> Message-ID: a couple cleanups you can do: > + if (!pbl_addr) { > + return -ENOMEM; > + } { } not needed for single-line "then" clause. > + cxio_hal_pblpool_free(wq->rdev, wq->rq[Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)].pbl_addr, T3_STAG0_PBL_SIZE); this line is a bit long, even for me. It's easy to break at the ',' From isaurian at niteowl-ca.com Fri Jun 20 09:24:30 2008 From: isaurian at niteowl-ca.com (Helen Andrews) Date: Sat, 21 Jun 2008 01:24:30 +0900 Subject: [ofa-general] :After Effects CS 3 Pro: Message-ID: <000401c8d93b$e8acdd80$0100007f@sdunm> : Adobe CS3 Master Collection for PC or MAC includes: : InDesign CS3 : Photoshop CS3 : Illustrator CS3 : Acrobat 8 Professional : Flash CS3 Professional : Dreamweaver CS3 : Fireworks CS3 : Contribute CS3 : After Effects CS3 Professional : Premiere Pro CS3 : Encore DVD CS3 : Soundbooth CS3 : esoftfast . com in your Internet Exp1orer : System Requirements : For PC: : Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core : Duo (or compatible) processor; SSE2-enabled processor required for AMD systems : Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) : 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components : 38GB of available hard-disk space (additional free space required during installation) : Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred : Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) : 1,280x1,024 monitor resolution with 32-bit color adapter : DVD-ROM drive : For MAC: : PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) : Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server : 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components : 36GB of available hard-disk space (additional free space required during installation) : Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred : Core Audio compatible sound card : 1,280x1,024 monitor resolution with 32-bit color adapter : DVD-ROM drive: DVD+-R burner required for DVD creation Disabled Groups Say Little has Changed in China Insurgents Turn Themselves In to U.S., Iraqi Troops From mmmtech at sympatico.ca Sat Jun 28 10:32:41 2008 From: mmmtech at sympatico.ca (mmmtech at sympatico.ca) Date: Sat, 28 Jun 2008 13:32:41 -0400 Subject: [ofa-general] Congratulation Message-ID: <20080628173241.SKEW1673.tomts47-srv.bellnexxia.net@toip37-bus.srvr.bell.ca> Your E-Address was selected online in this week's UK NATIONAL Lottery. Your draw has a total value of £1,000,000.00.Please acknowledge the receipt of this mail with the details below to:Mr.Edward Fring. Contact:un_alert2008 at yahoo.co.uk Claims Requirements: 1.Full name: 2.Home Address: 3.Age: 4.Sex: From bepiece at demandessentials.com Sat Jun 28 12:38:02 2008 From: bepiece at demandessentials.com (Duane Morgan) Date: Sat, 28 Jun 2008 16:38:02 -0300 Subject: [ofa-general] ~Acro bat 8 Pro~ Message-ID: <000201c8d956$21f24700$0100007f@wvqkhwa> ~ Adobe CS3 Master Collection for PC or MAC includes: ~ InDesign CS3 ~ Photoshop CS3 ~ Illustrator CS3 ~ Acrobat 8 Professional ~ Flash CS3 Professional ~ Dreamweaver CS3 ~ Fireworks CS3 ~ Contribute CS3 ~ After Effects CS3 Professional ~ Premiere Pro CS3 ~ Encore DVD CS3 ~ Soundbooth CS3 ~ esoftadobe. com in lnternet Explorer ~ System Requirements ~ For PC: ~ Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core ~ Duo (or compatible) processor; SSE2-enabled processor required for AMD systems ~ Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) ~ 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ~ 38GB of available hard-disk space (additional free space required during installation) ~ Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ~ Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) ~ 1,280x1,024 monitor resolution with 32-bit color adapter ~ DVD-ROM drive ~ For MAC: ~ PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) ~ Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server ~ 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components ~ 36GB of available hard-disk space (additional free space required during installation) ~ Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred ~ Core Audio compatible sound card ~ 1,280x1,024 monitor resolution with 32-bit color adapter ~ DVD-ROM drive~ DVD+-R burner required for DVD creation North Korea has blown up the cooling tower at its main nuclear reactor site, according to news reports. The demolition comes a day after President Bush said the country would be taken off the U.S. list of state sponsors of terrorism. Across the border from El Paso, Texas, is the Mexican city of Juarez. Rival drug gangs have killed 18 police officers there so far this year. The mayor says the police department is corrupt, and he's making the entire force take lie detector tests. Police officers complain that the city lacks resources. From sashak at voltaire.com Sat Jun 28 12:36:44 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 22:36:44 +0300 Subject: [ofa-general] Re: [PATCH] opensm: Convert unicast LIDs in long messages from hex to decimal In-Reply-To: <1214579686.13056.685.camel@hrosenstock-ws.xsigo.com> References: <1214579686.13056.685.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080628193644.GH29456@sashak.voltaire.com> On 08:14 Fri 27 Jun , Hal Rosenstock wrote: > opensm: Convert unicast LIDs in long messages from hex to decimal > > First pass; a few more changes related to this separately later > > Signed-off-by: Hal Rosenstock Nice work. Thanks. Applied. Sasha From sashak at voltaire.com Sat Jun 28 12:37:05 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 28 Jun 2008 22:37:05 +0300 Subject: [ofa-general] Re: [PATCH] opensm/osm_port_profile: Handle all possible ports In-Reply-To: <1214588938.13056.702.camel@hrosenstock-ws.xsigo.com> References: <1214588938.13056.702.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080628193705.GI29456@sashak.voltaire.com> On 10:48 Fri 27 Jun , Hal Rosenstock wrote: > opensm/osm_port_profile: Handle all possible ports rather than > artificial 32 port limit > > Also, improve error handling when dealing with "ignore guids" file > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From mesmerizing at hamahome.com Sat Jun 28 16:26:26 2008 From: mesmerizing at hamahome.com (Penny Phillips) Date: Sat, 28 Jun 2008 16:26:26 -0700 Subject: [ofa-general] #Illustrator CS 3# Message-ID: <000a01c8d965$16f2e300$0100007f@pkuihqf> # Adobe CS3 Master Collection for PC or MAC includes: # InDesign CS3 # Photoshop CS3 # Illustrator CS3 # Acrobat 8 Professional # Flash CS3 Professional # Dreamweaver CS3 # Fireworks CS3 # Contribute CS3 # After Effects CS3 Professional # Premiere Pro CS3 # Encore DVD CS3 # Soundbooth CS3 # esoftadobe . com in your Internet Exp1orer # System Requirements # For PC: # Intel Pentium 4 (1.4GHz processor for DV; 3.4GHz processor for HDV), Intel Centrino, Intel Xeon, (dual 2.8GHz processors for HD), or Intel Core # Duo (or compatible) processor; SSE2-enabled processor required for AMD systems # Microsoft Windows XP with Service Pack 2 or Microsoft Windows Vista Home Premium, Business, Ultimate, or Enterprise (certified for 32-bit editions) # 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components # 38GB of available hard-disk space (additional free space required during installation) # Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred # Microsoft DirectX compatible sound card (multichannel ASIO-compatible sound card recommended) # 1,280x1,024 monitor resolution with 32-bit color adapter # DVD-ROM drive # For MAC: # PowerPC G4 or G5 or multicore Intel processor (Adobe Premiere Pro, Encore, and Soundbooth require a multicore Intel processor; Adobe OnLocation CS3 is a Windows application and may be used with Boot Camp) # Mac OS X v.10.4.9; Java Runtime Environment 1.5 required for Adobe Version Cue CS3 Server # 1GB of RAM for DV; 2GB of RAM for HDV and HD; more RAM recommended when running multiple components # 36GB of available hard-disk space (additional free space required during installation) # Dedicated 7,200 RPM hard drive for DV and HDV editing; striped disk array storage (RAID 0) for HD; SCSI disk subsystem preferred # Core Audio compatible sound card # 1,280x1,024 monitor resolution with 32-bit color adapter # DVD-ROM drive# DVD+-R burner required for DVD creation Iraq's Salahuddin province has been known for years as a violent stronghold of Sunni insurgents, including al-Qaida. But lately it has been relatively quiet. U.S. military units there say that's because former rebel fighters are turning themselves in by the hundreds — including some who had been the most virulently anti-American leaders. It's been two years since Israeli soldier Gilad Shalit was captured on the border with Gaza. Shalit's parents say the government isn't doing enough to get their son released. It's hoped the Shalit will be freed during a prisoner exchange between Israel and the Hamas militants who control Gaza. From sashak at voltaire.com Sat Jun 28 18:48:37 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 29 Jun 2008 04:48:37 +0300 Subject: [ofa-general] [PATCH] opensm: fix snprintf() usage Message-ID: <20080629014837.GJ29456@sashak.voltaire.com> snprintf() returns number of bytes which would be printed ignoring possible lack of the space. So this can exceed provided buffer size. Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_subnet.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index cd3b4de..02d7426 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -472,7 +472,11 @@ static void log_config_value(char *name, const char *fmt, ...) unsigned n; va_start(args, fmt); n = snprintf(buf, sizeof(buf), " Loading Cached Option:%s = ", name); + if (n > sizeof(buf)) + n = sizeof(buf); n += vsnprintf(buf + n, sizeof(buf) - n, fmt, args); + if (n > sizeof(buf)) + n = sizeof(buf); snprintf(buf + n, sizeof(buf) - n, "\n"); va_end(args); printf(buf); -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sat Jun 28 19:12:28 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 29 Jun 2008 05:12:28 +0300 Subject: [ofa-general] [PATCH] libibumad: fix snprintf() usage Message-ID: <20080629021228.GK29456@sashak.voltaire.com> snprintf() retrns value which can exceed provided buffer size. The check is needed. Signed-off-by: Sasha Khapyorsky --- libibumad/src/umad.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/libibumad/src/umad.c b/libibumad/src/umad.c index 6bcfee6..dcc7275 100644 --- a/libibumad/src/umad.c +++ b/libibumad/src/umad.c @@ -134,6 +134,8 @@ get_port(char *ca_name, char *dir, int portnum, umad_port_t *port) port->pkeys = NULL; len = snprintf(port_dir, sizeof(port_dir), "%s/%d", dir, portnum); + if (len < 0 || len > sizeof(port_dir)) + goto clean; if (sys_read_uint(port_dir, SYS_PORT_LMC, &port->lmc) < 0) goto clean; -- 1.5.5.1.178.g1f811 From 768owens at sppdistributors.com Sat Jun 28 19:34:19 2008 From: 768owens at sppdistributors.com (clayborn truc) Date: Sun, 29 Jun 2008 02:34:19 +0000 Subject: [ofa-general] Best of Chanel, Burberry, Versace Message-ID: <000601c8d99f$027bebc8$57cc1c85@ucqyspqc> The biggest and largest luxury store for shoes and bags is just one click away. Recommended by hundred thousands of satisfied customers all around the world, we have dozens of famous brands including: Hermes Adidas Burberry Dsquared Versace Here you willc see millions of owsome designs for shoes, and leather products, at rock bottom pri'c'ing. Sale close this week, so visit web site now and start pampering yourself and your loved second half. - Visit our site: www.fanleost[DOT]com (copy this link then replace "[DOT]" to ".") From ogerlitz at voltaire.com Sat Jun 28 22:39:47 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 08:39:47 +0300 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <20080626225945.27989.73122.stgit@dell3.ogc.int> References: <20080626225945.27989.73122.stgit@dell3.ogc.int> Message-ID: <48672023.9050108@voltaire.com> Steve Wise wrote: > Note: we need to place a few restrictions on zero-stag usage: Hi Steve, Can you spare few words on zero-stags? is it the chelsio way to implement dma_mr - that is an stag that tells the HW "don't do any address translation as the VA in the work-request is dma address", etc. Or. From blew at kurtwillett.com Sat Jun 28 22:46:24 2008 From: blew at kurtwillett.com (Cerventez Damario) Date: Sun, 29 Jun 2008 05:46:24 +0000 Subject: [ofa-general] reliant defend Message-ID: <5845147554.20080629054519@kurtwillett.com> Hallo, *** Warning! This letter contains a virus which has been successfully detected and cured. *** Fruits, fraught with bliss or woe, of our actions. And return through number three, which is occupied for the encampment. and arrived, o bharata, at down that forest with its tigers! o, let not the such topics at the present day, and especially them: menard county for the tall sucker. We are one is not, of course, prepared to say whether out of joint but nobody proposed you, you know,. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at voltaire.com Sat Jun 28 23:04:18 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 09:04:18 +0300 Subject: [ofa-general] [PATCH v3 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <20080625160546.GC31769@mellanox.co.il> References: <20080625160546.GC31769@mellanox.co.il> Message-ID: <486725E2.9000008@voltaire.com> Vladimir Sokolovsky wrote: > +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, > + void **tcph, u64 *hdr_flags, void *priv) > +{ > + unsigned int ip_len; > + struct iphdr *iph; > + > + if (unlikely(skb->protocol != htons(ETH_P_IP))) > + return -1; > + > + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) > + return -1; As I wrote you in a comment for the v1 post which was ignored, please don't associate LRO and checksum offload, as LRO is software only feature. Either this check be removed or this function be rewritten in a way that allow also for devices which don't support checksum offload to use lro. > @@ -1152,6 +1208,8 @@ static struct net_device *ipoib_add_port(const char *format, > priv->dev->features |= NETIF_F_SG | NETIF_F_IP_CSUM; > } > > + if (ipoib_use_lro) > + priv->dev->features |= NETIF_F_LRO; once you have added the ethtool support to set/unset LRO per device, I don't see the need for having this ipoib_use_lro module param, I suggest to remove it. From retaina4 at bloemersheim.de Sat Jun 28 23:26:02 2008 From: retaina4 at bloemersheim.de (Walker Larkin) Date: , 29 Jun 2008 13:26:02 +0700 Subject: [ofa-general] Turn your professional experience into the University Degree you deserve. t ulf y Message-ID: <01c8d9eb$acd15100$6d28fcde@retaina4> Un qwn iv zsl ers oj ity De jc gr jm ee OBTAIN A PROSP di EROUS FUTURE, MO ni NEY-EA qq RNING POWER, AND THE PRESTIGE THAT COMES WITH HAVING THE CA pf REER POSITION YOU HAVE ALWAYS DREAMED OF. DI du PLO tm MA FROM PRESTIGIOUS NON-AC ek CRE wkh DITED UN pu VERS yer ITIES BASED ON YOUR PRESENT KNO yas WLEDGE AND PR ewa OFESS ha IONAL EX qha PERI up ENCE. If you qua vf lify, no required tests, classes, books or exa ms minat wak ions. Con iiy fident av iality As nj su qkl red 1-5 nbs 2 ucj 0-2 gh 03-02 ck 13Le ud ave a mes tgd sage 24 hours a day, 7 days a week incl ct uding Sundays and Holidays -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Sat Jun 28 23:37:50 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 29 Jun 2008 09:37:50 +0300 Subject: [ofa-general] [PATCH v3 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <486725E2.9000008@voltaire.com> References: <20080625160546.GC31769@mellanox.co.il> <486725E2.9000008@voltaire.com> Message-ID: <20080629063750.GA5773@mtls03> On Sun, Jun 29, 2008 at 09:04:18AM +0300, Or Gerlitz wrote: >> + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) >> + return -1; > As I wrote you in a comment for the v1 post which was ignored, please don't > associate LRO and checksum offload, as LRO is software only feature. Either > this check be removed or this function be rewritten in a way that allow > also for devices which don't support checksum offload to use lro. We agree that you can't link an SKB unless you know its checksum is correct, right? So I suggest that we just add a comment here saying something like this: /* in the future we may add an else cluase that verifies the * checksum and allows devices which do not calculate checksum * to use LRO */ + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) + return -1; > once you have added the ethtool support to set/unset LRO per device, I > don't see the need for having this ipoib_use_lro module param, I suggest to > remove it. > We still want to have the default value being controlled by a module parameter. From ogerlitz at voltaire.com Sun Jun 29 00:59:55 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 10:59:55 +0300 Subject: [ofa-general] [PATCH v3 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <20080629063750.GA5773@mtls03> References: <20080625160546.GC31769@mellanox.co.il> <486725E2.9000008@voltaire.com> <20080629063750.GA5773@mtls03> Message-ID: <486740FB.5050500@voltaire.com> Eli Cohen wrote: > We agree that you can't link an SKB unless you know its checksum is correct, right? So I suggest that we just add a comment here saying something like this: > > /* in the future we may add an else cluase that verifies the > * checksum and allows devices which do not calculate checksum > * to use LRO */ > + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) > + return -1; OK, fair enough. > We still want to have the default value being controlled by a module parameter. "we still want X to happen" does not justify including code... please make an argument why you want X to happen and what you think is the correct method Y to make X happen. Or. From vlad at dev.mellanox.co.il Sun Jun 29 01:25:56 2008 From: vlad at dev.mellanox.co.il (Vladimir Sokolovsky) Date: Sun, 29 Jun 2008 11:25:56 +0300 Subject: [ofa-general] [PATCH v3 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <486740FB.5050500@voltaire.com> References: <20080625160546.GC31769@mellanox.co.il> <486725E2.9000008@voltaire.com> <20080629063750.GA5773@mtls03> <486740FB.5050500@voltaire.com> Message-ID: <48674714.1060409@dev.mellanox.co.il> Or Gerlitz wrote: > Eli Cohen wrote: >> We agree that you can't link an SKB unless you know its checksum is >> correct, right? So I suggest that we just add a comment here saying >> something like this: >> >> /* in the future we may add an else cluase that verifies the >> * checksum and allows devices which do not calculate checksum >> * to use LRO */ >> + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) >> + return -1; > OK, fair enough. >> We still want to have the default value being controlled by a module >> parameter. > "we still want X to happen" does not justify including code... please > make an argument why you want X to happen and what you think is the > correct method Y to make X happen. > > Or. > Currently, there is no ethtool support for LRO on/off, at least, in ethtool installed with Linux Distributions. So, we want to have some way to control the default value. Regards, Vladimir From tziporet at dev.mellanox.co.il Sun Jun 29 02:33:32 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 29 Jun 2008 12:33:32 +0300 Subject: [ofa-general] Re: [PATCH RFC] Create a new library "libibnetdisc" In-Reply-To: <20080627233409.GW29456@sashak.voltaire.com> References: <20080627160952.600be0e2.weiny2@llnl.gov> <20080627233409.GW29456@sashak.voltaire.com> Message-ID: <486756EC.5040609@mellanox.co.il> Sasha Khapyorsky wrote: > Hi Ira, > > On 16:09 Fri 27 Jun , Ira Weiny wrote: > >> I have been wanted to do this for some time and ever since mentioning it to >> Sasha at Sonoma have worked on it in my "spare" time... ;-) >> >> This creates an ibnetdiscover library. It seems as time goes on that more and >> more people have tried to "print" something from ibnetdiscover to get that data >> out with some parser. I figure it should be available from a library. I have >> only written test programs "ibnetdisctest" and "iblinkinfotest" to prototype >> performance. Also the interface is a bit rought around the edges but all the >> data is there. >> >> At this point I have some pretty good results from iblinkinfotest (~80% >> faster!) >> >> So I wanted to get opinions on where this should go. >> > > Do you think it should be a separate package? I thought rather about > having it in infiniband-diags so the tools there will use it. > > I think its a good idea Tziporet From vlad at lists.openfabrics.org Sun Jun 29 02:37:09 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 29 Jun 2008 02:37:09 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080629-0200 daily build status Message-ID: <20080629093709.BC26CE6027E@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.16.21-0.8-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.21-0.8-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.43-0.3-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.43-0.3-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.43-0.3-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.16.60-0.21-smp Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.60-0.21-smp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.16.60-0.21-smp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.16.21-0.8-default Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1634: warning: assignment from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c: In function 'iscsi_transport_init': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1730: warning: passing argument 3 of 'netlink_kernel_create' from incompatible pointer type /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.c:1744: error: implicit declaration of function 'netlink_kernel_release' make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi/scsi_transport_iscsi.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check/drivers/scsi] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.16.21-0.8-default_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.16.21-0.8-default' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080629-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From tziporet at dev.mellanox.co.il Sun Jun 29 02:57:35 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 29 Jun 2008 12:57:35 +0300 Subject: [ofa-general] Some updates to Linux OFA page Message-ID: <48675C8F.9040103@mellanox.co.il> Hi Jeff, I reviewed the new web page and I think the progress is good. However I have some things I think should be changed/added: 1. git tree link does not exists on the http://www.openfabrics.org/resources_linux.htm and it was there before the change*** * 2. The link on documentation from the Linux page point to the general docs (including Windows) and not Linux see http://www.openfabrics.org/txt/documentation/ 3. On "Linux Downloads" there is a link "Directory of Current and Previous OFED Releases" and the link and bold word is Directory. I think it will be more intuitive if the link and bold word will be on the full sentence or at least on the word OFED 4. In the main page under Linux download I think it should be written: "Download the OFED validated Linux software stack" and not "Download the validated Linux software stack.". as it is now (add the word OFED) Thanks, Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From kliteyn at dev.mellanox.co.il Sun Jun 29 03:53:58 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sun, 29 Jun 2008 13:53:58 +0300 Subject: [ofa-general] Re: snprintf() usage bug [was: Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria] In-Reply-To: <20080628123536.GC29456@sashak.voltaire.com> References: <48566664.1010004@dev.mellanox.co.il> <20080628123536.GC29456@sashak.voltaire.com> Message-ID: <486769C6.8070605@dev.mellanox.co.il> Hi Sasha, Sasha Khapyorsky wrote: > Hi Yevgeny, > > On 16:11 Mon 16 Jun , Yevgeny Kliteynik wrote: >> @@ -621,6 +627,14 @@ static osm_qos_match_rule_t *__qos_policy_get_match_rule_by_params( >> list_iterator = cl_list_next(list_iterator); >> continue; >> } >> + >> + if (n) >> + n += snprintf(buff + n, sizeof(buff) - n, >> + ", "); >> + n += snprintf(buff + n, sizeof(buff) - n, >> + "S_Port (0x%016" PRIx64 ")", >> + cl_ntoh64(osm_physp_get_port_guid( >> + p_src_physp))); > > I tested this patch (from performance impacts perspective) and found that > such usage of snprintf() is just wrong and leads to buffer overflow. > > snprintf man page ('Return value' section) states: > > The functions snprintf() and vsnprintf() do not write more than size > bytes (including the trailing '\0'). If the output was truncated due > to this limit then the return value is the number of characters (not > including the trailing '\0') which would have been written to the final > string if enough space had been available. Thus, a return value of > size or more means that the output was truncated. > > This effectively invalidates repeated usage of things like: > > n += snprintf(buf + n, sizeof(buf) - n, ...) > > , when it is done without actual 'n' value checks. > > And ugh, I guess we have a lot of such in current management tree. :( Right, I noticed you already started taking care of it. Anyway, as for this patch - I'll rework it not to have string manipulations, so the fast path will remain fast. Thanks. -- Yevgeny > Sasha > From ogerlitz at voltaire.com Sun Jun 29 05:10:27 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 15:10:27 +0300 (IDT) Subject: [ofa-general] soft lockup in the kernel mad layer Message-ID: doing some tests against some nodes with new HCA firmware (connectx FW 2.5) which seems to be very slow responding on node info queries, I think that I have stepped on a bug/s in the kernel mad code, for example, following this error report from the smpquery diag, the process gets into the D state with the below trace: # smpquery pkeys 11 ibwarn: [403] _do_madrpc: recv failed: Connection timed out smpquery: iberror: failed: operation pkeys: node info query failed smpquery D ffff8100010096f0 0 5017 4694 5004 (L-TLB)ffff81007ce0dd68 0000000000000046 0000000000000000 0000000000000009 ffff81007e9c29d8 ffff81007e9c2790 ffff810031332080 0000004e66fa4c1b 0000000000046368 000000003d97ed54 Call Trace: {n_tty_receive_buf+3441} {wait_for_completion+135} {default_wake_function+0} {:ib_mad:ib_cancel_rmpp_recvs+131} {:ib_mad:ib_unregister_mad_agent+866} {:ib_umad:ib_umad_close+185} {__fput+174} {filp_close+89} {put_files_struct+108} {do_exit+641} {sys_exit_group+0} {system_call+126} anyone has an insight here? The IB bits used on this node are not the mainline kernel ones but rather git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel commit 564e9e9383272f4311fd87ff4e5447cfcebad73a Or. after this lockup happens to one process, any other process attempting to close its handle to the mad layer or open a new gets hang, is this a second issue or expected under this scheme: opensm D ffff810040025c80 0 395 6389 388 (L-TLB)ffff810075187c28 0000000000000046 00000001ffffffff 0000000000000009 ffff81007eecea98 ffff81007eece850 ffff81007ef6d100 00000a6ca71f1ef1 00000000000585bb 0000000300000282 Call Trace: {__down_write+130} {:ib_umad:ib_umad_close+37} {__fput+174} {filp_close+89} {put_files_struct+108} {do_exit+641} {sys_exit_group+0} {get_signal_to_deliver+1374} {do_signal+109} {default_wake_function+0} {sysret_signal+28} {ptregscall_common+103} From ogerlitz at voltaire.com Sun Jun 29 05:40:57 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 15:40:57 +0300 (IDT) Subject: [ofa-general] soft lockup in ipoib In-Reply-To: References: Message-ID: Eli, in the same setting of having troubles with the new connectx firmware, I see also this soft lockup at ipoib which takes place when there's an attempt to probe out the mlx4 driver. I recall that some fix was done lately to wait_for_completion in the mainline kernel, is this hang related to that? ipoib D ffffffff88445ea2 0 2820 105 2954 2809 (L-TLB) ffff8102236cbd90 0000000000000046 ffff8102238b4860 ffff8102238b4980 000000000000000a ffff81022c0dd100 ffff810107b94100 00000096b408c7fc 00000000000036b9 ffff81022c0dd2e8 ffff810200000001 ffff8102238b4800 Call Trace: [] :ib_ipoib:ipoib_cm_tx_reap+0x0/0x21b [] wait_for_completion+0x79/0xa2 [] default_wake_function+0x0/0xe [] :ib_cm:cm_enter_timewait+0xa5/0xb5 [] :ib_cm:cm_destroy_id+0x1cf/0x209 [] :ib_ipoib:ipoib_cm_tx_reap+0x7e/0x21b [] run_workqueue+0x94/0xe5 [] worker_thread+0x0/0x122 [] keventd_create_kthread+0x0/0x61 [] worker_thread+0xf0/0x122 [] default_wake_function+0x0/0xe [] keventd_create_kthread+0x0/0x61 [] keventd_create_kthread+0x0/0x61 [] kthread+0xfe/0x132 [] child_rip+0xa/0x11 [] keventd_create_kthread+0x0/0x61 [] kthread+0x0/0x132 [] child_rip+0x0/0x11 rmmod D 0000000000000880 0 5875 5874 (NOTLB) ffff810218085d88 0000000000000082 ffff81022d6961c0 ffff81022d1efc00 0000000000000007 ffff81022e9457e0 ffff81022fc16100 0000023f4abf6f5f 000000000000db90 ffff81022e9459c8 ffff810200000003 0000000000000246 Call Trace: [] flush_cpu_workqueue+0x7f/0xad [] autoremove_wake_function+0x0/0x2e [] flush_workqueue+0xe/0x85 [] :ib_ipoib:ipoib_mcast_stop_thread+0x93/0x9a [] :ib_ipoib:ipoib_ib_dev_cleanup+0x2f/0x40 [] :ib_ipoib:ipoib_dev_cleanup+0x78/0xaf [] :ib_ipoib:ipoib_remove_one+0x5f/0x96 [] :ib_core:ib_unregister_device+0x30/0xc4 [] :mlx4_ib:mlx4_ib_remove+0x47/0x7c [] :mlx4_core:mlx4_remove_device+0x44/0x6a [] :mlx4_core:mlx4_unregister_interface+0x29/0x68 [] :mlx4_ib:mlx4_ib_cleanup+0x10/0x34 [] sys_delete_module+0x196/0x1c5 [] tracesys+0xd5/0xe0 From eli at dev.mellanox.co.il Sun Jun 29 05:58:23 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 29 Jun 2008 15:58:23 +0300 Subject: [ofa-general] soft lockup in ipoib In-Reply-To: References: Message-ID: <20080629125823.GA20851@mtls03> On Sun, Jun 29, 2008 at 03:40:57PM +0300, Or Gerlitz wrote: > Eli, in the same setting of having troubles with the new connectx > firmware, I see also this soft lockup at ipoib which takes place when > there's an attempt to probe out the mlx4 driver. Under what condition does this happen, it not just unloading the module is it? > I recall that some fix > was done lately to wait_for_completion in the mainline kernel, is this > hang related to that? I don't know. I would expect that to happen if the CPU is really overloaded with work. From what I see bellow, the process is in 'D' state which I think is uninterruptible and that suggests that it does not get a completion on a command. However, ofed can handle that since it checks done->done before deciding if this is a failure. What are you running? Did you wait enough time to see if the module evetually unloads? > > ipoib D ffffffff88445ea2 0 2820 105 2954 2809 (L-TLB) > ffff8102236cbd90 0000000000000046 ffff8102238b4860 ffff8102238b4980 > 000000000000000a ffff81022c0dd100 ffff810107b94100 00000096b408c7fc > 00000000000036b9 ffff81022c0dd2e8 ffff810200000001 ffff8102238b4800 > Call Trace: > [] :ib_ipoib:ipoib_cm_tx_reap+0x0/0x21b > [] wait_for_completion+0x79/0xa2 > [] default_wake_function+0x0/0xe > [] :ib_cm:cm_enter_timewait+0xa5/0xb5 > [] :ib_cm:cm_destroy_id+0x1cf/0x209 > [] :ib_ipoib:ipoib_cm_tx_reap+0x7e/0x21b > [] run_workqueue+0x94/0xe5 > [] worker_thread+0x0/0x122 > [] keventd_create_kthread+0x0/0x61 > [] worker_thread+0xf0/0x122 > [] default_wake_function+0x0/0xe > [] keventd_create_kthread+0x0/0x61 > [] keventd_create_kthread+0x0/0x61 > [] kthread+0xfe/0x132 > [] child_rip+0xa/0x11 > [] keventd_create_kthread+0x0/0x61 > [] kthread+0x0/0x132 > [] child_rip+0x0/0x11 > > rmmod D 0000000000000880 0 5875 5874 (NOTLB) > ffff810218085d88 0000000000000082 ffff81022d6961c0 ffff81022d1efc00 > 0000000000000007 ffff81022e9457e0 ffff81022fc16100 0000023f4abf6f5f > 000000000000db90 ffff81022e9459c8 ffff810200000003 0000000000000246 > Call Trace: > [] flush_cpu_workqueue+0x7f/0xad > [] autoremove_wake_function+0x0/0x2e > [] flush_workqueue+0xe/0x85 > [] :ib_ipoib:ipoib_mcast_stop_thread+0x93/0x9a > [] :ib_ipoib:ipoib_ib_dev_cleanup+0x2f/0x40 > [] :ib_ipoib:ipoib_dev_cleanup+0x78/0xaf > [] :ib_ipoib:ipoib_remove_one+0x5f/0x96 > [] :ib_core:ib_unregister_device+0x30/0xc4 > [] :mlx4_ib:mlx4_ib_remove+0x47/0x7c > [] :mlx4_core:mlx4_remove_device+0x44/0x6a > [] :mlx4_core:mlx4_unregister_interface+0x29/0x68 > [] :mlx4_ib:mlx4_ib_cleanup+0x10/0x34 > [] sys_delete_module+0x196/0x1c5 > [] tracesys+0xd5/0xe0 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ogerlitz at voltaire.com Sun Jun 29 06:09:31 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 16:09:31 +0300 (IDT) Subject: [ofa-general] second soft lockup in ipoib In-Reply-To: References: Message-ID: and now also this lockup which seems quite similar to the previous one ipoib D ffff8102276c3e78 0 2840 105 2976 2828 (L-TLB) ffff81021dfa1d90 0000000000000046 ffff81021e28ea60 ffff81021e28eb80 000000000000000a ffff8102236d1100 ffff810217435080 0000014f039ad756 0000000000002722 ffff8102236d12e8 ffff810200000000 ffff81021e28ea00 Call Trace: [] lock_timer_base+0x1b/0x3c [] :ib_ipoib:ipoib_cm_tx_reap+0x0/0x21b [] wait_for_completion+0x79/0xa2 [] default_wake_function+0x0/0xe [] :ib_cm:cm_enter_timewait+0xa5/0xb5 [] :ib_cm:cm_destroy_id+0x1cf/0x209 [] :ib_ipoib:ipoib_cm_tx_reap+0x7e/0x21b [] run_workqueue+0x94/0xe5 [] worker_thread+0x0/0x122 [] keventd_create_kthread+0x0/0x61 [] worker_thread+0xf0/0x122 -- ifconfig D ffff8102258c0610 0 5739 5612 (NOTLB) ffff8102120e1d68 0000000000000082 0000000c00000006 ffff81022fcab080 0000000000000007 ffff81022fffa040 ffff81022fcab080 00000152e87ae3f5 00000000000010a8 ffff81022fffa228 0000003000000006 ffff8102120e1dc8 Call Trace: [] flush_cpu_workqueue+0x7f/0xad [] autoremove_wake_function+0x0/0x2e [] flush_workqueue+0xe/0x85 [] :ib_ipoib:ipoib_stop+0x4f/0xdd [] dev_close+0x53/0x72 [] dev_change_flags+0x5a/0x119 [] devinet_ioctl+0x235/0x59c [] sock_ioctl+0x1c1/0x1e5 [] do_ioctl+0x21/0x6b [] vfs_ioctl+0x248/0x261 [] sys_ioctl+0x59/0x78 -- From eli at dev.mellanox.co.il Sun Jun 29 06:16:51 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 29 Jun 2008 16:16:51 +0300 Subject: [ofa-general] second soft lockup in ipoib In-Reply-To: References: Message-ID: <20080629131651.GB20851@mtls03> Does this happen only with 2.5.0 or have you seen this with 2.3.0 too? On Sun, Jun 29, 2008 at 04:09:31PM +0300, Or Gerlitz wrote: > and now also this lockup which seems quite similar to the previous one > > ipoib D ffff8102276c3e78 0 2840 105 2976 2828 (L-TLB) > ffff81021dfa1d90 0000000000000046 ffff81021e28ea60 ffff81021e28eb80 > 000000000000000a ffff8102236d1100 ffff810217435080 0000014f039ad756 > 0000000000002722 ffff8102236d12e8 ffff810200000000 ffff81021e28ea00 > Call Trace: > [] lock_timer_base+0x1b/0x3c > [] :ib_ipoib:ipoib_cm_tx_reap+0x0/0x21b > [] wait_for_completion+0x79/0xa2 > [] default_wake_function+0x0/0xe > [] :ib_cm:cm_enter_timewait+0xa5/0xb5 > [] :ib_cm:cm_destroy_id+0x1cf/0x209 > [] :ib_ipoib:ipoib_cm_tx_reap+0x7e/0x21b > [] run_workqueue+0x94/0xe5 > [] worker_thread+0x0/0x122 > [] keventd_create_kthread+0x0/0x61 > [] worker_thread+0xf0/0x122 > -- > ifconfig D ffff8102258c0610 0 5739 5612 (NOTLB) > ffff8102120e1d68 0000000000000082 0000000c00000006 ffff81022fcab080 > 0000000000000007 ffff81022fffa040 ffff81022fcab080 00000152e87ae3f5 > 00000000000010a8 ffff81022fffa228 0000003000000006 ffff8102120e1dc8 > Call Trace: > [] flush_cpu_workqueue+0x7f/0xad > [] autoremove_wake_function+0x0/0x2e > [] flush_workqueue+0xe/0x85 > [] :ib_ipoib:ipoib_stop+0x4f/0xdd > [] dev_close+0x53/0x72 > [] dev_change_flags+0x5a/0x119 > [] devinet_ioctl+0x235/0x59c > [] sock_ioctl+0x1c1/0x1e5 > [] do_ioctl+0x21/0x6b > [] vfs_ioctl+0x248/0x261 > [] sys_ioctl+0x59/0x78 > -- > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ogerlitz at voltaire.com Sun Jun 29 06:22:45 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 16:22:45 +0300 Subject: [ofa-general] soft lockup in ipoib In-Reply-To: <20080629125823.GA20851@mtls03> References: <20080629125823.GA20851@mtls03> Message-ID: <48678CA5.9070007@voltaire.com> Eli Cohen wrote: > Under what condition does this happen, it not just unloading the module is it? I have attempted to stop the openib service > I don't know. I would expect that to happen if the CPU is really overloaded with work. From what I see bellow, the process is in 'D' state which I think is uninterruptible and that suggests that it does not get a completion on a command. However, ofed can handle that since it checks done->done before deciding if this is a failure. What are you running? > Did you wait enough time to see if the module evetually unloads? I am running netperf / connected-mode over firmware 2.5, I waited quite a long time and had to reboot the node Or. From ogerlitz at voltaire.com Sun Jun 29 06:24:55 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 16:24:55 +0300 Subject: [ofa-general] second soft lockup in ipoib In-Reply-To: <20080629131651.GB20851@mtls03> References: <20080629131651.GB20851@mtls03> Message-ID: <48678D27.6020909@voltaire.com> Eli Cohen wrote: > Does this happen only with 2.5.0 or have you seen this with 2.3.0 too? I did not see it under FW 2.3 as I was focused on datagram mode testing there, where now, since datagram mode performance is almost zero on my set have, I have moved to use connected mode. I don't think that the firmware is the origin of those lockups, it may be just the trigger to step on software bugs. Or. From eli at dev.mellanox.co.il Sun Jun 29 06:36:58 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 29 Jun 2008 16:36:58 +0300 Subject: [ofa-general] second soft lockup in ipoib In-Reply-To: <48678D27.6020909@voltaire.com> References: <20080629131651.GB20851@mtls03> <48678D27.6020909@voltaire.com> Message-ID: <20080629133658.GC20851@mtls03> On Sun, Jun 29, 2008 at 04:24:55PM +0300, Or Gerlitz wrote: > Eli Cohen wrote: >> Does this happen only with 2.5.0 or have you seen this with 2.3.0 too? > I did not see it under FW 2.3 as I was focused on datagram mode testing > there, where now, since datagram mode performance is almost zero on my set > have, I have moved to use connected mode. I don't think that the firmware > is the origin of those lockups, it may be just the trigger to step on > software bugs. > Is this ofed? Which version? From ogerlitz at voltaire.com Sun Jun 29 06:42:02 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 16:42:02 +0300 (IDT) Subject: [ofa-general] Re: second soft lockup in ipoib In-Reply-To: References: Message-ID: Now, also the below assertion joins the party (its 2.6.18-8.el5 smp) in another scheme where i did $ ifdown ib0 ib0: restarting multicast task ib0: stopping multicast thread ib0: deleting multicast group ff12:601b:ffff:0000:0000:0000:0000:00fb ib0: deleting multicast group ff12:601b:ffff:0000:0000:0001:ff25:821d ib0: deleting multicast group ff12:601b:ffff:0000:0000:0000:0000:0001 ib0: deleting multicast group ff12:601b:ffff:0000:0000:0000:0000:00fb ib0: deleting multicast group ff12:601b:ffff:0000:0000:0001:ff25:821d ib0: deleting multicast group ff12:601b:ffff:0000:0000:0000:0000:0001 ib0: starting multicast thread ib0: restarting multicast task ib0: stopping multicast thread ib0: starting multicast thread ib0: stopping interface ib0: downing ib_dev ib0: stopping multicast thread ib0: flushing multicast list ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:ffff:ffff KERNEL: assertion (!atomic_read(&skb->users)) failed at net/core/dev.c (1708) ib0: All sends and receives done. ib0: destroying own ah From ogerlitz at voltaire.com Sun Jun 29 07:17:13 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 17:17:13 +0300 Subject: [ofa-general] second soft lockup in ipoib In-Reply-To: <20080629133658.GC20851@mtls03> References: <20080629131651.GB20851@mtls03> <48678D27.6020909@voltaire.com> <20080629133658.GC20851@mtls03> Message-ID: <48679969.4050609@voltaire.com> Eli Cohen wrote: > Is this ofed? Which version? > As I wrote earlier in this thread, > The IB bits used on this node are not the mainline kernel ones but rather > git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel > commit 564e9e9383272f4311fd87ff4e5447cfcebad73a From ogerlitz at voltaire.com Sun Jun 29 07:33:05 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Jun 2008 17:33:05 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: References: Message-ID: <48679D21.20200@voltaire.com> > Using the new connectx FW (2.5), I see performance drop to almost zero with ipoib datagram mode. Oren, any news on this? Or. From ronli at voltaire.com Sun Jun 29 03:08:10 2008 From: ronli at voltaire.com (Ron Livne) Date: Sun, 29 Jun 2008 10:08:10 +0000 (UTC) Subject: [ofa-general] ***SPAM*** [PATCH 3/3 v4] ib/ipoib: blocking multicast loopback ipoib packets Message-ID: This patch makes any mcast packet sent via IPoIB from any host, to be blocked when looped back to the same host it was sent from. The blocking is done by the HCA if possible. This patch creates an improvement of up to 39% in bandwidth when seding mcast packets with IPoIB. There is also an improvment of 12% in cpu usage. Signed-off-by: Ron Livne Changes in v2: IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. Changes in v3: Whitespaces deleted. Changes in v4: The redundant initialization for create_flags (create_flags = 0) was deleted. --- linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) Index: kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c =================================================================== --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:12:37.000000000 +0300 +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-29 09:42:41.000000000 +0300 @@ -201,7 +201,10 @@ int ipoib_transport_dev_init(struct net_ init_attr.recv_cq = priv->recv_cq; if (priv->hca_caps & IB_DEVICE_UD_TSO) - init_attr.create_flags = IB_QP_CREATE_IPOIB_UD_LSO; + init_attr.create_flags |= IB_QP_CREATE_IPOIB_UD_LSO; + + if (priv->hca_caps & IB_DEVICE_BLOCK_LOOPBACK) + init_attr.create_flags |= IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK; if (dev->features & NETIF_F_SG) init_attr.cap.max_send_sge = MAX_SKB_FRAGS + 1; From tziporet at dev.mellanox.co.il Sun Jun 29 08:47:36 2008 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 29 Jun 2008 18:47:36 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: <48679D21.20200@voltaire.com> References: <48679D21.20200@voltaire.com> Message-ID: <4867AE98.8020607@mellanox.co.il> Or Gerlitz wrote: > >> Using the new connectx FW (2.5), I see performance drop to almost >> zero with ipoib datagram mode. > Oren, > > any news on this? > > Sending to the correct Oren (Sela) Tziporet From sashak at voltaire.com Sun Jun 29 09:02:14 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 29 Jun 2008 19:02:14 +0300 Subject: [ofa-general] [PATCH] infiniband-diags: fix snprintf() usage Message-ID: <20080629160214.GB12659@sashak.voltaire.com> snprintf() can return the value larger than provided buffer size - check is needed. Signed-off-by: Sasha Khapyorsky --- infiniband-diags/src/ibroute.c | 6 +++++- infiniband-diags/src/ibsysstat.c | 2 ++ 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/infiniband-diags/src/ibroute.c b/infiniband-diags/src/ibroute.c index 3a6907b..9f32b39 100644 --- a/infiniband-diags/src/ibroute.c +++ b/infiniband-diags/src/ibroute.c @@ -108,7 +108,11 @@ dump_mlid(char *str, int strlen, int mlid, int nports, mask = ntohs(mft[i][mlid%IB_MLIDS_IN_BLOCK]); if (mask) nonzero++; - n += snprintf(str + n, strlen -n, "%04hx", mask); + n += snprintf(str + n, strlen - n, "%04hx", mask); + if (n >= strlen) { + n = strlen; + break; + } } if (!nonzero && !dump_all) { str[0] = 0; diff --git a/infiniband-diags/src/ibsysstat.c b/infiniband-diags/src/ibsysstat.c index 8e00baf..2418c9e 100644 --- a/infiniband-diags/src/ibsysstat.c +++ b/infiniband-diags/src/ibsysstat.c @@ -100,6 +100,8 @@ mk_reply(int attr, void *data, int sz) for (i = 0; i < host_ncpu && sz > 0; i++) { n = snprintf(s, sz, "cpu %d: model %s MHZ %s\n", i, cpus[i].model, cpus[i].mhz); + if (n >= sz) + break; sz -= n; s += n; } -- 1.5.5.1.178.g1f811 From rdreier at cisco.com Sun Jun 29 09:04:41 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 29 Jun 2008 09:04:41 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <48672023.9050108@voltaire.com> (Or Gerlitz's message of "Sun, 29 Jun 2008 08:39:47 +0300") References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> Message-ID: > Can you spare few words on zero-stags? is it the chelsio way to > implement dma_mr - that is an stag that tells the HW "don't do any > address translation as the VA in the work-request is dma address", > etc. zero STag is defined by the iWARP spec. You can look it up. From swise at opengridcomputing.com Sun Jun 29 09:04:29 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Sun, 29 Jun 2008 11:04:29 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <48672023.9050108@voltaire.com> References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> Message-ID: <4867B28D.5090603@opengridcomputing.com> Or Gerlitz wrote: > Steve Wise wrote: >> Note: we need to place a few restrictions on zero-stag usage: > Hi Steve, > > Can you spare few words on zero-stags? is it the chelsio way to > implement dma_mr - that is an stag that tells the HW "don't do any > address translation as the VA in the work-request is dma address", etc. > > Or. Sort of. It is not exactly a dma_mr. dma_mrs allow remote IO. The zero-stag is an iWARP verbs concept. zero-stag can be used by kernel users as the lkey in any SGE that requires only local access rights. If the lkey is zero, then the rnic interprets the addr field in the SGE as a bus address. Steve. From sashak at voltaire.com Sun Jun 29 09:09:16 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 29 Jun 2008 19:09:16 +0300 Subject: [ofa-general] [PATCH] libibmad: fix snprintf() usage Message-ID: <20080629160916.GC12659@sashak.voltaire.com> snprintf() can return a value larger than provided buffer size, so this case should be checked. Also there is rework of dump_linkwidth() and dump_linkspeed() functions which addresses this requirement. Signed-off-by: Sasha Khapyorsky --- libibmad/src/dump.c | 71 +++++++++++++++++++++++++++++++------------------ libibmad/src/portid.c | 4 +++ 2 files changed, 49 insertions(+), 26 deletions(-) diff --git a/libibmad/src/dump.c b/libibmad/src/dump.c index b2798b8..61933ee 100644 --- a/libibmad/src/dump.c +++ b/libibmad/src/dump.c @@ -193,21 +193,23 @@ mad_dump_linkwidth(char *buf, int bufsz, void *val, int valsz) static void dump_linkwidth(char *buf, int bufsz, int width) { - char *s = buf, *e = s + bufsz; + int n = 0; if (width & 0x1) - s += snprintf(s, e - s, "1X or "); - if (s < e && (width & 0x2)) - s += snprintf(s, e - s, "4X or "); - if (s < e && (width & 0x4)) - s += snprintf(s, e - s, "8X or "); - if (s < e && (width & 0x8)) - s += snprintf(s, e - s, "12X or "); - - if ((width >> 4) || s == buf) - s += snprintf(s, e - s, "undefined (%d)", width); - else - s[-4] = 0; + n += snprintf(buf + n, bufsz - n, "1X or "); + if (n < bufsz && (width & 0x2)) + n += snprintf(buf + n, bufsz - n, "4X or "); + if (n < bufsz && (width & 0x4)) + n += snprintf(buf + n, bufsz - n, "8X or "); + if (n < bufsz && (width & 0x8)) + n += snprintf(buf + n, bufsz - n, "12X or "); + + if (n >= bufsz) + return; + else if (width == 0 || (width >> 4)) + snprintf(buf + n, bufsz - n, "undefined (%d)", width); + else if (bufsz > 3) + buf[n-4] = '\0'; } void @@ -265,19 +267,25 @@ mad_dump_linkspeed(char *buf, int bufsz, void *val, int valsz) static void dump_linkspeed(char *buf, int bufsz, int speed) { - char *s = buf, *e = s + bufsz; + int n = 0; if (speed & 0x1) - s += snprintf(s, e - s, "2.5 Gbps or "); - if (s < e && (speed & 0x2)) - s += snprintf(s, e - s, "5.0 Gbps or "); - if (s < e && (speed & 0x4)) - s += snprintf(s, e - s, "10.0 Gbps or "); - - if ((speed >> 3) || s == buf) - s += snprintf(s, e - s, "undefined (%d)", speed); - else - s[-4] = 0; + n += snprintf(buf + n, bufsz - n, "2.5 Gbps or "); + if (n < bufsz && (speed & 0x2)) + n += snprintf(buf + n, bufsz - n, "5.0 Gbps or "); + if (n < bufsz && (speed & 0x4)) + n += snprintf(buf + n, bufsz - n, "10.0 Gbps or "); + + if (n >= bufsz) + return; + else if (speed == 0 || (speed >> 3)) { + n += snprintf(buf + n, bufsz - n, "undefined (%d)", speed); + if (n >= bufsz) + return; + } else if (bufsz > 3) { + buf[n-4] = '\0'; + n -= 4; + } switch (speed) { case 1: @@ -287,7 +295,7 @@ dump_linkspeed(char *buf, int bufsz, int speed) break; default: if (!(speed >> 3)) - snprintf(s, e - s, " (IBA extension)"); + snprintf(buf + n, bufsz - n, " (IBA extension)"); break; } } @@ -613,6 +621,8 @@ mad_dump_sltovl(char *buf, int bufsz, void *val, int valsz) for (i = 0; i < 16; i++) { ib_slvl_get_i(p_slvl_tbl, i, &vl); n += snprintf(buf + n, bufsz - n, "%2u|", vl); + if (n >= bufsz) + break; } snprintf(buf + n, bufsz - n, "\n"); } @@ -627,15 +637,24 @@ mad_dump_vlarbitration(char *buf, int bufsz, void *val, int num) num /= sizeof(p_vla_tbl->vl_entry[0]); n = snprintf(buf, bufsz, "\nVL : |"); + if (n >= bufsz) + return; for (i = 0; i < num; i++) { ib_vl_arb_get_vl(p_vla_tbl->vl_entry[i].res_vl, &vl); n += snprintf(buf + n, bufsz - n, "0x%-2X|", vl); + if (n >= bufsz) + return; } n += snprintf(buf + n, bufsz - n, "\nWEIGHT: |"); - for (i = 0; i < num; i++) + if (n >= bufsz) + return; + for (i = 0; i < num; i++) { n += snprintf(buf + n, bufsz - n, "0x%-2X|", p_vla_tbl->vl_entry[i].weight); + if (n >= bufsz) + return; + } snprintf(buf + n, bufsz - n, "\n"); } diff --git a/libibmad/src/portid.c b/libibmad/src/portid.c index c6b5135..f8114f2 100644 --- a/libibmad/src/portid.c +++ b/libibmad/src/portid.c @@ -115,8 +115,12 @@ drpath2str(ib_dr_path_t *path, char *dstr, size_t dstr_size) int i = 0; int rc = snprintf(dstr, dstr_size, "slid %d; dlid %d; %d", path->drslid, path->drdlid, path->p[0]); + if (rc >= dstr_size) + return dstr; for (i = 1; i <= path->cnt; i++) { rc += snprintf(dstr+rc, dstr_size-rc, ",%d", path->p[i]); + if (rc >= dstr_size) + break; } return (dstr); } -- 1.5.5.1.178.g1f811 From sashak at voltaire.com Sun Jun 29 09:26:22 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 29 Jun 2008 19:26:22 +0300 Subject: [ofa-general] Re: snprintf() usage bug [was: Re: [PATCH v3] opensm/osm_qos_policy.c: log matched QoS criteria] In-Reply-To: <486769C6.8070605@dev.mellanox.co.il> References: <48566664.1010004@dev.mellanox.co.il> <20080628123536.GC29456@sashak.voltaire.com> <486769C6.8070605@dev.mellanox.co.il> Message-ID: <20080629162622.GE12659@sashak.voltaire.com> On 13:53 Sun 29 Jun , Yevgeny Kliteynik wrote: > > Right, I noticed you already started taking care of it. Yes, I passed over all places now. > Anyway, as for this patch - I'll rework it not to have string > manipulations, so the fast path will remain fast. Thanks. Sasha From vladsk at gmail.com Sun Jun 29 11:33:48 2008 From: vladsk at gmail.com (Vladimir Sokolovsky) Date: Sun, 29 Jun 2008 21:33:48 +0300 Subject: [ofa-general] [PATCH v4 for-2.6.27] IPOIB: add LRO support. Message-ID: <4867D58C.5030404@gmail.com> From c859eed63331e43993b0c3f0c903e48d1e27def2 Mon Sep 17 00:00:00 2001 From: Vladimir Sokolovsky Date: Mon, 23 Jun 2008 15:38:27 +0300 Subject: [PATCH] IPOIB: add LRO support. add "ipoib_use_lro" module parameter to enable LRO. add "ipoib_lro_max_aggr" module parameter to set the Max number of packets to be aggregated. LRO statistics accessible through ethtool. Signed-off-by: Vladimir Sokolovsky Signed-off-by: Eli Cohen --- Changes from v3: Added comment for checksum validation. drivers/infiniband/ulp/ipoib/Kconfig | 1 + drivers/infiniband/ulp/ipoib/ipoib.h | 13 +++++ drivers/infiniband/ulp/ipoib/ipoib_ethtool.c | 52 +++++++++++++++++++++ drivers/infiniband/ulp/ipoib/ipoib_ib.c | 8 +++- drivers/infiniband/ulp/ipoib/ipoib_main.c | 62 ++++++++++++++++++++++++++ 5 files changed, 135 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/Kconfig b/drivers/infiniband/ulp/ipoib/Kconfig index 1f76bad..691525c 100644 --- a/drivers/infiniband/ulp/ipoib/Kconfig +++ b/drivers/infiniband/ulp/ipoib/Kconfig @@ -1,6 +1,7 @@ config INFINIBAND_IPOIB tristate "IP-over-InfiniBand" depends on NETDEVICES && INET && (IPV6 || IPV6=n) + select INET_LRO ---help--- Support for the IP-over-InfiniBand protocol (IPoIB). This transports IP packets over InfiniBand so you can use your IB diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h index 8754b36..985ee87 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib.h +++ b/drivers/infiniband/ulp/ipoib/ipoib.h @@ -50,6 +50,7 @@ #include #include #include +#include /* constants */ @@ -94,6 +95,9 @@ enum { IPOIB_MCAST_FLAG_BUSY = 2, /* joining or already joined */ IPOIB_MCAST_FLAG_ATTACHED = 3, + IPOIB_MAX_LRO_DESCRIPTORS = 8, + IPOIB_LRO_MAX_AGGR = 64, + MAX_SEND_CQE = 16, IPOIB_CM_COPYBREAK = 256, }; @@ -248,6 +252,11 @@ struct ipoib_ethtool_st { u16 max_coalesced_frames; }; +struct ipoib_lro { + struct net_lro_mgr lro_mgr; + struct net_lro_desc lro_desc[IPOIB_MAX_LRO_DESCRIPTORS]; +}; + /* * Device private locking: tx_lock protects members used in TX fast * path (and we use LLTX so upper layers don't do extra locking). @@ -334,6 +343,8 @@ struct ipoib_dev_priv { int hca_caps; struct ipoib_ethtool_st ethtool; struct timer_list poll_timer; + + struct ipoib_lro lro; }; struct ipoib_ah { @@ -679,6 +690,8 @@ extern struct ib_sa_client ipoib_sa_client; #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG extern int ipoib_debug_level; +extern int ipoib_use_lro; +extern int ipoib_lro_max_aggr; #define ipoib_dbg(priv, format, arg...) \ do { \ diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c index 10279b7..86cf3e7 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c @@ -86,11 +86,63 @@ static int ipoib_set_coalesce(struct net_device *dev, return 0; } +#define IPOIB_STATS_LEN ARRAY_SIZE(ipoib_gstrings_stats) + +static const char ipoib_gstrings_stats[][ETH_GSTRING_LEN] = { + "LRO aggregated", "LRO flushed", + "LRO avg aggr", "LRO no_desc" +}; + +static void +ipoib_get_strings(struct net_device *netdev, u32 stringset, u8 *data) +{ + switch (stringset) { + case ETH_SS_STATS: + memcpy(data, *ipoib_gstrings_stats, + sizeof(ipoib_gstrings_stats)); + data += sizeof(ipoib_gstrings_stats); + break; + } +} + +static int ipoib_get_sset_count(struct net_device *dev, int sset) +{ + switch (sset) { + case ETH_SS_STATS: + return IPOIB_STATS_LEN; + default: + return -EOPNOTSUPP; + } +} + +static void ipoib_get_ethtool_stats(struct net_device *dev, + struct ethtool_stats *stats, uint64_t *data) +{ + struct ipoib_dev_priv *priv = netdev_priv(dev); + int index = 0; + + /* Get LRO statistics */ + data[index++] = priv->lro.lro_mgr.stats.aggregated; + data[index++] = priv->lro.lro_mgr.stats.flushed; + if (priv->lro.lro_mgr.stats.flushed) + data[index++] = priv->lro.lro_mgr.stats.aggregated / + priv->lro.lro_mgr.stats.flushed; + else + data[index++] = 0; + data[index++] = priv->lro.lro_mgr.stats.no_desc; + +} + static const struct ethtool_ops ipoib_ethtool_ops = { .get_drvinfo = ipoib_get_drvinfo, .get_tso = ethtool_op_get_tso, .get_coalesce = ipoib_get_coalesce, .set_coalesce = ipoib_set_coalesce, + .get_flags = ethtool_op_get_flags, + .set_flags = ethtool_op_set_flags, + .get_strings = ipoib_get_strings, + .get_sset_count = ipoib_get_sset_count, + .get_ethtool_stats = ipoib_get_ethtool_stats, }; void ipoib_set_ethtool_ops(struct net_device *dev) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index eca8518..8bd7db8 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -288,7 +288,10 @@ static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc) if (test_bit(IPOIB_FLAG_CSUM, &priv->flags) && likely(wc->csum_ok)) skb->ip_summed = CHECKSUM_UNNECESSARY; - netif_receive_skb(skb); + if (dev->features & NETIF_F_LRO) + lro_receive_skb(&priv->lro.lro_mgr, skb, 0); + else + netif_receive_skb(skb); repost: if (unlikely(ipoib_ib_post_receive(dev, wr_id))) @@ -448,6 +451,9 @@ poll_more: goto poll_more; } + if (dev->features & NETIF_F_LRO) + lro_flush_all(&priv->lro.lro_mgr); + return done; } diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index bfe1dbf..1c5f6c4 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -60,6 +60,17 @@ MODULE_PARM_DESC(send_queue_size, "Number of descriptors in send queue"); module_param_named(recv_queue_size, ipoib_recvq_size, int, 0444); MODULE_PARM_DESC(recv_queue_size, "Number of descriptors in receive queue"); +int ipoib_use_lro __read_mostly; +module_param_named(ipoib_use_lro, ipoib_use_lro, int, 0444); +MODULE_PARM_DESC(ipoib_use_lro, " Large Receive Offload, 1: enable, " + "0: disable, Default = 0"); + +int ipoib_lro_max_aggr __read_mostly = IPOIB_LRO_MAX_AGGR; +module_param_named(ipoib_lro_max_aggr, ipoib_lro_max_aggr, int, 0644); +MODULE_PARM_DESC(ipoib_lro_max_aggr, " LRO: Max packets to be aggregated. " + "Default = 64"); + + #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG int ipoib_debug_level; @@ -936,6 +947,53 @@ static const struct header_ops ipoib_header_ops = { .create = ipoib_hard_header, }; +static int get_skb_hdr(struct sk_buff *skb, void **iphdr, + void **tcph, u64 *hdr_flags, void *priv) +{ + unsigned int ip_len; + struct iphdr *iph; + + if (unlikely(skb->protocol != htons(ETH_P_IP))) + return -1; + + /* In the future we may add an else cluase that verifies the + * checksum and allows devices which do not calculate checksum + * to use LRO + */ + if (unlikely(skb->ip_summed != CHECKSUM_UNNECESSARY)) + return -1; + + /* non tcp packet */ + skb_reset_network_header(skb); + iph = ip_hdr(skb); + if (iph->protocol != IPPROTO_TCP) + return -1; + + ip_len = ip_hdrlen(skb); + skb_set_transport_header(skb, ip_len); + *tcph = tcp_hdr(skb); + + /* check if ip header and tcp header are complete */ + if (iph->tot_len < ip_len + tcp_hdrlen(skb)) + return -1; + + *hdr_flags = LRO_IPV4 | LRO_TCP; + *iphdr = iph; + + return 0; +} + +static void ipoib_lro_setup(struct ipoib_dev_priv *priv) +{ + priv->lro.lro_mgr.max_aggr = ipoib_lro_max_aggr; + priv->lro.lro_mgr.max_desc = IPOIB_MAX_LRO_DESCRIPTORS; + priv->lro.lro_mgr.lro_arr = priv->lro.lro_desc; + priv->lro.lro_mgr.get_skb_header = get_skb_hdr; + priv->lro.lro_mgr.features = LRO_F_NAPI; + priv->lro.lro_mgr.dev = priv->dev; + priv->lro.lro_mgr.ip_summed_aggr = CHECKSUM_UNNECESSARY; +} + static void ipoib_setup(struct net_device *dev) { struct ipoib_dev_priv *priv = netdev_priv(dev); @@ -975,6 +1033,8 @@ static void ipoib_setup(struct net_device *dev) priv->dev = dev; + ipoib_lro_setup(priv); + spin_lock_init(&priv->lock); spin_lock_init(&priv->tx_lock); @@ -1152,6 +1212,8 @@ static struct net_device *ipoib_add_port(const char *format, priv->dev->features |= NETIF_F_SG | NETIF_F_IP_CSUM; } + if (ipoib_use_lro) + priv->dev->features |= NETIF_F_LRO; /* * Set the full membership bit, so that we join the right * broadcast group, etc. From sashak at voltaire.com Sun Jun 29 14:33:06 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 00:33:06 +0300 Subject: [ofa-general] Re: [PATCH 0/4] opensm: Unicast Routing Cache In-Reply-To: <481D888A.7080608@dev.mellanox.co.il> References: <481D888A.7080608@dev.mellanox.co.il> Message-ID: <20080629213306.GF12659@sashak.voltaire.com> Hi Yevgeny, On 12:57 Sun 04 May , Yevgeny Kliteynik wrote: > > The following series of 4 patches implements unicast routing cache > in OpenSM. > > None of the current routing engines is scalable when we're talking > about big clusters. On ~5K cluster with ~1.3K switches, it takes > about two minutes to calculate the routing. The problem is, each > time the routing is calculated from scratch. I like a lot the idea to not rebuild entire routing from scratch. However I'm very unhappy with this implementation - it is huge, complicated and try to copy everything that OpenSM already have - topology, LFTs, lid matrices, etc.. > Incremental routing (which is on my to-do list) aims to address this > problem when there is some "local" change in fabric (e.g. single > switch failure, single link failure, link added, etc). > In such cases we can use the routing that was already calculated in > the previous heavy sweep, and then we just have to modify it according > to the change. > > For instance, if some switch has disappeared from the fabric, we can > use the routing that existed with this switch, take a step back from > this switch and see if it is possible to route all the lids that were > routed through this switch some other way (which is usually the case). > > To implement incremental routing, we need to create some kind of unicast > routing cache, which is what these patches implement. osm_switch struct has already LFT images (if you like to convert its cl_vector's implementation to single array like ucat_mgr has I will appretiate this :)). I don't see why it cannot be used. If later when switch will back we wil want to restore it as is, it will require quite small modification in drop manager - instead of removing and freeing switch object instance move this to same old_switches qmap. > In addition to being > a step toward the incremental routing, routing cache is usefull by itself. I fully agree with it (assuming that routing cache is something that saves full routing tables calculation for us). > This cache can save us routing calculation in case of change in the leaf > switches or in hosts. For instance, if some node is rebooted, OpenSM would > start a heavy sweep with full routing recalculation when the HCA is going > down, and another one when HCA is brought up, when in fact both of these > routing calculation can be replaced by using of unicast routing cache. > > Unicast routing cache comprises the following: > - Topology: a data structure with all the switches and CAs of the fabric > - LFTs: each switch has an LFT cached > - Lid matrices: each switch has lid matrices cached, which is needed for > multicast routing (which is not cached). Again, all this we have already - in OpenSM tables. What is the reason to copy this again? > There is a topology matching function that compares the current topology > with the cached one to find out whether the cache is usable (valid) or not. > > The cache is used the following way: > - SM is executed - it starts first routing calculation > - calculated routing is stored in the cache > - at some point new heavy sweep is triggered > - unicast manager checks whether the cache can be used instead > of new routing calculation. > In one of the following cases we can use cached routing > + there is no topology change > + one or more CAs disappeared (they exist in the cached topology > model, but missing in the newly discovered fabric) > + one or more leaf switches disappeared > In these cases cached routing is written to the switches as is > (unless the switch doesn't exist). > If there is any other topology change: > - existing cache is invalidated > - topology is cached > - routing is calculated as usual > - routing is cached > > My simulations show that when the usual routing phase of the heavy > sweep on the topology that I mentioned above takes ~2 minutes, > cached routing reduces this time to 6 seconds (which is nice, if you > ask me...). It is nice. But I think that much simpler and even more faster implementation would be like this: - validate topology change during subnet discovery phase (just by updating some p_subn->change_status mask or ->need_rerouting flag), - check this mask/flag at beginning of osm_ucast_mgr_process() and if rerouting was not needed just return. Another thing we may want to add (but it is not related to routing cache or incremental routing) it to keep two sets of LFTs with switch object for validation purposes - "requested" (filled by routing algorithm) and "real" (filled from responded MADs). In any case I passed over some submitted code anyway, will send the comments. Sasha From sashak at voltaire.com Sun Jun 29 14:43:45 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 00:43:45 +0300 Subject: [ofa-general] Re: [PATCH 1/4] opensm/osm_ucast_cache.{c, h}: ucast routing cache implementation In-Reply-To: <481D8905.1010207@dev.mellanox.co.il> References: <481D8905.1010207@dev.mellanox.co.il> Message-ID: <20080629214345.GG12659@sashak.voltaire.com> Hi Yevgeny, Sorry about huge delay with looking at this. On 12:59 Sun 04 May , Yevgeny Kliteynik wrote: > Unicast routing cache implementation. > > Unicast routing cache comprises the following: > - Topology: a data structure with all the switches and CAs of the fabric > - LFTs: each switch has an LFT cached > - Lid matrices: each switch has lid matrices cached, which is needed for > multicast routing (which is not cached). > > There is also a topology matching function that compares the current topology > with the cached one to find out whether the cache is usable (valid) or not. As I wrote in another email, I believe that saving needless full-reroutig cycle is a good idea and very needed for OpenSM, but I don't like this implementation. In order to understand it better I was need to pass over code, some comments may be useful. It is below. Sasha > Signed-off-by: Yevgeny Kliteynik > --- > opensm/include/opensm/osm_ucast_cache.h | 319 ++++++++ > opensm/opensm/osm_ucast_cache.c | 1197 +++++++++++++++++++++++++++++++ > 2 files changed, 1516 insertions(+), 0 deletions(-) > create mode 100644 opensm/include/opensm/osm_ucast_cache.h > create mode 100644 opensm/opensm/osm_ucast_cache.c > > diff --git a/opensm/include/opensm/osm_ucast_cache.h b/opensm/include/opensm/osm_ucast_cache.h > new file mode 100644 > index 0000000..a3b40f9 > --- /dev/null > +++ b/opensm/include/opensm/osm_ucast_cache.h > @@ -0,0 +1,319 @@ > +/* > + * Copyright (c) 2002-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. > + * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * > + * This software is available to you under a choice of one of two > + * licenses. You may choose to be licensed under the terms of the GNU > + * General Public License (GPL) Version 2, available from the file > + * COPYING in the main directory of this source tree, or the > + * OpenIB.org BSD license below: > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * - Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the following > + * disclaimer. > + * > + * - Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > + * SOFTWARE. > + * > + */ > + > +/* > + * Abstract: > + * Declaration of osm_ucast_cache_t. > + * This object represents the Unicast Cache object. > + * > + * Environment: > + * Linux User Mode > + * > + * $Revision: 1.4 $ > + */ > + > +#ifndef _OSM_UCAST_CACHE_H_ > +#define _OSM_UCAST_CACHE_H_ > + > +#ifdef __cplusplus > +# define BEGIN_C_DECLS extern "C" { > +# define END_C_DECLS } > +#else /* !__cplusplus */ > +# define BEGIN_C_DECLS > +# define END_C_DECLS > +#endif /* __cplusplus */ > + > +BEGIN_C_DECLS > + > +struct _osm_ucast_mgr; > + > +#define UCAST_CACHE_TOPOLOGY_MATCH 0x0000 > +#define UCAST_CACHE_TOPOLOGY_LESS_SWITCHES 0x0001 > +#define UCAST_CACHE_TOPOLOGY_LINK_TO_LEAF_SW_MISSING 0x0002 > +#define UCAST_CACHE_TOPOLOGY_LINK_TO_CA_MISSING 0x0004 > +#define UCAST_CACHE_TOPOLOGY_MORE_SWITCHES 0x0008 > +#define UCAST_CACHE_TOPOLOGY_NEW_LID 0x0010 > +#define UCAST_CACHE_TOPOLOGY_LINK_TO_SW_MISSING 0x0020 > +#define UCAST_CACHE_TOPOLOGY_LINK_ADDED 0x0040 > +#define UCAST_CACHE_TOPOLOGY_NEW_SWITCH 0x0080 > +#define UCAST_CACHE_TOPOLOGY_NEW_CA 0x0100 > +#define UCAST_CACHE_TOPOLOGY_NO_MATCH 0x0200 > + > +/****h* OpenSM/Unicast Manager/Unicast Cache > +* NAME > +* Unicast Cache > +* > +* DESCRIPTION > +* The Unicast Cache object encapsulates the information > +* needed to cache and write unicast routing of the subnet. > +* > +* The Unicast Cache object is NOT thread safe. > +* > +* This object should be treated as opaque and should be > +* manipulated only through the provided functions. > +* > +* AUTHOR > +* Yevgeny Kliteynik, Mellanox > +* > +*********/ > + > + > +/****s* OpenSM: Unicast Cache/osm_ucast_cache_t > +* NAME > +* osm_ucast_cache_t > +* > +* DESCRIPTION > +* Unicast Cache structure. > +* > +* This object should be treated as opaque and should > +* be manipulated only through the provided functions. > +* > +* SYNOPSIS > +*/ > +typedef struct osm_ucast_cache_t_ { > + struct _osm_ucast_mgr * p_ucast_mgr; > + cl_qmap_t sw_tbl; > + cl_qmap_t ca_tbl; > + boolean_t topology_valid; > + boolean_t routing_valid; > + boolean_t need_update; > +} osm_ucast_cache_t; > +/* > +* FIELDS > +* p_ucast_mgr > +* Pointer to the Unicast Manager for this subnet. > +* > +* sw_tbl > +* Cached switches table. > +* > +* ca_tbl > +* Cached CAs table. > +* > +* topology_valid > +* TRUE if the cache is populated with the fabric topology. > +* > +* routing_valid > +* TRUE if the cache is populated with the unicast routing > +* in addition to the topology. > +* > +* need_update > +* TRUE if the cached routing needs to be updated. > +* > +* SEE ALSO > +* Unicast Manager object > +*********/ > + > +/****f* OpenSM: Unicast Cache/osm_ucast_cache_construct > +* NAME > +* osm_ucast_cache_construct > +* > +* DESCRIPTION > +* This function constructs a Unicast Cache object. > +* > +* SYNOPSIS > +*/ > +osm_ucast_cache_t * > +osm_ucast_cache_construct(struct _osm_ucast_mgr * const p_mgr); > +/* > +* PARAMETERS > +* p_mgr > +* [in] Pointer to a Unicast Manager object. > +* > +* RETURN VALUE > +* This function return the created Ucast Cache object on success, > +* or NULL on any error. > +* > +* NOTES > +* Allows osm_ucast_cache_destroy > +* > +* Calling osm_ucast_mgr_construct is a prerequisite to > +* calling any other method. > +* > +* SEE ALSO > +* Unicast Cache object, osm_ucast_cache_destroy > +*********/ > + > +/****f* OpenSM: Unicast Cache/osm_ucast_cache_destroy > +* NAME > +* osm_ucast_cache_destroy > +* > +* DESCRIPTION > +* The osm_ucast_cache_destroy function destroys the object, > +* releasing all resources. > +* > +* SYNOPSIS > +*/ > +void osm_ucast_cache_destroy(osm_ucast_cache_t * p_cache); > +/* > +* PARAMETERS > +* p_cache > +* [in] Pointer to the object to destroy. > +* > +* RETURN VALUE > +* This function does not return any value. > +* > +* NOTES > +* Performs any necessary cleanup of the specified > +* Unicast Cache object. > +* Further operations should not be attempted on the > +* destroyed object. > +* This function should only be called after a call to > +* osm_ucast_cache_construct. > +* > +* SEE ALSO > +* Unicast Cache object, osm_ucast_cache_construct > +*********/ > + > +/****f* OpenSM: Unicast Cache/osm_ucast_cache_refresh_topo > +* NAME > +* osm_ucast_cache_refresh_topo > +* > +* DESCRIPTION > +* The osm_ucast_cache_refresh_topo function re-reads the > +* updated topology. > +* > +* SYNOPSIS > +*/ > +void osm_ucast_cache_refresh_topo(osm_ucast_cache_t * p_cache); > +/* > +* PARAMETERS > +* p_cache > +* [in] Pointer to the cache object to refresh. > +* > +* RETURN VALUE > +* This function does not return any value. > +* > +* NOTES > +* This function invalidates the existing unicast cache > +* and re-reads the updated topology. > +* > +* SEE ALSO > +* Unicast Cache object, osm_ucast_cache_construct > +*********/ > + > +/****f* OpenSM: Unicast Cache/osm_ucast_cache_refresh_lid_matrices > +* NAME > +* osm_ucast_cache_refresh_lid_matrices > +* > +* DESCRIPTION > +* The osm_ucast_cache_refresh_topo function re-reads the > +* updated lid matrices. > +* > +* SYNOPSIS > +*/ > +void osm_ucast_cache_refresh_lid_matrices(osm_ucast_cache_t * p_cache); > +/* > +* PARAMETERS > +* p_cache > +* [in] Pointer to the cache object to refresh. > +* > +* RETURN VALUE > +* This function does not return any value. > +* > +* NOTES > +* This function re-reads the updated lid matrices. > +* > +* SEE ALSO > +* Unicast Cache object, osm_ucast_cache_construct > +*********/ > + > +/****f* OpenSM: Unicast Cache/osm_ucast_cache_apply > +* NAME > +* osm_ucast_cache_apply > +* > +* DESCRIPTION > +* The osm_ucast_cache_apply function tries to apply > +* the cached unicast routing on the subnet switches. > +* > +* SYNOPSIS > +*/ > +int osm_ucast_cache_apply(osm_ucast_cache_t * p_cache); > +/* > +* PARAMETERS > +* p_cache > +* [in] Pointer to the cache object to be used. > +* > +* RETURN VALUE > +* 0 if unicast cache was successfully written to switches, > +* non-zero for any error. > +* > +* NOTES > +* Compares the current topology to the cached topology, > +* and if the topology matches, or if changes in topology > +* have no impact on routing tables, writes the cached > +* unicast routing to the subnet switches. > +* > +* SEE ALSO > +* Unicast Cache object > +*********/ > + > +/****f* OpenSM: Unicast Cache/osm_ucast_cache_set_sw_fwd_table > +* NAME > +* osm_ucast_cache_set_sw_fwd_table > +* > +* DESCRIPTION > +* The osm_ucast_cache_set_sw_fwd_table function sets > +* (caches) linear forwarding table for the specified > +* switch. > +* > +* SYNOPSIS > +*/ > +void > +osm_ucast_cache_set_sw_fwd_table(osm_ucast_cache_t * p_cache, > + uint8_t * ucast_mgr_lft_buf, > + osm_switch_t * p_osm_sw); > +/* > +* PARAMETERS > +* p_cache > +* [in] Pointer to the cache object to be used. > +* > +* ucast_mgr_lft_buf > +* [in] LFT to set. > +* > +* p_osm_sw > +* [in] pointer to the switch that the LFT refers to. > +* > +* RETURN VALUE > +* This function does not return any value. > +* > +* NOTES > +* > +* SEE ALSO > +* Unicast Cache object > +*********/ > + > +END_C_DECLS > +#endif /* _OSM_UCAST_MGR_H_ */ > + > diff --git a/opensm/opensm/osm_ucast_cache.c b/opensm/opensm/osm_ucast_cache.c > new file mode 100644 > index 0000000..4ad7c30 > --- /dev/null > +++ b/opensm/opensm/osm_ucast_cache.c > @@ -0,0 +1,1197 @@ > +/* > + * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. > + * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * > + * This software is available to you under a choice of one of two > + * licenses. You may choose to be licensed under the terms of the GNU > + * General Public License (GPL) Version 2, available from the file > + * COPYING in the main directory of this source tree, or the > + * OpenIB.org BSD license below: > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * - Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the following > + * disclaimer. > + * > + * - Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > + * SOFTWARE. > + * > + */ > + > +/* > + * Abstract: > + * Implementation of OpenSM Cached routing > + * > + * Environment: > + * Linux User Mode > + * > + */ > + > +#if HAVE_CONFIG_H > +# include > +#endif > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +struct cache_sw_t_; > +struct cache_ca_t_; > +struct cache_port_t_; > + > +typedef union cache_sw_or_ca_ { > + struct cache_sw_t_ * p_sw; > + struct cache_ca_t_ * p_ca; > +} cache_node_t; > + > +typedef struct cache_port_t_ { > + uint8_t remote_node_type; > + cache_node_t remote_node; > +} cache_port_t; > + > +typedef struct cache_ca_t_ { > + cl_map_item_t map_item; > + uint16_t lid_ho; > +} cache_ca_t; > + > +typedef struct cache_sw_t_ { > + cl_map_item_t map_item; > + uint16_t lid_ho; > + uint16_t max_lid_ho; > + osm_switch_t *p_osm_sw; /* pointer to the updated switch object */ > + uint8_t num_ports; > + cache_port_t ** ports; > + uint8_t **lid_matrix; > + uint8_t * lft_buff; > + boolean_t is_leaf; Please use tabs for indentation. > +} cache_sw_t; > + > +/********************************************************************** > + **********************************************************************/ > + > +static osm_switch_t * > +__ucast_cache_get_starting_osm_sw(osm_ucast_cache_t * p_cache) > +{ > + osm_port_t * p_osm_port; > + osm_node_t * p_osm_node; > + osm_physp_t * p_osm_physp; > + > + CL_ASSERT(p_cache->p_ucast_mgr); > + > + /* find the OSM node */ > + p_osm_port = osm_get_port_by_guid( > + p_cache->p_ucast_mgr->p_subn, > + p_cache->p_ucast_mgr->p_subn->sm_port_guid); > + CL_ASSERT(p_osm_port); > + > + p_osm_node = p_osm_port->p_node; > + switch (osm_node_get_type(p_osm_node)) { > + case IB_NODE_TYPE_SWITCH: > + /* OpenSM runs on switch - we're done */ > + return p_osm_node->sw; > + > + case IB_NODE_TYPE_CA: > + /* SM runs on CA - get the switch > + that CA is connected to. */ > + p_osm_physp = p_osm_port->p_physp; > + p_osm_physp = osm_physp_get_remote(p_osm_physp); > + p_osm_node = osm_physp_get_node_ptr(p_osm_physp); > + CL_ASSERT(p_osm_node); > + return p_osm_node->sw; > + > + default: > + /* SM runs on some other node - not supported */ > + return NULL; > + } > +} /* __ucast_cache_get_starting_osm_sw() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static cache_sw_t * > +__ucast_cache_get_sw(osm_ucast_cache_t * p_cache, > + uint16_t lid_ho) > +{ > + cache_sw_t * p_sw; > + > + p_sw = (cache_sw_t *) cl_qmap_get(&p_cache->sw_tbl, lid_ho); > + if (p_sw == (cache_sw_t *) cl_qmap_end(&p_cache->sw_tbl)) > + return NULL; > + > + return p_sw; > +} /* __ucast_cache_get_sw() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static cache_ca_t * > +__ucast_cache_get_ca(osm_ucast_cache_t * p_cache, > + uint16_t lid_ho) > +{ > + cache_ca_t * p_ca; > + > + p_ca = (cache_ca_t *) cl_qmap_get(&p_cache->ca_tbl, lid_ho); > + if (p_ca == (cache_ca_t *) cl_qmap_end(&p_cache->ca_tbl)) > + return NULL; > + > + return p_ca; > +} /* __ucast_cache_get_ca() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static cache_port_t * > +__ucast_cache_add_port(osm_ucast_cache_t * p_cache, > + uint8_t remote_node_type, > + uint16_t lid_ho) > +{ > + cache_port_t * p_port = (cache_port_t *) malloc(sizeof(cache_port_t)); > + memset(p_port, 0, sizeof(cache_port_t)); > + > + p_port->remote_node_type = remote_node_type; > + if (remote_node_type == IB_NODE_TYPE_SWITCH) > + { > + cache_sw_t * p_sw = __ucast_cache_get_sw( > + p_cache, lid_ho); > + CL_ASSERT(p_sw); > + p_port->remote_node.p_sw = p_sw; > + } > + else { > + cache_ca_t * p_ca = __ucast_cache_get_ca( > + p_cache, lid_ho); > + CL_ASSERT(p_ca); > + p_port->remote_node.p_ca = p_ca; > + } > + > + return p_port; > +} /* __ucast_cache_add_port() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static cache_sw_t * > +__ucast_cache_add_sw(osm_ucast_cache_t * p_cache, > + osm_switch_t * p_osm_sw) > +{ > + cache_sw_t *p_sw = (cache_sw_t*)malloc(sizeof(cache_sw_t)); > + memset(p_sw, 0, sizeof(cache_sw_t)); > + > + p_sw->p_osm_sw = p_osm_sw; > + > + p_sw->lid_ho = > + cl_ntoh16(osm_node_get_base_lid(p_osm_sw->p_node, 0)); > + > + p_sw->num_ports = osm_node_get_num_physp(p_osm_sw->p_node); > + p_sw->ports = (cache_port_t **) > + malloc(p_sw->num_ports * sizeof(cache_port_t *)); > + memset(p_sw->ports, 0, p_sw->num_ports * sizeof(cache_port_t *)); > + > + cl_qmap_insert(&p_cache->sw_tbl, p_sw->lid_ho, &p_sw->map_item); > + return p_sw; > +} /* __ucast_cache_add_sw() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static cache_ca_t * > +__ucast_cache_add_ca(osm_ucast_cache_t * p_cache, > + uint16_t lid_ho) > +{ > + cache_ca_t *p_ca = (cache_ca_t*)malloc(sizeof(cache_ca_t)); > + memset(p_ca, 0, sizeof(cache_ca_t)); > + > + p_ca->lid_ho = lid_ho; > + > + cl_qmap_insert(&p_cache->ca_tbl, p_ca->lid_ho, &p_ca->map_item); > + return p_ca; > +} /* __ucast_cache_add_ca() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__cache_port_destroy(cache_port_t * p_port) > +{ > + if (!p_port) > + return; > + free(p_port); > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__cache_sw_destroy(cache_sw_t * p_sw) > +{ > + int i; > + > + if (!p_sw) > + return; > + > + if (p_sw->ports) { > + for (i = 0; i < p_sw->num_ports; i++) > + if (p_sw->ports[i]) > + __cache_port_destroy(p_sw->ports[i]); > + free(p_sw->ports); > + } > + > + if (p_sw->lid_matrix) { > + for (i = 0; i <= p_sw->max_lid_ho; i++) > + if (p_sw->lid_matrix[i]) > + free(p_sw->lid_matrix[i]); > + free(p_sw->lid_matrix); > + } > + > + if (p_sw->lft_buff) > + free(p_sw->lft_buff); > + > + free(p_sw); > +} /* __cache_sw_destroy() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__cache_ca_destroy(cache_ca_t * p_ca) > +{ > + if (!p_ca) > + return; > + free(p_ca); > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +static int > +__ucast_cache_populate(osm_ucast_cache_t * p_cache) > +{ > + cl_list_t sw_bfs_list; cl_list, cl_map, etc. (w/out 'q') are slow. It is really better to use cl_q* version. > + osm_switch_t * p_osm_sw; > + osm_switch_t * p_remote_osm_sw; Seems that those variables (and maybe others) are never used together. Use just one is so. > + osm_node_t * p_osm_node; > + osm_node_t * p_remote_osm_node; > + osm_physp_t * p_osm_physp; > + osm_physp_t * p_remote_osm_physp; > + cache_sw_t * p_sw; > + cache_sw_t * p_remote_sw; > + cache_ca_t * p_remote_ca; > + uint16_t remote_lid_ho; > + unsigned num_ports; > + unsigned i; > + int res = 0; > + osm_log_t * p_log = p_cache->p_ucast_mgr->p_log; > + > + OSM_LOG_ENTER(p_log); > + > + cl_list_init(&sw_bfs_list, 10); > + > + /* Use management switch or switch that is connected > + to management CA as a BFS scan starting point */ > + > + p_osm_sw = __ucast_cache_get_starting_osm_sw(p_cache); > + if (!p_osm_sw) { > + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 3A51: " > + "failed getting cache population starting point\n"); > + res = 1; > + goto Exit; > + } > + > + /* switch is cached BEFORE entering to the BFS list, > + so we will know whether this switch was "visited" */ > + > + p_sw = __ucast_cache_add_sw(p_cache, p_osm_sw); > + cl_list_insert_tail(&sw_bfs_list, p_sw); > + > + /* Create cached switches in the BFS order. > + This will ensure that the fabric scan is done each > + time the same way and will allow accurate matching > + between the current fabric and the cached one. */ Why BFSing is needed there? Would not it be simpler to run over p_subn->sw_guid_tbl? > + while (!cl_is_list_empty(&sw_bfs_list)) { > + p_sw = (cache_sw_t *) cl_list_remove_head(&sw_bfs_list); > + p_osm_sw = p_sw->p_osm_sw; > + p_osm_node = p_osm_sw->p_node; > + num_ports = osm_node_get_num_physp(p_osm_node); > + > + /* skipping port 0 on switches */ > + for (i = 1; i < num_ports; i++) { > + p_osm_physp = osm_node_get_physp_ptr(p_osm_node, i); > + if (!p_osm_physp || > + !osm_physp_is_valid(p_osm_physp) || > + !osm_link_is_healthy(p_osm_physp)) > + continue; > + > + p_remote_osm_physp = osm_physp_get_remote(p_osm_physp); > + if (!p_remote_osm_physp || > + !osm_physp_is_valid(p_remote_osm_physp) || > + !osm_link_is_healthy(p_remote_osm_physp)) > + continue; > + > + p_remote_osm_node = > + osm_physp_get_node_ptr(p_remote_osm_physp); > + if (!p_remote_osm_node) { > + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 3A52: " > + "no node for remote port\n"); > + res = 1; > + goto Exit; > + } > + > + if (osm_node_get_type(p_remote_osm_node) == > + IB_NODE_TYPE_SWITCH) { > + > + remote_lid_ho = cl_ntoh16( > + osm_node_get_base_lid( > + p_remote_osm_node, 0)); > + > + p_remote_osm_sw = p_remote_osm_node->sw; > + CL_ASSERT(p_remote_osm_sw); > + > + p_remote_sw = __ucast_cache_get_sw( > + p_cache, > + remote_lid_ho); > + > + /* If the remote switch hasn't been > + cached yet, add it to the cache > + and insert it into the BFS list */ > + > + if (!p_remote_sw) { > + p_remote_sw = __ucast_cache_add_sw( > + p_cache, > + p_remote_osm_sw); > + cl_list_insert_tail(&sw_bfs_list, > + p_remote_sw); > + } > + } > + else { opensm/osn_indent will suggest about '} else {' style. > + remote_lid_ho = cl_ntoh16( > + osm_physp_get_base_lid( > + p_remote_osm_physp)); > + > + p_sw->is_leaf = TRUE; > + p_remote_ca = __ucast_cache_add_ca( > + p_cache, remote_lid_ho); > + > + /* no need to add this node to BFS list */ > + } > + > + /* cache this port */ > + p_sw->ports[i] = __ucast_cache_add_port( > + p_cache, > + osm_node_get_type(p_remote_osm_node), > + remote_lid_ho); > + } > + } > + > + cl_list_destroy(&sw_bfs_list); Tabs... > + p_cache->topology_valid = TRUE; > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "cache populated (%u SWs, %u CAs)\n", > + cl_qmap_count(&p_cache->sw_tbl), > + cl_qmap_count(&p_cache->ca_tbl)); > + > + Exit: > + OSM_LOG_EXIT(p_log); > + return res; > +} /* __ucast_cache_populate() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_read_sw_lid_matrix(cl_map_item_t * const p_map_item, > + void *context) > +{ > + cache_sw_t *p_sw = (cache_sw_t * const)p_map_item; > + uint16_t target_lid_ho; > + uint8_t port_num; > + > + if (!p_sw->p_osm_sw) > + return; > + > + /* allocate lid matrices buffer: > + lid_matrix[target_lids][port_nums] */ > + CL_ASSERT(!p_sw->lid_matrix); > + p_sw->lid_matrix = (uint8_t **) > + malloc((p_sw->max_lid_ho + 1) * sizeof(uint8_t*)); > + > + for (target_lid_ho = 0; > + target_lid_ho <= p_sw->max_lid_ho; target_lid_ho++){ > + > + /* set hops for this target through every switch port */ > + > + p_sw->lid_matrix[target_lid_ho] = > + (uint8_t *)malloc(p_sw->num_ports); > + memset(p_sw->lid_matrix[target_lid_ho], > + OSM_NO_PATH, p_sw->num_ports); > + > + for (port_num = 1; port_num < p_sw->num_ports; port_num++) > + p_sw->lid_matrix[target_lid_ho][port_num] = > + osm_switch_get_hop_count(p_sw->p_osm_sw, > + target_lid_ho, > + port_num); Original switches keep lid matrices for switches only, and not for CAs, it was done to sleep LID matrix generation and to save a lot of memory. > + } > +} /* __ucast_cache_read_sw_lid_matrix() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_write_sw_routing(cl_map_item_t * const p_map_item, > + void * context) > +{ > + cache_sw_t *p_sw = (cache_sw_t * const)p_map_item; > + osm_ucast_cache_t * p_cache = (osm_ucast_cache_t *) context; > + uint8_t *ucast_mgr_lft_buf = p_cache->p_ucast_mgr->lft_buf; > + uint16_t target_lid_ho; > + uint8_t port_num; > + uint8_t hops; > + osm_log_t * p_log = p_cache->p_ucast_mgr->p_log; > + > + OSM_LOG_ENTER(p_log); > + > + if (!p_sw->p_osm_sw) { > + /* some switches (leaf switches) may exist in the > + cache, but not exist in the current topology */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "cached switch 0x%04x doesn't exist in the fabric\n", > + p_sw->lid_ho); Now we are using decimal format for unicast LIDs representation. Also what about to use OSM_LOG_DEBUG for debug purposes? This file has 30 OSM_LOG_VERBOSE message, osm_ucast_mgr.c - only 3. > + goto Exit; > + } > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "writing routing for cached switch 0x%04x, " > + "max_lid_ho = 0x%04x\n", > + p_sw->lid_ho, p_sw->max_lid_ho); > + > + /* write cached LFT to this switch: clear existing > + ucast mgr lft buffer, write the cached lft to the > + ucast mgr buffer, and set this lft on switch */ > + CL_ASSERT(p_sw->lft_buff); > + memset(ucast_mgr_lft_buf, OSM_NO_PATH, IB_LID_UCAST_END_HO + 1); Why memset()? Isn't it enough to memcpy() and max_lid_ho setup which you do below anyway. > + if (p_sw->max_lid_ho > 0) > + memcpy(ucast_mgr_lft_buf, p_sw->lft_buff, > + p_sw->max_lid_ho + 1); > + > + p_sw->p_osm_sw->max_lid_ho = p_sw->max_lid_ho; > + osm_ucast_mgr_set_fwd_table(p_cache->p_ucast_mgr,p_sw->p_osm_sw); > + > + /* write cached lid matrix to this switch */ > + > + osm_switch_prepare_path_rebuild(p_sw->p_osm_sw, p_sw->max_lid_ho); > + > + /* set hops to itself */ > + osm_switch_set_hops(p_sw->p_osm_sw,p_sw->lid_ho,0,0); > + > + for (target_lid_ho = 0; > + target_lid_ho <= p_sw->max_lid_ho; target_lid_ho++){ > + /* port 0 on switches lid matrices is used > + for storing minimal hops to the target > + lid, so we iterate from port 1 */ > + for (port_num = 1; port_num < p_sw->num_ports; port_num++) { > + hops = p_sw->lid_matrix[target_lid_ho][port_num]; > + if (hops != OSM_NO_PATH) > + osm_switch_set_hops(p_sw->p_osm_sw, > + target_lid_ho, port_num, hops); > + } As above - switches need lid matrices only for switch nodes. > + } > + Exit: > + OSM_LOG_EXIT(p_log); > +} /* __ucast_cache_write_sw_routing() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_clear_sw_routing(cl_map_item_t * const p_map_item, > + void *context) > +{ > + cache_sw_t *p_sw = (cache_sw_t * const)p_map_item; > + unsigned lid; > + > + if(p_sw->lft_buff) { > + free(p_sw->lft_buff); > + p_sw->lft_buff = NULL; > + } > + > + if(p_sw->lid_matrix) { > + for (lid = 0; lid < p_sw->max_lid_ho; lid++) > + if (p_sw->lid_matrix[lid]) > + free(p_sw->lid_matrix[lid]); > + free(p_sw->lid_matrix); > + p_sw->lid_matrix = NULL; > + } > + > + p_sw->max_lid_ho = 0; > +} /* __ucast_cache_clear_sw_routing() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_clear_routing(osm_ucast_cache_t * p_cache) > +{ > + cl_qmap_apply_func(&p_cache->sw_tbl, __ucast_cache_clear_sw_routing, > + (void *)p_cache); > + p_cache->routing_valid = FALSE; > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_invalidate(osm_ucast_cache_t * p_cache) > +{ > + cache_sw_t * p_sw; > + cache_sw_t * p_next_sw; > + cache_ca_t * p_ca; > + cache_ca_t * p_next_ca; > + > + p_next_sw = (cache_sw_t *) cl_qmap_head(&p_cache->sw_tbl); > + while (p_next_sw != (cache_sw_t *) cl_qmap_end(&p_cache->sw_tbl)) { > + p_sw = p_next_sw; > + p_next_sw = (cache_sw_t *) cl_qmap_next(&p_sw->map_item); > + __cache_sw_destroy(p_sw); > + } > + cl_qmap_remove_all(&p_cache->sw_tbl); > + > + p_next_ca = (cache_ca_t *) cl_qmap_head(&p_cache->ca_tbl); > + while (p_next_ca != (cache_ca_t *) cl_qmap_end(&p_cache->ca_tbl)) { > + p_ca = p_next_ca; > + p_next_ca = (cache_ca_t *) cl_qmap_next(&p_ca->map_item); > + __cache_ca_destroy(p_ca); > + } > + cl_qmap_remove_all(&p_cache->ca_tbl); > + > + p_cache->routing_valid = FALSE; > + p_cache->topology_valid = FALSE; > + p_cache->need_update = FALSE; > +} /* __ucast_cache_invalidate() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +static int > +__ucast_cache_read_topology(osm_ucast_cache_t * p_cache) > +{ > + CL_ASSERT(p_cache && p_cache->p_ucast_mgr); > + > + return __ucast_cache_populate(p_cache); > +} What is a reason to make this wrapper function? > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_read_lid_matrices(osm_ucast_cache_t * p_cache) > +{ > + CL_ASSERT(p_cache && p_cache->p_ucast_mgr && > + p_cache->topology_valid); > + > + if (p_cache->routing_valid) > + __ucast_cache_clear_routing(p_cache); I see that this two lines are already presented in osm_ucast_cache_refresh_lid_matrices() and it is only place where __ucast_cache_read_lid_matrices() called. For me it looks that whole logic could be simplified if you will have separate reread_lfts() and reread_lid_matrices() primitives. > + > + cl_qmap_apply_func(&p_cache->sw_tbl, > + __ucast_cache_read_sw_lid_matrix, > + (void *)p_cache); > + p_cache->routing_valid = TRUE; > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_write_routing(osm_ucast_cache_t * p_cache) > +{ > + CL_ASSERT(p_cache && p_cache->p_ucast_mgr && > + p_cache->topology_valid && p_cache->routing_valid); > + > + cl_qmap_apply_func(&p_cache->sw_tbl, > + __ucast_cache_write_sw_routing, > + (void *)p_cache); > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +static void > +__ucast_cache_sw_clear_osm_ptr(cl_map_item_t * const p_map_item, > + void *context) > +{ > + ((cache_sw_t * const)p_map_item)->p_osm_sw = NULL; > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +static int > +__ucast_cache_validate(osm_ucast_cache_t * p_cache) For me it seems that whole validation can be better (and faster) performed during subnet discovery, many things are already there (details are below). Actually I thought already about having something like osm_sm.sweep_stat bitmask instead of just osm_sm.master_sm_found to indicate various events which were found during discovery. > +{ > + osm_switch_t * p_osm_sw; > + osm_node_t * p_osm_node; > + osm_node_t * p_remote_osm_node; > + osm_physp_t * p_osm_physp; > + osm_physp_t * p_remote_osm_physp; > + cache_sw_t * p_sw; > + cache_sw_t * p_remote_sw; > + cache_ca_t * p_remote_ca; > + uint16_t lid_ho; > + uint16_t remote_lid_ho; > + uint8_t remote_node_type; > + unsigned num_ports; > + unsigned i; > + int res = UCAST_CACHE_TOPOLOGY_MATCH; > + boolean_t fabric_link_exists; > + osm_log_t * p_log = p_cache->p_ucast_mgr->p_log; > + cl_qmap_t * p_osm_sw_guid_tbl; > + > + OSM_LOG_ENTER(p_log); > + > + p_osm_sw_guid_tbl = &p_cache->p_ucast_mgr->p_subn->sw_guid_tbl; > + > + if (cl_qmap_count(p_osm_sw_guid_tbl) > > + cl_qmap_count(&p_cache->sw_tbl)) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "current subnet has more switches than the cache - " > + "cache is invalid\n"); > + res |= UCAST_CACHE_TOPOLOGY_MORE_SWITCHES; > + goto Exit; > + } > + > + if (cl_qmap_count(p_osm_sw_guid_tbl) < > + cl_qmap_count(&p_cache->sw_tbl)) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "current subnet has less switches than the cache - " > + "continuing validation\n"); > + res |= UCAST_CACHE_TOPOLOGY_LESS_SWITCHES; > + } This handled already in drop manager. > + > + /* Clear the pointers to osm switch on all the cached switches. > + These pointers might be invalid right now: some cached switch > + might be missing in the real subnet, and some missing switch > + might reappear, such as in case of switch reboot. */ > + cl_qmap_apply_func(&p_cache->sw_tbl, __ucast_cache_sw_clear_osm_ptr, > + NULL); > + > + > + for (p_osm_sw = (osm_switch_t *) cl_qmap_head(p_osm_sw_guid_tbl); > + p_osm_sw != (osm_switch_t *) cl_qmap_end(p_osm_sw_guid_tbl); > + p_osm_sw = (osm_switch_t *) cl_qmap_next(&p_osm_sw->map_item)) { > + > + lid_ho = cl_ntoh16(osm_node_get_base_lid(p_osm_sw->p_node,0)); > + p_sw = __ucast_cache_get_sw(p_cache, lid_ho); > + if (!p_sw) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "new lid (0x%04x)is in the fabric - " > + "cache is invalid\n", lid_ho); > + res |= UCAST_CACHE_TOPOLOGY_NEW_LID; > + goto Exit; > + } New ports are tracked already with 'is_new' field of osm_port structure (it is necessary anyway for port in/out traps sending). > + > + p_sw->p_osm_sw = p_osm_sw; > + > + /* scan all the ports and check if the cache is valid */ > + > + p_osm_node = p_osm_sw->p_node; > + num_ports = osm_node_get_num_physp(p_osm_node); > + > + /* skipping port 0 on switches */ > + for (i = 1; i < num_ports; i++) { > + p_osm_physp = osm_node_get_physp_ptr(p_osm_node, i); > + > + fabric_link_exists = FALSE; > + if (p_osm_physp && > + osm_physp_is_valid(p_osm_physp) && osm_node_get_physp_ptr() returns NULL if port is not "valid". > + osm_link_is_healthy(p_osm_physp)) { > + p_remote_osm_physp = > + osm_physp_get_remote(p_osm_physp); > + if (p_remote_osm_physp && > + osm_physp_is_valid(p_remote_osm_physp) && > + osm_link_is_healthy(p_remote_osm_physp)) > + fabric_link_exists = TRUE; > + } > + > + if (!fabric_link_exists && !p_sw->ports[i]) > + continue; > + > + if (fabric_link_exists && !p_sw->ports[i]) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, link exists " > + "in the fabric, but not cached - " > + "cache is invalid\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_LINK_ADDED; > + goto Exit; > + } > + > + if (!fabric_link_exists && p_sw->ports[i]){ > + /* > + * link exists in cache, but missing > + * in current fabric > + */ > + if (p_sw->ports[i]->remote_node_type == > + IB_NODE_TYPE_SWITCH) { > + p_remote_sw = > + p_sw->ports[i]->remote_node.p_sw; > + /* cache is allowed to have a > + leaf switch that is missing > + in the current subnet */ > + if (!p_remote_sw->is_leaf) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "fabric is missing a link " > + "to non-leaf switch - " > + "cache is invalid\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_LINK_TO_SW_MISSING; > + goto Exit; > + } > + else { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "fabric is missing a link " > + "to leaf switch - " > + "continuing validation\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_LINK_TO_LEAF_SW_MISSING; > + continue; > + } > + } > + else { > + /* this means that link to > + non-switch node is missing */ > + CL_ASSERT(p_sw->is_leaf); > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "fabric is missing a link " > + "to CA - " > + "continuing validation\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_LINK_TO_CA_MISSING; > + continue; > + } > + } I think all this can be tracked in port_info. > + > + /* > + * Link exists both in fabric and in cache. > + * Compare remote nodes. > + */ > + > + p_remote_osm_node = > + osm_physp_get_node_ptr(p_remote_osm_physp); > + if (!p_remote_osm_node) { > + /* No node for remote port! > + Something wrong is going on here, > + so we better not use cache... */ > + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 3A53: " > + "lid 0x%04x, port %d, " > + "no node for remote port - " > + "cache mismatch\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + > + remote_node_type = > + osm_node_get_type(p_remote_osm_node); > + > + if (remote_node_type != > + p_sw->ports[i]->remote_node_type) { > + /* remote node type in the current fabric > + differs from the cached one - looks like > + node was replaced by something else */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "remote node type mismatch - " > + "cache is invalid\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } Why are nodes and not ports compared? Will this handle the case when CA port 1 was disconnected and port 2 connected by the same cable (and will get another LID value)? > + > + if (remote_node_type == IB_NODE_TYPE_SWITCH) { > + remote_lid_ho = > + cl_ntoh16(osm_node_get_base_lid( > + p_remote_osm_node, 0)); > + > + p_remote_sw = __ucast_cache_get_sw( > + p_cache, > + remote_lid_ho); And if switch was changed, but the same LID value reassigned for some reason? Wouldn't it be easier to compare port GUIDs? > + > + if (!p_remote_sw) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, " > + "new switch in the fabric - " > + "cache is invalid\n", > + remote_lid_ho); > + res |= UCAST_CACHE_TOPOLOGY_NEW_SWITCH; > + goto Exit; > + } > + > + if (p_sw->ports[i]->remote_node.p_sw != > + p_remote_sw) { > + /* remote cached switch that pointed > + by the port is not equal to the > + switch that was obtained for the > + remote lid - link was changed */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "link location changed " > + "(remote node mismatch) - " > + "cache is invalid\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } Could you elaborate, when will this be possible? (I'm starting to miss things :( ) > + } > + else { > + if (!p_sw->is_leaf) { > + /* remote node type is CA, but the > + cached switch is not marked as > + leaf - something has changed */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "link changed - " > + "cache is invalid\n", > + lid_ho, i); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + > + remote_lid_ho = > + cl_ntoh16(osm_physp_get_base_lid( > + p_remote_osm_physp)); > + > + p_remote_ca = __ucast_cache_get_ca( > + p_cache, remote_lid_ho); > + > + if (!p_remote_ca) { > + /* new lid is in the fabric - > + cache is invalid */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "new CA in the fabric " > + "(lid 0x%04x) - " > + "cache is invalid\n", > + lid_ho, i, remote_lid_ho); > + res |= UCAST_CACHE_TOPOLOGY_NEW_CA; > + goto Exit; > + } > + > + if (p_sw->ports[i]->remote_node.p_ca != > + p_remote_ca) { > + /* remote cached CA that pointed > + by the port is not equal to the > + CA that was obtained for the > + remote lid - link was changed */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "lid 0x%04x, port %d, " > + "link to CA (lid 0x%04x) " > + "has changed - " > + "cache is invalid\n", > + lid_ho, i, remote_lid_ho); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + } > + } /* done comparing the ports of the switch */ I think it will be *much* easies to track in osm_port_info.c - look at osm_physp's and osm_switch's need_update flag setup, and also where osm_node_unlink() called. > + } /* done comparing all the switches */ > + > + /* At this point we have four possible flags on: > + 1. UCAST_CACHE_TOPOLOGY_MATCH > + We have a perfect topology match to the cache > + 2. UCAST_CACHE_TOPOLOGY_LESS_SWITCHES > + Cached topology has one or more switches that do not exist > + in the current topology. There are two types of such switches: > + leaf switches and the regular switches. But if some regular > + switch was missing, we would exit the comparison with the > + UCAST_CACHE_TOPOLOGY_LINK_TO_SW_MISSING flag, so if some switch > + in the topology is missing, it has to be leaf switch. > + 3. UCAST_CACHE_TOPOLOGY_LINK_TO_LEAF_SW_MISSING > + One or more link to leaf switches are missing in the current > + topology. > + 4. UCAST_CACHE_TOPOLOGY_LINK_TO_CA_MISSING > + One or more CAs are missing in the current topology. > + In all these cases the cache is perfectly usable - it just might > + have routing to unexisting lids. */ > + > + if (res & UCAST_CACHE_TOPOLOGY_LESS_SWITCHES) { > + /* if there are switches in the cache that don't exist > + in the current topology, make sure that they are > + all leaf switches, otherwise cache is useless */ > + for (p_sw = (cache_sw_t *) cl_qmap_head(&p_cache->sw_tbl); > + p_sw != (cache_sw_t *) cl_qmap_end(&p_cache->sw_tbl); > + p_sw = (cache_sw_t *) cl_qmap_next(&p_sw->map_item)) { > + if (!p_sw->p_osm_sw && !p_sw->is_leaf) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "non-leaf switch in the fabric is " > + "missing - cache is invalid\n"); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + } > + } > + > + if ((res & UCAST_CACHE_TOPOLOGY_LINK_TO_LEAF_SW_MISSING) && > + !(res & UCAST_CACHE_TOPOLOGY_LESS_SWITCHES)) { > + /* some link to leaf switch is missing, but there are > + no missing switches - link failure or topology > + changes, which means that we probably shouldn't > + use the cache here */ > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "topology change - cache is invalid\n"); > + res |= UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + > + Exit: > + OSM_LOG_EXIT(p_log); > + return res; > + > +} /* __ucast_cache_validate() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +int > +osm_ucast_cache_apply(osm_ucast_cache_t * p_cache) > +{ > + int res = 0; > + osm_log_t * p_log; > + > + if (!p_cache) > + return 1; > + > + p_log = p_cache->p_ucast_mgr->p_log; > + > + OSM_LOG_ENTER(p_log); > + if (!p_cache->topology_valid) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "unicast cache is empty - can't " > + "use it on this sweep\n"); > + res = UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + > + if (!p_cache->routing_valid) { > + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 3A55: " > + "cached routing invalid\n"); > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "invalidating cache\n"); > + __ucast_cache_invalidate(p_cache); > + res = UCAST_CACHE_TOPOLOGY_NO_MATCH; > + goto Exit; > + } > + > + res = __ucast_cache_validate(p_cache); > + > + if ((res & UCAST_CACHE_TOPOLOGY_NO_MATCH ) || > + (res & UCAST_CACHE_TOPOLOGY_MORE_SWITCHES ) || > + (res & UCAST_CACHE_TOPOLOGY_LINK_ADDED ) || > + (res & UCAST_CACHE_TOPOLOGY_LINK_TO_SW_MISSING) || > + (res & UCAST_CACHE_TOPOLOGY_NEW_SWITCH ) || > + (res & UCAST_CACHE_TOPOLOGY_NEW_CA ) || > + (res & UCAST_CACHE_TOPOLOGY_NEW_LID ) || > + (res & UCAST_CACHE_TOPOLOGY_LINK_TO_SW_MISSING)) { Why to not make single return status? > + /* The change in topology doesn't allow us to use the. > + existing cache. Cache should be invalidated, and new > + cache should be built after the routing recalculation. */ > + OSM_LOG(p_log, OSM_LOG_INFO, > + "changes in topology (0x%x) - " > + "invalidating cache\n", res); > + __ucast_cache_invalidate(p_cache); > + goto Exit; > + } > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "cache is valid (status 0x%04x) - using the cached routing\n",res); > + > + /* existing cache can be used - write back the cached routing */ > + __ucast_cache_write_routing(p_cache); > + > + /* > + * ToDo: Detailed result of the topology comparison will > + * ToDo: be needed later for the Incremental Routing, > + * ToDo: where based on this result, the routing algorithm > + * ToDo: will try to route "around" the missing components. > + * ToDo: For now - reset the result whenever the cache > + * ToDo: is valid. > + */ > + res = 0; > + > + Exit: > + OSM_LOG_EXIT(p_log); > + return res; > +} /* osm_ucast_cache_apply() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +void osm_ucast_cache_set_sw_fwd_table(osm_ucast_cache_t * p_cache, > + uint8_t * ucast_mgr_lft_buf, > + osm_switch_t * p_osm_sw) > +{ > + uint16_t lid_ho = > + cl_ntoh16(osm_node_get_base_lid(p_osm_sw->p_node, 0)); > + cache_sw_t * p_sw = __ucast_cache_get_sw(p_cache, lid_ho); > + > + OSM_LOG_ENTER(p_cache->p_ucast_mgr->p_log); > + > + OSM_LOG(p_cache->p_ucast_mgr->p_log, OSM_LOG_VERBOSE, > + "caching lft for switch 0x%04x\n", > + lid_ho); > + > + if (!p_sw || !p_sw->p_osm_sw) { > + OSM_LOG(p_cache->p_ucast_mgr->p_log, OSM_LOG_ERROR, > + "ERR 3A57: " > + "fabric switch 0x%04x %s in the unicast cache\n", > + lid_ho, > + (p_sw) ? "is not initialized" : "doesn't exist"); > + goto Exit; > + } > + > + CL_ASSERT(p_sw->p_osm_sw == p_osm_sw); > + CL_ASSERT(!p_sw->lft_buff); > + > + p_sw->max_lid_ho = p_osm_sw->max_lid_ho; > + > + /* allocate linear forwarding table buffer and fill it */ > + p_sw->lft_buff = (uint8_t *)malloc(IB_LID_UCAST_END_HO + 1); > + memcpy(p_sw->lft_buff, p_cache->p_ucast_mgr->lft_buf, > + IB_LID_UCAST_END_HO + 1); > + > + Exit: > + OSM_LOG_EXIT(p_cache->p_ucast_mgr->p_log); > +} /* osm_ucast_cache_set_sw_fwd_table() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +void osm_ucast_cache_refresh_topo(osm_ucast_cache_t * p_cache) > +{ > + osm_log_t * p_log = p_cache->p_ucast_mgr->p_log; > + OSM_LOG_ENTER(p_log); > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "starting ucast cache topology refresh\n"); > + > + if (p_cache->topology_valid) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "invalidating existing ucast cache\n"); > + __ucast_cache_invalidate(p_cache); > + } > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, "caching topology\n"); > + > + if (__ucast_cache_read_topology(p_cache) != 0) { > + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 3A56: " > + "cache population failed\n"); > + __ucast_cache_invalidate(p_cache); > + goto Exit; > + } > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "ucast cache topology refresh done\n"); > + Exit: > + OSM_LOG_EXIT(p_log); > +} /* osm_ucast_cache_refresh_topo() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +void osm_ucast_cache_refresh_lid_matrices(osm_ucast_cache_t * p_cache) > +{ > + osm_log_t * p_log = p_cache->p_ucast_mgr->p_log; > + OSM_LOG_ENTER(p_log); > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "starting ucast cache lid matrices refresh\n"); > + > + if (!p_cache->topology_valid) { > + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 3A54: " > + "cached topology is invalid\n"); > + goto Exit; > + } > + > + if (p_cache->routing_valid) { > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "invalidating existing ucast routing cache\n"); > + __ucast_cache_clear_routing(p_cache); > + } > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "caching lid matrices\n"); > + > + __ucast_cache_read_lid_matrices(p_cache); > + > + OSM_LOG(p_log, OSM_LOG_VERBOSE, > + "ucast cache lid matrices refresh done\n"); > + Exit: > + OSM_LOG_EXIT(p_log); > +} /* osm_ucast_cache_refresh_lid_matrices() */ > + > +/********************************************************************** > + **********************************************************************/ > + > +osm_ucast_cache_t * > +osm_ucast_cache_construct(osm_ucast_mgr_t * const p_mgr) > +{ > + if (p_mgr->p_subn->opt.lmc > 0) { > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A50: " > + "Unicast cache is not supported for LMC>0\n"); > + return NULL; > + } > + > + osm_ucast_cache_t * p_cache = > + (osm_ucast_cache_t*)malloc(sizeof(osm_ucast_cache_t)); > + if (!p_cache) > + return NULL; > + > + memset(p_cache, 0, sizeof(osm_ucast_cache_t)); > + > + cl_qmap_init(&p_cache->sw_tbl); > + cl_qmap_init(&p_cache->ca_tbl); > + p_cache->p_ucast_mgr = p_mgr; > + > + return p_cache; > +} > + > +/********************************************************************** > + **********************************************************************/ > + > +void > +osm_ucast_cache_destroy(osm_ucast_cache_t * p_cache) > +{ > + if (!p_cache) > + return; > + __ucast_cache_invalidate(p_cache); > + free(p_cache); > +} > + > +/********************************************************************** > + **********************************************************************/ > -- > 1.5.1.4 > From sashak at voltaire.com Sun Jun 29 14:50:13 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 00:50:13 +0300 Subject: [ofa-general] Re: [PATCH 0/4] opensm: Unicast Routing Cache In-Reply-To: <20080629213306.GF12659@sashak.voltaire.com> References: <481D888A.7080608@dev.mellanox.co.il> <20080629213306.GF12659@sashak.voltaire.com> Message-ID: <20080629215013.GH12659@sashak.voltaire.com> On 00:33 Mon 30 Jun , Sasha Khapyorsky wrote: > > Another thing we may want to add (but it is not related to routing cache > or incremental routing) it to keep two sets of LFTs with switch object > for validation purposes - "requested" (filled by routing algorithm) and > "real" (filled from responded MADs). Actually even this is not needed. If OpenSM tried to set some LFT block and failed it is better to try again (or re-request a heavy sweep), such validation could be done on per-block basic in LFT MADs receiver - nothing fancy. Sasha From Dianne at bortolomiol.com Mon Jun 30 15:45:28 2008 From: Dianne at bortolomiol.com (Dianne Chappell) Date: Mon, 30 Jun 2008 01:45:28 -2100 Subject: [ofa-general] Luxury costs cheaper now Message-ID: <3fe801c8da39$d3e15490$c0a80203@Dianne> Bring these perfectly crafted luxurious items home to your wife! http://dimaalve.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Sun Jun 29 16:37:31 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 02:37:31 +0300 Subject: [ofa-general] [PATCH] opensm/osm_sa_lft_record: validate LFT block number Message-ID: <20080629233731.GI12659@sashak.voltaire.com> Return nothing when block number was specified but not configured in switch's LFT. Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_sa_lft_record.c | 11 ++++++----- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/opensm/opensm/osm_sa_lft_record.c b/opensm/opensm/osm_sa_lft_record.c index 71f624e..b67430c 100644 --- a/opensm/opensm/osm_sa_lft_record.c +++ b/opensm/opensm/osm_sa_lft_record.c @@ -187,13 +187,14 @@ __osm_lftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, } /* now we need to decide which blocks to output */ + max_block = osm_switch_get_max_block_id_in_use(p_sw); if (comp_mask & IB_LFTR_COMPMASK_BLOCK) { - max_block = min_block = cl_ntoh16(p_rcvd_rec->block_num); - } else { - /* use as many blocks as "in use" */ + min_block = cl_ntoh16(p_rcvd_rec->block_num); + if (min_block > max_block) + return; + max_block = min_block; + } else /* use as many blocks as "in use" */ min_block = 0; - max_block = osm_switch_get_max_block_id_in_use(p_sw); - } /* so we can add these blocks one by one ... */ for (block = min_block; block <= max_block; block++) -- 1.5.5.1.178.g1f811 From paulquinlan at bellnet.ca Sun Jun 29 18:14:37 2008 From: paulquinlan at bellnet.ca (Irish Lottery News Center) Date: Sun, 29 Jun 2008 21:14:37 -0400 Subject: [ofa-general] Confirm Your Winning Amount Message-ID: <20080630011438.LJCB1729.tomts35-srv.bellnexxia.net@toip39-bus.srvr.bell.ca> Ref:UK/9420X2/68 Batch:074/05/ZY369 You have won £750, 000.00. To claim your prize,Contact Mr Derek White with the following informations. Name,Address,Age,Occupation,Country Email:irn.customersupportunit at live.com Mr Clark Johnson From ogerlitz at voltaire.com Sun Jun 29 21:51:44 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 30 Jun 2008 07:51:44 +0300 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <4867B28D.5090603@opengridcomputing.com> References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> Message-ID: <48686660.9020001@voltaire.com> Steve Wise wrote: > Sort of. It is not exactly a dma_mr. dma_mrs allow remote IO. The > zero-stag is an iWARP verbs concept. zero-stag can be used by kernel > users as the lkey in any SGE that requires only local access rights. > If the lkey is zero, then the rnic interprets the addr field in the > SGE as a bus address. > > I see. Assuming we want to narrow the differences between RDMA transports from the ULP point of view and that IB does not support this verb - I'd like to see if its possible to have zero stags implemented as dma mrs. How about enhancing the iWARP HW drivers implementation of get_dma_mr to switch on the access flags provided by the caller, and if no remote access it needed, just return zero stag for mr.lkey without doing any registration. Or. From rdreier at cisco.com Sun Jun 29 22:09:32 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 29 Jun 2008 22:09:32 -0700 Subject: [ofa-general] Re: [PATCH RFCv3 1/2] RDMA: Add protocol statistics attributes to the RDMA sysfs. In-Reply-To: <20080619153424.15418.99244.stgit@dell3.ogc.int> (Steve Wise's message of "Thu, 19 Jun 2008 10:34:24 -0500") References: <20080619153252.15418.21923.stgit@dell3.ogc.int> <20080619153424.15418.99244.stgit@dell3.ogc.int> Message-ID: > + u64 ipOutNoRoutes; > + u64 ipOutNoRoutes; should the fact that this doesn't compile make me wonder about how much testing was done? anyway I fixed this up and applied it... From rdreier at cisco.com Sun Jun 29 22:15:50 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 29 Jun 2008 22:15:50 -0700 Subject: [ofa-general] Re: [PATCH RFCv3 2/2] RDMA/cxgb3: Add support for protocol statistics. In-Reply-To: <20080619153426.15418.30544.stgit@dell3.ogc.int> (Steve Wise's message of "Thu, 19 Jun 2008 10:34:26 -0500") References: <20080619153252.15418.21923.stgit@dell3.ogc.int> <20080619153426.15418.30544.stgit@dell3.ogc.int> Message-ID: > --- a/include/rdma/ib_verbs.h > +++ b/include/rdma/ib_verbs.h > @@ -61,7 +61,7 @@ struct iw_protocol_stats { > u64 ipInReceives; > u64 ipInHdrErrors; > u64 ipInTooBigErrors; > - u64 ipOutNoRoutes; > + u64 ipInNoRoutes; > u64 ipInAddrErrors; > u64 ipInUnknownProtos; > u64 ipInTruncatedPkts; heh, I know exactly what happened now. anyway, applied both patches, thanks. - R. From rdreier at cisco.com Sun Jun 29 22:55:50 2008 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 29 Jun 2008 22:55:50 -0700 Subject: [ofa-general] Re: [PATCH 3/3 v4] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: (Ron Livne's message of "Sun, 29 Jun 2008 10:08:10 +0000 (UTC)") References: Message-ID: thanks, applied all 3. two requests for future patches that make it easier for me to apply things without editing by hand: > There is also an improvment of 12% in cpu usage. > > > Signed-off-by: Ron Livne > > Changes in v2: > IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called > IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. > > Changes in v3: > Whitespaces deleted. > > Changes in v4: > The redundant initialization for create_flags (create_flags = 0) was deleted. > > --- Please put the patch's changelog and any other information that should not be included in the actual commit message after the "---" line; then it will be automatically stripped by standard patch import scripts. > --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:12:37.000000000 +0300 > +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-29 09:42:41.000000000 +0300 Please generate your patch so it applied with "git apply" or "patch -p1" -- in other words the paths should just be like "old/drivers/infiniband/...". From kliteyn at dev.mellanox.co.il Sun Jun 29 23:45:38 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 09:45:38 +0300 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <20080625165534.GS22159@sashak.voltaire.com> References: <4861E87C.3000007@dev.mellanox.co.il> <20080625165534.GS22159@sashak.voltaire.com> Message-ID: <48688112.8070006@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Hi Yevgeny, > > On 09:41 Wed 25 Jun , Yevgeny Kliteynik wrote: >> Fixing some data types and corresponding log messages, >> thus solving compiler warnings. >> >> Signed-off-by: Yevgeny Kliteynik >> --- >> opensm/opensm/osm_state_mgr.c | 9 +++++---- >> 1 files changed, 5 insertions(+), 4 deletions(-) >> >> diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c >> index f15a4af..6ca276b 100644 >> --- a/opensm/opensm/osm_state_mgr.c >> +++ b/opensm/opensm/osm_state_mgr.c >> @@ -863,7 +863,8 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) >> osm_port_t *p_port; >> osm_port_t *p_next_port; >> cl_ptr_vector_t *p_port_lid_tbl; >> - size_t max_lid, ref_size, curr_size, lid; >> + size_t ref_size, curr_size; >> + uint16_t max_lid, lid; > > Finally it stores ref_size or cur_size values in max_lid, so it can > overflow. Could you be more specific about compiler warnings? Sure: /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c: In function __osm_state_mgr_check_tbl_consistency: /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c:922: warning: format %u expects type unsigned int, but argument 5 has type size_t /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c:934: warning: format %u expects type unsigned int, but argument 8 has type size_t /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c:946: warning: format %u expects type unsigned int, but argument 5 has type size_t Patch shortly. -- Yevgeny From kliteyn at dev.mellanox.co.il Sun Jun 29 23:49:05 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 09:49:05 +0300 Subject: [ofa-general] [PATCH v2] [TRIVIAL] opensm/osm_state_mgr.c: fixing some data types Message-ID: <486881E1.1090603@dev.mellanox.co.il> Fixing some data types and solving compiler warnings in the corresponding log messages. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_state_mgr.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index 9a8409b..734df79 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -862,7 +862,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) osm_port_t *p_port; osm_port_t *p_next_port; cl_ptr_vector_t *p_port_lid_tbl; - size_t max_lid, ref_size, curr_size, lid; + uint16_t max_lid, ref_size, curr_size, lid; osm_port_t *p_port_ref, *p_port_stored; cl_ptr_vector_t ref_port_lid_tbl; uint16_t min_lid_ho; -- 1.5.1.4 From kliteyn at dev.mellanox.co.il Sun Jun 29 23:53:52 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 09:53:52 +0300 Subject: [ofa-general] [PATCH] opensm/include/iba/ib_types.h: Definition of Congestion Control MADs Message-ID: <48688300.5050902@dev.mellanox.co.il> Hi Sasha, Adding definition of all the Congestion Control (CC) MADs to ib_types.h (IB Spec, Annex 10). Signed-off-by: Yevgeny Kliteynik --- opensm/include/iba/ib_types.h | 786 +++++++++++++++++++++++++++++++++++++++++ 1 files changed, 786 insertions(+), 0 deletions(-) diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index 09ec257..81d16d3 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -443,6 +443,17 @@ BEGIN_C_DECLS */ #define IB_MCLASS_PERF 0x04 /**********/ +/****d* IBA Base: Constants/IB_MCLASS_CC +* NAME +* IB_MCLASS_CC +* +* DESCRIPTION +* Management Class, Congestion Control (A10.4.1) +* +* SOURCE +*/ +#define IB_MCLASS_CC 0x21 +/**********/ /****d* IBA Base: Constants/IB_MCLASS_BM * NAME * IB_MCLASS_BM @@ -1005,6 +1016,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_SMPL_RSLT (CL_HTON16(0x0011)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CONG_INFO +* NAME +* IB_MAD_ATTR_CONG_INFO +* +* DESCRIPTION +* CongestionInfo attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CONG_INFO (CL_HTON16(0x0011)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_SWITCH_INFO * NAME * IB_MAD_ATTR_SWITCH_INFO @@ -1029,6 +1052,30 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_CNTRS (CL_HTON16(0x0012)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CONG_KEY_INFO +* NAME +* IB_MAD_ATTR_CONG_KEY_INFO +* +* DESCRIPTION +* CongestionKeyInfo attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CONG_KEY_INFO (CL_HTON16(0x0012)) +/**********/ + +/****d* IBA Base: Constants/IB_MAD_ATTR_CONG_LOG +* NAME +* IB_MAD_ATTR_CONG_LOG +* +* DESCRIPTION +* CongestionLog attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CONG_LOG (CL_HTON16(0x0013)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_GUID_INFO * NAME * IB_MAD_ATTR_GUID_INFO @@ -1041,6 +1088,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_GUID_INFO (CL_HTON16(0x0014)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_SW_CONG_SETTING +* NAME +* IB_MAD_ATTR_SW_CONG_SETTING +* +* DESCRIPTION +* SwitchCongestionSetting attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_SW_CONG_SETTING (CL_HTON16(0x0014)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_PORT_INFO * NAME * IB_MAD_ATTR_PORT_INFO @@ -1053,6 +1112,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_INFO (CL_HTON16(0x0015)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_SW_PORT_CONG_SETTING +* NAME +* IB_MAD_ATTR_SW_PORT_CONG_SETTING +* +* DESCRIPTION +* SwitchPortCongestionSetting attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_SW_PORT_CONG_SETTING (CL_HTON16(0x0015)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_P_KEY_TABLE * NAME * IB_MAD_ATTR_P_KEY_TABLE @@ -1065,6 +1136,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_P_KEY_TABLE (CL_HTON16(0x0016)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CA_CONG_SETTING +* NAME +* IB_MAD_ATTR_CA_CONG_SETTING +* +* DESCRIPTION +* CACongestionSetting attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CA_CONG_SETTING (CL_HTON16(0x0016)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_SLVL_TABLE * NAME * IB_MAD_ATTR_SLVL_TABLE @@ -1077,6 +1160,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_SLVL_TABLE (CL_HTON16(0x0017)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CC_TBL +* NAME +* IB_MAD_ATTR_CC_TBL +* +* DESCRIPTION +* CongestionControlTable attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CC_TBL (CL_HTON16(0x0017)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_VL_ARBITRATION * NAME * IB_MAD_ATTR_VL_ARBITRATION @@ -1089,6 +1184,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_VL_ARBITRATION (CL_HTON16(0x0018)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_TIME_STAMP +* NAME +* IB_MAD_ATTR_TIME_STAMP +* +* DESCRIPTION +* TimeStamp attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_TIME_STAMP (CL_HTON16(0x0018)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_LIN_FWD_TBL * NAME * IB_MAD_ATTR_LIN_FWD_TBL @@ -3246,6 +3353,23 @@ ib_path_rec_hop_limit(IN const ib_path_rec_t * const p_rec) #define IB_CLASS_RESP_TIME_MASK 0x1F /*********/ +/****s* IBA Base: Constants/IB_CLASS_ENH_PORT0_CC_MASK +* NAME +* IB_CLASS_ENH_PORT0_CC_MASK +* +* DESCRIPTION +* ClassPortInfo CapabilityMask bits. +* Switch only: This bit will be set if the EnhacedPort0 +* supports CA Congestion Control (A10.4.3.1). +* +* SEE ALSO +* ib_class_port_info_t +* +* SOURCE +*/ +#define IB_CLASS_ENH_PORT0_CC_MASK 0x0100 +/*********/ + /****s* IBA Base: Types/ib_class_port_info_t * NAME * ib_class_port_info_t @@ -7126,6 +7250,19 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated uint8_t details[54]; } PACK_SUFFIX raw_data; + struct _ntc_0 { // CC_Key violation [54 bytes] + ib_net16_t source_lid; // Source LID from offending packet LRH + uint8_t method; // Method, from common MAD header + uint8_t resv0; + ib_net16_t attr_id; // Attribute ID, from common MAD header + ib_net16_t resv1; + ib_net32_t attr_mod; // Attribute Modif, from common MAD header + ib_net32_t qp; // 8b pad, 24b dest QP from BTH + ib_net64_t cc_key; // CC key of the offending packet + ib_gid_t source_gid; // GID from GRH of the offending packet + uint8_t padding[14]; // Padding - ignored on read + } PACK_SUFFIX ntc_0; + struct _ntc_64_67 { uint8_t res[6]; ib_gid_t gid; // the Node or Multicast Group that came in/out @@ -7219,6 +7356,11 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated #define TRAP_144_MASK_LINK_WIDTH_ENABLE_CHANGE (CL_HTON16(0x0002)) #define TRAP_144_MASK_NODE_DESCRIPTION_CHANGE (CL_HTON16(0x0001)) +/** + * Trap 0 masks + */ +#define TRAP_0_MASK_QP (CL_HTON32(0x00FFFFFF)) + /****f* IBA Base: Types/ib_notice_is_generic * NAME * ib_notice_is_generic @@ -7843,6 +7985,650 @@ typedef struct _ib_port_samples_result { } PACK_SUFFIX ib_port_samples_result_t; #include +/****s* IBA Base: Types/ib_cc_mad_t +* NAME +* ib_cc_mad_t +* +* DESCRIPTION +* IBA defined Congestion Control MAD format. (A10.4.1) +* +* SYNOPSIS +*/ +#define IB_CC_LOG_DATA_SIZE 32 +#define IB_CC_MGT_DATA_SIZE 192 +#define IB_CC_MAD_HDR_SIZE (sizeof(ib_sa_mad_t) - IB_CC_LOG_DATA_SIZE \ + - IB_CC_MGT_DATA_SIZE) + +#include +typedef struct _ib_cc_mad { + ib_mad_t header; + ib_net64_t cc_key; + uint8_t log_data[IB_CC_LOG_DATA_SIZE]; + uint8_t mgt_data[IB_CC_MGT_DATA_SIZE]; +} PACK_SUFFIX ib_cc_mad_t; +#include +/* +* FIELDS +* header +* Common MAD header. +* +* cc_key +* CC_Key of the Congestion Control MAD. +* +* log_data +* Congestion Control log data of the CC MAD. +* +* mgt_data +* Congestion Control management data of the CC MAD. +* +* SEE ALSO +* ib_mad_t +*********/ + +/****f* IBA Base: Types/ib_cc_mad_get_cc_key +* NAME +* ib_cc_mad_get_cc_key +* +* DESCRIPTION +* Gets a CC_Key of the CC MAD. +* +* SYNOPSIS +*/ +static inline ib_net64_t OSM_API +ib_cc_mad_get_cc_key(IN const ib_cc_mad_t * const p_cc_mad) +{ + return p_cc_mad->cc_key; +} +/* +* PARAMETERS +* p_cc_mad +* [in] Pointer to the CC MAD packet. +* +* RETURN VALUES +* CC_Key of the provided CC MAD packet. +* +* NOTES +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****f* IBA Base: Types/ib_cc_mad_get_log_data_ptr +* NAME +* ib_cc_mad_get_mgt_data_ptr +* +* DESCRIPTION +* Gets a pointer to the CC MAD's log data area. +* +* SYNOPSIS +*/ +static inline void * OSM_API +ib_cc_mad_get_log_data_ptr(IN const ib_cc_mad_t * const p_cc_mad) +{ + return ((void *)p_cc_mad->log_data); +} +/* +* PARAMETERS +* p_cc_mad +* [in] Pointer to the CC MAD packet. +* +* RETURN VALUES +* Pointer to CC MAD log data area. +* +* NOTES +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****f* IBA Base: Types/ib_cc_mad_get_mgt_data_ptr +* NAME +* ib_cc_mad_get_mgt_data_ptr +* +* DESCRIPTION +* Gets a pointer to the CC MAD's management data area. +* +* SYNOPSIS +*/ +static inline void * OSM_API +ib_cc_mad_get_mgt_data_ptr(IN const ib_cc_mad_t * const p_cc_mad) +{ + return ((void *)p_cc_mad->mgt_data); +} +/* +* PARAMETERS +* p_cc_mad +* [in] Pointer to the CC MAD packet. +* +* RETURN VALUES +* Pointer to CC MAD management data area. +* +* NOTES +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cong_info_t +* NAME +* ib_cong_info_t +* +* DESCRIPTION +* IBA defined CongestionInfo attribute (A10.4.3.3) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_info { + uint8_t cong_info; + uint8_t resv; + uint8_t ctrl_table_cap; +} PACK_SUFFIX ib_cong_info_t; +#include +/* +* FIELDS +* cong_info +* Congestion control capabilities of the node. +* +* ctrl_table_cap +* Number of 64 entry blocks in the CongestionControlTable. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cong_key_info_t +* NAME +* ib_cong_key_info_t +* +* DESCRIPTION +* IBA defined CongestionKeyInfo attribute (A10.4.3.4) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_key_info { + ib_net64_t cc_key; + uint8_t protect_bit; + uint8_t resv; + ib_net16_t lease_period; + ib_net16_t violations; +} PACK_SUFFIX ib_cong_key_info_t; +#include +/* +* FIELDS +* cc_key +* 8-byte CC Key. +* +* protect_bit +* LSB is a CC Key Protect Bit, other 7 bits are reserved. +* +* lease_period +* How long the CC Key protect bit is to remain non-zero. +* +* violations +* Number of received MADs that violated CC Key. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cong_log_event_sw_t +* NAME +* ib_cong_log_event_sw_t +* +* DESCRIPTION +* IBA defined CongestionLogEvent (SW) entry (A10.4.3.5) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_log_event_sw { + ib_net16_t slid; + ib_net16_t dlid; + uint8_t resv0_sl; + uint8_t resv1; + ib_net16_t resv2; + ib_net32_t time_stamp; +} PACK_SUFFIX ib_cong_log_event_sw_t; +#include +/* +* FIELDS +* slid +* Source LID of congestion event. +* +* slid +* Destination LID of congestion event. +* +* sl +* bits [3:0] SL of congestion event. +* bits [7:4] reserved. +* +* time_stamp +* Timestamp of congestion event. +* +* SEE ALSO +* ib_cc_mad_t, ib_cong_log_t +*********/ + +/****s* IBA Base: Types/ib_cong_log_event_ca_t +* NAME +* ib_cong_log_event_ca_t +* +* DESCRIPTION +* IBA defined CongestionLogEvent (CA) entry (A10.4.3.5) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_log_event_ca { + ib_net32_t resv0_local_qp; + ib_net32_t remote_qp_sl_service_type; + ib_net16_t remote_lid; + ib_net16_t resv1; + ib_net32_t time_stamp; +} PACK_SUFFIX ib_cong_log_event_ca_t; +#include +/* +* FIELDS +* resv0_local_qp +* bits [23:0] local QP that reached CN threshold. +* bits [31:24] reserved. +* +* remote_qp_sl_service_type +* bits [23:0] remote QP that is connected to local QP. +* bits [27:24] SL of the local QP. +* bits [31:28] Service Type of the local QP. +* +* remote_lid +* LID of the remote port that is connected to local QP. +* +* time_stamp +* Timestamp when threshold reached. +* +* SEE ALSO +* ib_cc_mad_t, ib_cong_log_t +*********/ + +/****s* IBA Base: Types/ib_cong_log_t +* NAME +* ib_cong_log_t +* +* DESCRIPTION +* IBA defined CongestionLog attribute (A10.4.3.5) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_log { + uint8_t log_type; + union _log_details + { + struct _log_sw { + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net32_t time_stamp; + uint8_t port_map[32]; + ib_cong_log_event_sw_t entry_list[15]; + } PACK_SUFFIX log_sw; + + struct _log_ca { + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net16_t event_map; + ib_net16_t resv; + ib_net32_t time_stamp; + ib_cong_log_event_ca_t log_event[13]; + } PACK_SUFFIX log_ca; + + } log_details; +} PACK_SUFFIX ib_cong_log_t; +#include +/* +* FIELDS +* +* log_{sw,ca}.log_type +* Log type: 0x1 is for Switch, 0x2 is for CA +* +* log_{sw,ca}.cong_flags +* Congestion Flags. +* +* log_{sw,ca}.event_counter +* Number of events since log last sent. +* +* log_{sw,ca}.time_stamp +* Timestamp when log sent. +* +* log_sw.port_map +* If a bit set to 1, then the corresponding port +* has marked packets with a FECN. +* bits 0 and 255 - reserved +* bits [254..1] - ports [254..1]. +* +* log_sw.entry_list +* Array of 13 most recent congestion log events. +* +* log_ca.event_map +* array 16 bits, one for each SL. +* +* log_ca.log_event +* Array of 13 most recent congestion log events. +* +* SEE ALSO +* ib_cc_mad_t, ib_cong_log_event_sw_t, ib_cong_log_event_ca_t +*********/ + +/****s* IBA Base: Types/ib_sw_cong_setting_t +* NAME +* ib_sw_cong_setting_t +* +* DESCRIPTION +* IBA defined SwitchCongestionSetting attribute (A10.4.3.6) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_sw_cong_setting { + ib_net32_t control_map; + uint8_t victim_mask[32]; + uint8_t credit_mask[32]; + uint8_t threshold; + uint8_t packet_size; + uint8_t cs_threshold; + uint8_t resv0; + ib_net16_t cs_return_delay; + ib_net16_t marking_rate; +} PACK_SUFFIX ib_sw_cong_setting_t; +#include +/* +* FIELDS +* +* control_map +* Indicates which components of this attribute are valid +* +* victim_mask +* If the bit set to 1, then the port corresponding to +* that bit shall mark packets that encounter congestion +* with a FECN, whether they are the source or victim +* of congestion. (See A10.2.1.1.1) +* bit 0: port 0 (enhanced port 0 only) +* bits [254..1]: ports [254..1] +* bit 255: reserved +* +* credit_mask +* If the bit set to 1, then the port corresponding +* to that bit shall apply Credit Starvation. +* bit 0: port 0 (enhanced port 0 only) +* bits [254..1]: ports [254..1] +* bit 255: reserved +* +* threshold +* bits [3..0] Indicates how agressive cong. marking should be +* bits [7..4] Reserved +* +* packet_size +* Any packet less than this size won't be marked with FECN +* +* cs_threshold +* bits [3..0] How agressive Credit Starvation should be +* bits [7..4] Reserved +* +* cs_return_delay +* Value that controls credit return rate. +* +* marking_rate +* The value that provides the mean number of packets +* between marking eligible packets with FECN. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_sw_port_cong_setting_element_t +* NAME +* ib_sw_port_cong_setting_element_t +* +* DESCRIPTION +* IBA defined SwitchPortCongestionSettingElement (A10.4.3.7) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_sw_port_cong_setting_element { + uint8_t valid_ctrl_type_res_threshold; + uint8_t packet_size; + ib_net16_t cong_param; +} PACK_SUFFIX ib_sw_port_cong_setting_element_t; +#include +/* +* FIELDS +* +* valid_ctrl_type_res_threshold +* bit 0: "Valid" +* when set to 1, indicates this switch +* port congestion setting element is valid. +* bit 1: "Control Type" +* Indicates which type of attribute is being set: +* 0b = Congestion Control parameters are being set. +* 1b = Credit Starvation parameters are being set. +* bits [3..2]: reserved +* bits [7..4]: "Threshold" +* When Control Type is 0, contains the congestion +* threshold value (Threshold) for this port. +* When Control Type is 1, contains the credit +* starvation threshold (CS_Threshold) value for +* this port. +* +* packet_size +* When Control Type is 0, this field contains the minimum +* size of packets that may be marked with a FECN. +* When Control Type is 1, this field is reserved. +* +* cong_parm +* When Control Type is 0, this field contains the port +* marking_rate. +* When Control Type is 1, this field is reserved. +* +* SEE ALSO +* ib_cc_mad_t, ib_sw_port_cong_setting_t +*********/ + +/****d* IBA Base: Types/ib_sw_port_cong_setting_block_t +* NAME +* ib_sw_port_cong_setting_block_t +* +* DESCRIPTION +* Defines the SwitchPortCongestionSetting Block (A10.4.3.7). +* +* SOURCE +*/ +typedef ib_sw_port_cong_setting_element_t ib_sw_port_cong_setting_block_t[32]; +/**********/ + +/****s* IBA Base: Types/ib_sw_port_cong_setting_t +* NAME +* ib_sw_port_cong_setting_t +* +* DESCRIPTION +* IBA defined SwitchPortCongestionSetting attribute (A10.4.3.7) +* +* SYNOPSIS +*/ + +#include +typedef struct _ib_sw_port_cong_setting { + ib_sw_port_cong_setting_block_t block; +} PACK_SUFFIX ib_sw_port_cong_setting_t; +#include +/* +* FIELDS +* +* block +* SwitchPortCongestionSetting block. +* +* SEE ALSO +* ib_cc_mad_t, ib_sw_port_cong_setting_element_t +*********/ + +/****s* IBA Base: Types/ib_ca_cong_entry_t +* NAME +* ib_ca_cong_entry_t +* +* DESCRIPTION +* IBA defined CACongestionEntry (A10.4.3.8) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_ca_cong_entry { + ib_net16_t ccti_timer; + uint8_t ccti_increase; + uint8_t trigger_threshold; + uint8_t ccti_min; + uint8_t resv0; + ib_net16_t resv1; +} PACK_SUFFIX ib_ca_cong_entry_t; +#include +/* +* FIELDS +* +* ccti_timer +* When the timer expires it will be reset to its specified +* value, and 1 will be decremented from the CCTI. +* +* ccti_increase +* The number to be added to the table Index (CCTI) +* on the receipt of a BECN. +* +* trigger_threshold +* When the CCTI is equal to this value, an event +* is logged in the CAs cyclic event log. +* +* ccti_min +* The minimum value permitted for the CCTI. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_ca_cong_setting_t +* NAME +* ib_ca_cong_setting_t +* +* DESCRIPTION +* IBA defined CACongestionSetting attribute (A10.4.3.8) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_ca_cong_setting { + uint8_t port_control; + uint8_t resv0; + ib_net16_t control_map; + ib_ca_cong_entry_t entry_list[16]; +} PACK_SUFFIX ib_ca_cong_setting_t; +#include +/* +* FIELDS +* +* port_control +* Congestion attributes for this port: +* LSB = 0: QP based CC +* LSB = 1: SL/Port based CC +* All other bits are reserved +* +* control_map +* An array of sixteen bits, one for each SL. Each bit indicates +* whether or not the corresponding entry is to be modified. +* +* entry_list +* List of 16 CACongestionEntries, one per SL. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cc_tbl_entry_t +* NAME +* ib_cc_tbl_entry_t +* +* DESCRIPTION +* IBA defined CongestionControlTableEntry (A10.4.3.9) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cc_tbl_entry { + ib_net16_t shift_multiplier; +} PACK_SUFFIX ib_cc_tbl_entry_t; +#include +/* +* FIELDS +* +* shift_multiplier +* bits [1..0] - CCT Shift +* used when calculating the injection rate delay +* bits [15..2] - CCT Multiplier +* used when calculating the injection rate delay +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cc_tbl_t +* NAME +* ib_cc_tbl_t +* +* DESCRIPTION +* IBA defined CongestionControlTable attribute (A10.4.3.9) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cc_tbl { + ib_net16_t ccti_limit; + ib_net16_t resv; + ib_cc_tbl_entry_t entry_list[64]; +} PACK_SUFFIX ib_cc_tbl_t; +#include +/* +* FIELDS +* +* ccti_limit +* Maximum valid CCTI for this table. +* +* entry_list +* List of up to 64 CongestionControlTableEntries. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_time_stamp_t +* NAME +* ib_time_stamp_t +* +* DESCRIPTION +* IBA defined TimeStamp attribute (A10.4.3.10) +* +* SOURCE +*/ +#include +typedef struct _ib_time_stamp { + ib_net32_t value; +} PACK_SUFFIX ib_time_stamp_t; +#include +/* +* FIELDS +* +* value +* Free running clock that provides relative time info +* for a device. Time is kept in 1.024 usec units. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + /****d* IBA Base: Types/DM_SVC_NAME * NAME * DM_SVC_NAME -- 1.5.1.4 From sashak at voltaire.com Mon Jun 30 00:26:24 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 10:26:24 +0300 Subject: [ofa-general] Re: [PATCH] [TRIVIAL] opensm/osm_state_mgr.c: fixing soma data types and a corresponding log messages In-Reply-To: <48688112.8070006@dev.mellanox.co.il> References: <4861E87C.3000007@dev.mellanox.co.il> <20080625165534.GS22159@sashak.voltaire.com> <48688112.8070006@dev.mellanox.co.il> Message-ID: <20080630072624.GK12659@sashak.voltaire.com> On 09:45 Mon 30 Jun , Yevgeny Kliteynik wrote: > > /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c: In > function __osm_state_mgr_check_tbl_consistency: > /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c:922: > warning: format %u expects type unsigned int, but argument 5 has type > size_t > /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c:934: > warning: format %u expects type unsigned int, but argument 8 has type > size_t > /swgwork/kliteyn/git/management_master/opensm/opensm/osm_state_mgr.c:946: > warning: format %u expects type unsigned int, but argument 5 has type > size_t I see. Then '%zu' format should be used in printf instead of just '%u'. > Patch shortly. Thanks. Sasha From sashak at voltaire.com Mon Jun 30 00:29:54 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 10:29:54 +0300 Subject: [ofa-general] Re: [PATCH v2] [TRIVIAL] opensm/osm_state_mgr.c: fixing some data types In-Reply-To: <486881E1.1090603@dev.mellanox.co.il> References: <486881E1.1090603@dev.mellanox.co.il> Message-ID: <20080630072954.GL12659@sashak.voltaire.com> On 09:49 Mon 30 Jun , Yevgeny Kliteynik wrote: > Fixing some data types and solving compiler warnings > in the corresponding log messages. > > Signed-off-by: Yevgeny Kliteynik > --- > opensm/opensm/osm_state_mgr.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c > index 9a8409b..734df79 100644 > --- a/opensm/opensm/osm_state_mgr.c > +++ b/opensm/opensm/osm_state_mgr.c > @@ -862,7 +862,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) > osm_port_t *p_port; > osm_port_t *p_next_port; > cl_ptr_vector_t *p_port_lid_tbl; > - size_t max_lid, ref_size, curr_size, lid; > + uint16_t max_lid, ref_size, curr_size, lid; Those values are used for storing result of cl_ptr_vector_get_size() which has size_t type. I think the proper solution for this is to use '%zu' format in OSM_LOG(). Sasha From kliteyn at dev.mellanox.co.il Mon Jun 30 00:40:43 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 10:40:43 +0300 Subject: [ofa-general] [PATCH] ibutils: enhance ibis_gsi_mad_ctrl_bind() to have class version parameter Message-ID: <48688DFB.9070604@dev.mellanox.co.il> Hi Oren, Enhancing ibis_gsi_mad_ctrl_bind() to have class version parameter. Up until now, we needed only class version 1. Congestion Control introduced class version 2. Signed-off-by: Yevgeny Kliteynik --- ibis/src/ibbbm.c | 2 +- ibis/src/ibcr.c | 2 +- ibis/src/ibis_api.h | 4 ++++ ibis/src/ibis_gsi_mad_ctrl.c | 3 ++- ibis/src/ibpm.c | 2 +- ibis/src/ibsm.c | 2 +- ibis/src/ibvs.c | 4 ++-- 7 files changed, 12 insertions(+), 7 deletions(-) diff --git a/ibis/src/ibbbm.c b/ibis/src/ibbbm.c index c23e363..ee09bdb 100644 --- a/ibis/src/ibbbm.c +++ b/ibis/src/ibbbm.c @@ -167,7 +167,7 @@ ibbbm_bind( status = ibis_gsi_mad_ctrl_bind( &(IbisObj.mad_ctrl), - BBM_CLASS, + BBM_CLASS, 1, &p_ibbbm->h_bind ); diff --git a/ibis/src/ibcr.c b/ibis/src/ibcr.c index 13e4f28..50702c4 100644 --- a/ibis/src/ibcr.c +++ b/ibis/src/ibcr.c @@ -121,7 +121,7 @@ ibcr_bind( status = ibis_gsi_mad_ctrl_bind( &(IbisObj.mad_ctrl), - CR_CLASS, + CR_CLASS, 1, &p_ibcr->h_bind ); diff --git a/ibis/src/ibis_api.h b/ibis/src/ibis_api.h index ba71a26..cbf935c 100644 --- a/ibis/src/ibis_api.h +++ b/ibis/src/ibis_api.h @@ -331,6 +331,7 @@ ib_api_status_t ibis_gsi_mad_ctrl_bind( IN ibis_gsi_mad_ctrl_t* const p_ctrl, IN const uint8_t mad_class, + IN const uint8_t class_version, IN osm_bind_handle_t *p_h_bind); /* * PARAMETERS @@ -340,6 +341,9 @@ ibis_gsi_mad_ctrl_bind( * mad_class * [in] The specific mad class we register for. * +* class_version +* [in] The specific class version we register for. +* * * RETURN VALUES * None diff --git a/ibis/src/ibis_gsi_mad_ctrl.c b/ibis/src/ibis_gsi_mad_ctrl.c index cdbca0f..4aab145 100644 --- a/ibis/src/ibis_gsi_mad_ctrl.c +++ b/ibis/src/ibis_gsi_mad_ctrl.c @@ -635,6 +635,7 @@ ib_api_status_t ibis_gsi_mad_ctrl_bind( IN ibis_gsi_mad_ctrl_t* const p_ctrl, IN const uint8_t mad_class, + IN const uint8_t class_version, IN osm_bind_handle_t *p_h_bind) { osm_bind_info_t bind_info; @@ -651,7 +652,7 @@ ibis_gsi_mad_ctrl_bind( goto Exit; } - bind_info.class_version = 1; + bind_info.class_version = class_version; bind_info.is_responder = FALSE; bind_info.is_report_processor = FALSE; bind_info.is_trap_processor = FALSE; diff --git a/ibis/src/ibpm.c b/ibis/src/ibpm.c index c09159d..977a1af 100644 --- a/ibis/src/ibpm.c +++ b/ibis/src/ibpm.c @@ -117,7 +117,7 @@ ibpm_bind( status = ibis_gsi_mad_ctrl_bind( &(IbisObj.mad_ctrl), - IB_MCLASS_PERF, + IB_MCLASS_PERF, 1, &p_ibpm->h_bind ); diff --git a/ibis/src/ibsm.c b/ibis/src/ibsm.c index c607e75..6e48a80 100644 --- a/ibis/src/ibsm.c +++ b/ibis/src/ibsm.c @@ -119,7 +119,7 @@ ibsm_bind( be handled by the osm_vendor_bind if asked for LID route */ status = ibis_gsi_mad_ctrl_bind( &(IbisObj.mad_ctrl), - IB_MCLASS_SUBN_LID, + IB_MCLASS_SUBN_LID, 1, &p_ibsm->lid_route_bind ); diff --git a/ibis/src/ibvs.c b/ibis/src/ibvs.c index 9dca5ee..00787e5 100644 --- a/ibis/src/ibvs.c +++ b/ibis/src/ibvs.c @@ -121,7 +121,7 @@ ibvs_bind( status = ibis_gsi_mad_ctrl_bind( &(IbisObj.mad_ctrl), - VS_CLASS, + VS_CLASS, 1, &p_ibvs->h_bind ); @@ -132,7 +132,7 @@ ibvs_bind( status = ibis_gsi_mad_ctrl_bind( &(IbisObj.mad_ctrl), - IB_MCLASS_SUBN_LID, + IB_MCLASS_SUBN_LID, 1, &p_ibvs->h_smp_bind ); -- 1.5.1.4 From orenmeron at dev.mellanox.co.il Mon Jun 30 00:52:40 2008 From: orenmeron at dev.mellanox.co.il (Oren Meron) Date: Mon, 30 Jun 2008 10:52:40 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: <4867AE98.8020607@mellanox.co.il> References: <48679D21.20200@voltaire.com> <4867AE98.8020607@mellanox.co.il> Message-ID: <486890C8.2070409@dev.mellanox.co.il> Tziporet Koren wrote: > Or Gerlitz wrote: >> >>> Using the new connectx FW (2.5), I see performance drop to almost >>> zero with ipoib datagram mode. >> Oren, >> >> any news on this? >> >> > Sending to the correct Oren (Sela) > > Tziporet > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > hi, i do not manage to see a drop in datagram mode with the following params: netperf-2.4.1, 64k message, RHAS5.0, fw 2.5.0, OFED-1.3, hermon-eagle-DDR, Intel servers. connected produce ~1050 Mbytes/sec and datagram produce ~800 Mbytes/sec. these results comply with the ones i have in my database of both releases: 1.3+2.3.0 and 1.3.1+2.5.0. can you recheck all params and help us reproduce: test version, mtu, device etc. BUT, i do see a bw drop in CONNECTED mode to 30-40 Mbytes/sec with the following params: netperf-2.4.1, 64k message, RHAS5.1, fw 2.5.0, OFED-1.3.1, hermon-eagle-DDR, AMD servers. Eli is debugging it now. From kliteyn at dev.mellanox.co.il Mon Jun 30 01:31:41 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 11:31:41 +0300 Subject: [ofa-general] [PATCH 0/3] ibutils/ibis: adding support for Congestion Control Message-ID: <486899ED.1050400@dev.mellanox.co.il> Hi Oren, The following series of 3 patches implement support for Congestion Control (CC) MADs in ibis (IB Spec Annex 10). -- Yevgeny From kliteyn at dev.mellanox.co.il Mon Jun 30 01:33:16 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 11:33:16 +0300 Subject: [ofa-general] [PATCH 1/3] ibutils/ibis: adding support for Congestion Control Message-ID: <48689A4C.4040408@dev.mellanox.co.il> Hi Oren, This is the main patch of the CC support: Implementing all the attributes and adding get/set methods for these attributes. Signed-off-by: Yevgeny Kliteynik --- ibis/src/ibcc.c | 372 ++++++++++++++ ibis/src/ibcc.h | 322 ++++++++++++ ibis/src/ibcc.i | 1535 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 2229 insertions(+), 0 deletions(-) create mode 100644 ibis/src/ibcc.c create mode 100644 ibis/src/ibcc.h create mode 100644 ibis/src/ibcc.i diff --git a/ibis/src/ibcc.c b/ibis/src/ibcc.c new file mode 100644 index 0000000..9db247f --- /dev/null +++ b/ibis/src/ibcc.c @@ -0,0 +1,372 @@ +/* + * Copyright (c) 2008 Mellanox Technologies LTD. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id$ + */ + +/* + * Abstract: + * Implementation of ibcc_t. + * This object represents the Congestion Control object. + * This object is part of the IBIS family of objects. + * + * Environment: + * Linux User Mode + * + * $Revision: 1.0 $ + */ + +#include +#include +#include +#include +#include +#include "ibcc.h" +#include +#include +#include +#include + + +/********************************************************************** + **********************************************************************/ + +ibcc_t * +ibcc_construct() +{ + ibcc_t* p_ibcc; + OSM_LOG_ENTER(&(IbisObj.log)); + + p_ibcc = malloc(sizeof(ibcc_t)); + if (p_ibcc == NULL) { + goto Exit; + } + memset(p_ibcc, 0, sizeof(ibcc_t)); + + Exit : + OSM_LOG_EXIT(&(IbisObj.log)); + return p_ibcc; +} + +/********************************************************************** + **********************************************************************/ + +void +ibcc_destroy( + IN ibcc_t* const p_ibcc) +{ + OSM_LOG_ENTER(&(IbisObj.log)); + p_ibcc->state = IBCC_STATE_INIT; + OSM_LOG_EXIT( &(IbisObj.log) ); +} + +/********************************************************************** + **********************************************************************/ + +ib_api_status_t +ibcc_init( + IN ibcc_t* const p_ibcc) +{ + ib_api_status_t status = IB_SUCCESS; + OSM_LOG_ENTER(&(IbisObj.log)); + p_ibcc->state = IBCC_STATE_INIT; + OSM_LOG_EXIT(&(IbisObj.log)); + return status; +} + +/********************************************************************** + **********************************************************************/ + +ib_api_status_t +ibcc_bind( + IN ibcc_t* const p_ibcc) +{ + ib_api_status_t status = IB_SUCCESS; + + OSM_LOG_ENTER(&(IbisObj.log)); + + /* Bind CongestionControl Management Class */ + if ((status = ibis_gsi_mad_ctrl_bind( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, 2, + &p_ibcc->lid_route_bind)) != IB_SUCCESS) + goto Exit; + + /* Set callback for ClassPortInfo attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_CLASS_PORT_INFO), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for Notice attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_NOTICE), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for CongestionInfo attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_CONG_INFO), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for CongestionKeyInfo attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_CONG_KEY_INFO), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for CongestionLog attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_CONG_LOG), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for SwitchCongestionSetting attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_SW_CONG_SETTING), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for SwitchPortCongestionSetting attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_SW_PORT_CONG_SETTING), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for CACongestionSetting attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_CA_CONG_SETTING), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for CongestionControlTable attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_CC_TBL), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + /* Set callback for TimeStamp attribute */ + if ((status = ibis_gsi_mad_ctrl_set_class_attr_cb( + &(IbisObj.mad_ctrl), + IB_MCLASS_CC, + CL_NTOH16(IB_MAD_ATTR_TIME_STAMP), + ibis_gsi_sync_mad_batch_callback, + (void *)p_ibcc)) != IB_SUCCESS) + goto Exit; + + Exit : + OSM_LOG_EXIT( &(IbisObj.log) ); + return status; +} + +/********************************************************************** + **********************************************************************/ + +static void +__ibcc_prep_cc_mad( + IN ibcc_t *p_ibcc, + IN uint16_t lid, + IN uint8_t method, + IN uint16_t attribute_id, + IN uint32_t attribute_mod, + IN uint64_t cc_key, + IN uint8_t *cc_log_data, + IN size_t cc_log_data_size, + IN uint8_t *cc_mgt_data, + IN size_t cc_mgt_data_size, + OUT osm_madw_t **pp_madw) +{ + osm_mad_addr_t mad_addr; + osm_madw_t *p_madw; + + OSM_LOG_ENTER(&(IbisObj.log)); + + mad_addr.dest_lid = cl_hton16(lid); + mad_addr.path_bits = 0; + mad_addr.static_rate = 0; + + mad_addr.addr_type.gsi.remote_qp=cl_hton32(1); + mad_addr.addr_type.gsi.remote_qkey = cl_hton32(0x80010000); + mad_addr.addr_type.gsi.pkey_ix = 0; + mad_addr.addr_type.gsi.service_level = 0; + mad_addr.addr_type.gsi.global_route = FALSE; + + p_madw = osm_mad_pool_get(&(IbisObj.mad_pool), + p_ibcc->lid_route_bind, + MAD_BLOCK_SIZE, + &mad_addr); + *pp_madw = p_madw; + + p_madw->resp_expected = TRUE; + + memset((char*)p_madw->p_mad, 0, MAD_BLOCK_SIZE); + + ((ib_mad_t *)p_madw->p_mad)->base_ver = 1; + ((ib_mad_t *)p_madw->p_mad)->mgmt_class = IB_MCLASS_CC; + ((ib_mad_t *)p_madw->p_mad)->class_ver = 2; + ((ib_mad_t *)p_madw->p_mad)->method = method; + ((ib_mad_t *)p_madw->p_mad)->trans_id = ibis_get_tid(); + ((ib_mad_t *)p_madw->p_mad)->attr_id = cl_hton16(attribute_id); + ((ib_mad_t *)p_madw->p_mad)->attr_mod = cl_hton32(attribute_mod); + + ((ib_cc_mad_t *)p_madw->p_mad)->cc_key = cl_hton64(cc_key); + + memset(((ib_cc_mad_t *)p_madw->p_mad)->log_data, + 0, IB_CC_LOG_DATA_SIZE); + memset(((ib_cc_mad_t *)p_madw->p_mad)->mgt_data, + 0, IB_CC_MGT_DATA_SIZE); + + if (method == IB_MAD_METHOD_GET) { + /* In SubnGet() method we leave log and + management data clean - they should + be filled by the response MAD only.*/ + } + else { + if (cc_log_data_size > 0) { + CL_ASSERT(cc_log_data_size <= IB_CC_LOG_DATA_SIZE); + CL_ASSERT(cc_log_data); + memcpy(((ib_cc_mad_t *)p_madw->p_mad)->log_data, + cc_log_data, cc_log_data_size); + } + + if (cc_mgt_data_size > 0) { + CL_ASSERT(cc_mgt_data_size <= IB_CC_MGT_DATA_SIZE); + CL_ASSERT(cc_mgt_data); + memcpy(((ib_cc_mad_t *)p_madw->p_mad)->mgt_data, + cc_mgt_data, cc_mgt_data_size); + } + } + + OSM_LOG_EXIT(&(IbisObj.log)); +} /* __ibcc_prep_cc_mad() */ + +/********************************************************************** + **********************************************************************/ + +ib_api_status_t +ibcc_send_mad_by_lid ( + ibcc_t *p_ibcc, + uint64_t cc_key, + uint8_t *cc_log_data, + size_t cc_log_data_size, + uint8_t *cc_mgt_data, + size_t cc_mgt_data_size, + uint16_t lid, + uint16_t attribute_id, + uint32_t attribute_mod, + uint16_t method) +{ + osm_madw_t *p_madw; + ib_cc_mad_t response_mad; + ib_api_status_t status; + + OSM_LOG_ENTER(&(IbisObj.log)); + + osm_log(&(IbisObj.log), OSM_LOG_DEBUG, + "ibcc_send_mad_by_lid: " + "Sending to lid:0x%04X method:0x%02X " + "attr:0x%04X mod:0x%08x\n", + lid, method, attribute_id, attribute_mod); + + memset(&response_mad, 0, sizeof(ib_cc_mad_t)); + + __ibcc_prep_cc_mad(p_ibcc, + lid, + method, + attribute_id, + attribute_mod, + cc_key, + cc_log_data, + cc_log_data_size, + cc_mgt_data, + cc_mgt_data_size, + &p_madw); + + /* send and wait */ + status = ibis_gsi_send_sync_mad_batch( + &(IbisObj.mad_ctrl), + p_ibcc->lid_route_bind, + 1, + &p_madw, + sizeof(ib_cc_mad_t), + (uint8_t*)&response_mad); + + if (!response_mad.header.method) + status = IB_TIMEOUT; + + if (status == IB_SUCCESS) { + + if (cc_log_data) + memcpy(cc_log_data, + response_mad.log_data, + cc_log_data_size); + + if (cc_mgt_data) + memcpy(cc_mgt_data, + response_mad.mgt_data, + cc_mgt_data_size); + + if (cl_ntoh16(response_mad.header.status) & 0x7fff) + status = cl_ntoh16(response_mad.header.status); + } + + OSM_LOG_EXIT(&(IbisObj.log)); + return (status); +} /* ibcc_send_mad_by_lid() */ + diff --git a/ibis/src/ibcc.h b/ibis/src/ibcc.h new file mode 100644 index 0000000..149675c --- /dev/null +++ b/ibis/src/ibcc.h @@ -0,0 +1,322 @@ +/* + * Copyright (c) 2008 Mellanox Technologies LTD. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id$ + */ + +/* + * Abstract: + * Definition of ibcc_t. + * This object represents the Congestion Control Packets Interface + * This object is part of the IBIS family of objects. + * + * Environment: + * Linux User Mode + * + * $Revision: 1.0 $ + */ + +#ifndef _IBCC_H_ +#define _IBCC_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include "ibis_api.h" +#include "ibis.h" + +typedef enum _ibcc_state +{ + IBCC_STATE_INIT, + IBCC_STATE_READY, + IBCC_STATE_BUSY, +} ibcc_state_t; + +#define MAD_PAYLOAD_SIZE 256 + +#define IBCC_DEAFULT_KEY 0 + +/****s* IBIS: ibcc/ibcc_t +* NAME ibcc_t +* +* +* DESCRIPTION +* ibcc structure +* +* SYNOPSIS +*/ +typedef struct _ibcc +{ + ibcc_state_t state; + osm_bind_handle_t lid_route_bind; +} ibcc_t; +/* +* FIELDS +* +* state +* The ibcc state: INIT, READ or BUSY +* +* lid_route_bind +* The handle to bind with the lower level for lid routed packets +* +* SEE ALSO +* +*********/ + +/****f* IBIS: ibcc/ibcc_construct +* NAME +* ibcc_construct +* +* DESCRIPTION +* Allocation of ibcc_t struct +* +* SYNOPSIS +*/ +ibcc_t* +ibcc_construct(void); +/* +* PARAMETERS +* +* +* RETURN VALUE +* Return a pointer to an ibcc struct. Null if fails to do so. +* +* NOTES +* First step of the creation of ibcc_t +* +* SEE ALSO +* ibcc_destroy ibcc_init +*********/ + +/****s* IBIS: ibcc/ibcc_destroy +* NAME +* ibcc_destroy +* +* DESCRIPTION +* release of ibcc_t struct +* +* SYNOPSIS +*/ +void +ibcc_destroy( + IN ibcc_t* const p_ibcc); +/* +* PARAMETERS +* p_ibcc +* A pointer to the ibcc_t struct that is about to be released +* +* RETURN VALUE +* +* NOTES +* Final step of the releasing of ibcc_t +* +* SEE ALSO +* ibcc_construct +*********/ + +/****f* IBIS: ibcc/ibcc_init +* NAME +* ibcc_init +* +* DESCRIPTION +* Initialization of an ibcc_t struct +* +* SYNOPSIS +*/ +ib_api_status_t +ibcc_init( + IN ibcc_t* const p_ibcc); +/* +* PARAMETERS +* p_ibcc +* A pointer to the ibcc_t struct that is about to be initialized +* +* RETURN VALUE +* The status of the function. +* +* NOTES +* +* SEE ALSO +* ibcc_construct +* *********/ + + +/****f* IBIS: ibcc/ibcc_bind +* NAME +* ibcc_bind +* +* DESCRIPTION +* Binding the ibcc object to a lower level. +* +* SYNOPSIS +*/ +ib_api_status_t +ibcc_bind( + IN ibcc_t* const p_ibcc); +/* +* PARAMETERS +* p_ibcc +* A pointer to the ibcc_t struct that is about to be binded +* +* RETURN VALUE +* The status of the function. +* +* NOTES +* +* SEE ALSO +* ibcc_construct +*********/ + +/****f* IBIS: ibcc/ibcc_send_mad_by_lid +* NAME +* ibcc_send_mad_by_lid +* +* DESCRIPTION +* Send a CC mad to the given LID. +* +* SYNOPSIS +* ibcc_send_mad_by_lid(p_ibcc, p_mad, lid, attr, mod, meth) +* Note that all values are in host order. +*/ +ib_api_status_t +ibcc_send_mad_by_lid ( + ibcc_t *p_ibcc, + uint64_t cc_key, + uint8_t *cc_log_data, + size_t cc_log_data_size, + uint8_t *cc_mgt_data, + size_t cc_mgt_data_size, + uint16_t lid, + uint16_t attribute_id, + uint32_t attribute_mod, + uint16_t method); +/* +* PARAMETERS +* p_ibcc +* A pointer to the ibcc_t struct +* +* cc_key +* Congestion Control key +* +* cc_log_data +* [in/out] A pointer to CC log data. +* Will be overwritten in case of response. +* +* cc_log_data_size +* [in] The size of the log data block +* +* cc_mgt_data +* [in/out] A pointer to CC management data. +* Will be overwritten in case of response. +* +* cc_mgt_data_size +* [in] The size of the mgt data block +* +* lid +* The Destination lid of the MAD +* +* attribute_id +* The Attribute ID +* +* attribute_mod +* Attribute modifier value +* +* method +* The MAD method: Set/Get/Trap... +* +* RETURN VALUE +* The status of the function or response status. +* +* NOTES +* +* SEE ALSO +* +*********/ + +/****s* IBA Base: Types/ibcc_ca_cong_log_t +* NAME +* ibcc_ca_cong_log_t +* +* DESCRIPTION +* IBA defined CongestionLog attribute (A10.4.3.5) +* has a union that includes Congestion Log for +* switches and CAs. +* Instead of dealing with the union in SWIG, the +* following struct is defined to deal only with +* CA congestion log. +* For more details, please see ib_cong_log_t +* definition in ib_types.h +* +* SYNOPSIS +*/ +typedef struct _ib_ca_cong_log { + uint8_t log_type; + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net16_t event_map; + ib_net16_t resv; + ib_net32_t time_stamp; + ib_cong_log_event_ca_t log_event[13]; +} ibcc_ca_cong_log_t; +/*********/ + +/****s* IBA Base: Types/ibcc_sw_cong_log_t +* NAME +* ibcc_sw_cong_log_t +* +* DESCRIPTION +* IBA defined CongestionLog attribute (A10.4.3.5) +* has a union that includes Congestion Log for +* switches and CAs. +* Instead of dealing with the union in SWIG, the +* following struct is defined to deal only with +* switch congestion log. +* For more details, please see ib_cong_log_t +* definition in ib_types.h +* +* SYNOPSIS +*/ +typedef struct _ib_sw_cong_log { + uint8_t log_type; + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net32_t time_stamp; + uint8_t port_map[32]; + ib_cong_log_event_sw_t entry_list[15]; +} ibcc_sw_cong_log_t; +/*********/ + +#endif /* _IBCC_H_ */ diff --git a/ibis/src/ibcc.i b/ibis/src/ibcc.i new file mode 100644 index 0000000..49d9a89 --- /dev/null +++ b/ibis/src/ibcc.i @@ -0,0 +1,1535 @@ +/* + * Copyright (c) 2008 Mellanox Technologies LTD. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id$ + */ + +%text %{ + + Congestion Control: + The CC interface supports sending Congestion Control Packets to + the fabric devices. + + A set of object types is defined. The user can allocate a new attribute + object. Then the object can be set using configure or the direct + manipulation methods. + + Extra methods are provided for each object: + setByLid + getByLid +%} + +/* + * NOTE : The simplest way to implement that interface is to + * code the Set section as embedded tcl code that will create a new + * object of the attribute type and use the given set of flags + * as configuration command. This way we let swig do the coding + * and parsing of each field. + */ + +%{ +#include "stdio.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include "ibis_api.h" + + /* the global pointer to this CC mads manager */ + static ibcc_t *gp_ibcc; + + /* we probably want to use our own naming for classes */ + typedef ib_class_port_info_t ccClassPortInfo; + typedef ib_mad_notice_attr_t ccNotice; + typedef ib_cong_info_t ccCongestionInfo; + typedef ib_cong_key_info_t ccCongestionKeyInfo; + typedef ibcc_ca_cong_log_t ccCACongestionLog; + typedef ibcc_sw_cong_log_t ccSWCongestionLog; + typedef ib_sw_cong_setting_t ccSWCongestionSetting; + typedef ib_sw_port_cong_setting_t ccSWPortCongestionSetting; + typedef ib_ca_cong_setting_t ccCACongestionSetting; + typedef ib_cc_tbl_t ccTable; + typedef ib_time_stamp_t ccTimeStamp; + + /* these are the global objects to be used + for set/get (one for each attribute) */ + ib_class_port_info_t ibcc_class_port_info_obj; + ib_mad_notice_attr_t ibcc_notice_obj; + ib_cong_info_t ibcc_cong_info_obj; + ib_cong_key_info_t ibcc_cong_key_info_obj; + ibcc_ca_cong_log_t ibcc_ca_cong_log_obj; + ibcc_sw_cong_log_t ibcc_sw_cong_log_obj; + ib_sw_cong_setting_t ibcc_sw_cong_setting_obj; + ib_sw_port_cong_setting_t ibcc_sw_port_cong_setting_obj; + ib_ca_cong_setting_t ibcc_ca_cong_setting_obj; + ib_cc_tbl_t ibcc_table_obj; + ib_time_stamp_t ibcc_time_stamp_obj; + +%} + +// +// STANDARD IB TYPE MAPS: +// + +%typemap(tcl8,in) ib_gid_t*(ib_gid_t temp) +{ + char buf[38]; + char *p_prefix, *p_guid; + char *str_token; + + strcpy(buf, Tcl_GetStringFromObj($source,NULL)); + p_prefix = strtok_r(buf,":", &str_token); + p_guid = strtok_r(NULL, " ", &str_token); + temp.unicast.prefix = cl_hton64(strtoull(p_prefix, NULL, 16)); + errno = 0; + if (errno) { + printf("Wrong format for gid prefix:%s\n", p_prefix); + return TCL_ERROR; + } + + temp.unicast.interface_id = cl_hton64(strtoull(p_guid, NULL, 16)); + if (errno) { + printf("Wrong format for gid guid:%s\n", p_guid); + return TCL_ERROR; + } + + $target = &temp; +} + +// +// INTERFACE DEFINITION (~copy of h file) +// + +%section "IBCC Constants" +/* These constants are provided by IBCC: */ + +%section "IBCC Objects",pre +/* This section describes the various object types exposed by IBCC. */ +%text %{ + +%} + +/************************************************** + *** ClassPortInfo (A10.4.3.1) *** + **************************************************/ + +typedef struct _ibcc_class_port_info { + uint8_t base_ver; + uint8_t class_ver; + ib_net16_t cap_mask; + ib_net32_t cap_mask2_resp_time; + ib_gid_t redir_gid; + ib_net32_t redir_tc_sl_fl; + ib_net16_t redir_lid; + ib_net16_t redir_pkey; + ib_net32_t redir_qp; + ib_net32_t redir_qkey; + ib_gid_t trap_gid; + ib_net32_t trap_tc_sl_fl; + ib_net16_t trap_lid; + ib_net16_t trap_pkey; + ib_net32_t trap_hop_qp; + ib_net32_t trap_qkey; +} ccClassPortInfo; + +%addmethods ccClassPortInfo { + + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccClassPortInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CLASS_PORT_INFO), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } + + int setByLid(uint16_t lid) { + return ibcc_send_mad_by_lid( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccClassPortInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CLASS_PORT_INFO), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } +} + +/************************************************** + *** Notice - Trap 0 (A10.4.3.2) *** + **************************************************/ + +typedef struct _ibcc_notice +{ + uint8_t generic_type; + + union _notice_g_or_v { + struct { + uint8_t prod_type_msb; + ib_net16_t prod_type_lsb; + ib_net16_t trap_num; + } generic; + } g_or_v; + + ib_net16_t issuer_lid; + ib_net16_t toggle_count; + + union _data_details + { + struct _cc_raw_data { + uint8_array_t details[54]; + } raw_data; + + struct { + ib_net16_t source_lid; // Source LID from offending packet LRH + uint8_t method; // Method, from common MAD header + uint8_t resv0; + ib_net16_t attr_id; // Attribute ID, from common MAD header + ib_net16_t resv1; + ib_net32_t attr_mod; // Attribute Modif, from common MAD header + ib_net32_t qp; // 8b pad, 24b dest QP from BTH + ib_net64_t cc_key; // CC key of the offending packet + ib_gid_t source_gid; // GID from GRH of the offending packet + uint8_array_t padding[14];// Padding - ignored on read + } ntc_0; + + } data_details; + + ib_gid_t issuer_gid; +} ccNotice; + +%addmethods ccNotice { + int trapByLid(uint16_t lid) { + return ibcc_send_mad_by_lid( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccNotice), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_NOTICE), + 0, // attribute modifier + IB_MAD_METHOD_TRAP); + } +} + +/************************************************** + *** CongestionInfo (A10.4.3.3) *** + **************************************************/ + +typedef struct _ib_cong_info { + uint8_t cong_info; + uint8_t resv; + uint8_t ctrl_table_cap; +} ccCongestionInfo; + +%addmethods ccCongestionInfo { + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCongestionInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_INFO), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +} + +/************************************************** + *** CongestionKeyInfo (A10.4.3.4) *** + **************************************************/ + +typedef struct _ib_cong_key_info { + ib_net64_t cc_key; + uint8_t protect_bit; + uint8_t resv; + ib_net16_t lease_period; + ib_net16_t violations; +} ccCongestionKeyInfo; + +%addmethods ccCongestionKeyInfo { + + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCongestionKeyInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_KEY_INFO), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } + + int setByLid(uint16_t lid) { + return ibcc_send_mad_by_lid( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCongestionKeyInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_KEY_INFO), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } +} + +/************************************************** + *** CongestionLog (A10.4.3.5) *** + **************************************************/ + +/* + * IBA defined CongestionLog attribute (A10.4.3.5) + * has the following definition: + * + * typedef struct _ib_cong_log { + * uint8_t log_type; + * union _log_details + * { + * struct { + * uint8_t cong_flags; + * ib_net16_t event_counter; + * ib_net32_t time_stamp; + * uint8_array_t port_map[32]; + * ib_cong_log_event_sw_t entry_list[15]; + * } log_sw; + * + * struct { + * uint8_t cong_flags; + * ib_net16_t event_counter; + * ib_net16_t event_map; + * ib_net16_t resv; + * ib_net32_t time_stamp; + * ib_cong_log_event_ca_t log_event[13]; + * } log_ca; + * + * } log_details; + * } ccCongestionLog; + * + * This definition has a union that includes Congestion Log for + * switches and CAs. + * Instead of dealing with the union in SWIG, we define two separate logs: + * ccSWCongestionLog (for dealing only with switches) + * ccCACongestionLog (for dealing only with CAs) + */ + +%typemap(in) ib_cong_log_event_sw_t[ANY] (ib_cong_log_event_sw_t entrys[$dim0]) { + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, $source, &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj($source,NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure log_event: + * 1. Configure the whole list by providing list of value groups: + * ccSWCongestionLogMad configure -log_event {{1 2 3 4} {5 6 7 8}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccSWCongestionLogMad configure -log_event {{4 1 2 3 4} {8 5 6 7 8}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 4) + option = 1; + else if (numElements == 5) + option = 2; + else { + printf("Error: wrong number of elements for SW Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].slid = cl_hton16(value); break; + case 1: entrys[i].dlid = cl_hton16(value); break; + case 2: entrys[i].resv0_sl = value; break; + case 3: entrys[i++].time_stamp = cl_hton32(value); break; + default: break; + } + } + } + + for (; i < $dim0; i++) { + entrys[i].slid = 0; + entrys[i].dlid = 0; + entrys[i].resv0_sl = 0; + entrys[i].time_stamp = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 5) { + printf("Error: wrong number of elements for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 5; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].slid = cl_hton16(value); break; + case 2: entrys[entry_index].dlid = cl_hton16(value); break; + case 3: entrys[entry_index].resv0_sl = value; break; + case 4: entrys[entry_index].time_stamp = cl_hton32(value); break; + default: break; + } + } + + } + } + + $target = entrys; +} + +%typemap(memberin) ib_cong_log_event_sw_t[ANY] { + int i; + for (i=0; i <$dim0 ; i++) { + $target[i] = *($source+i); + } +} + +%typemap(tcl8,out) ib_cong_log_event_sw_t[ANY] { + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <$dim0 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].slid) + sprintf(buff, " -slid 0x%04x", cl_ntoh16($source[i].slid)); + else + sprintf(buff, " -slid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].dlid) + sprintf(buff, " -dlid 0x%04x", cl_ntoh16($source[i].dlid)); + else + sprintf(buff, " -dlid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].resv0_sl) + sprintf(buff, " -resv0_sl 0x%02x", $source[i].resv0_sl); + else + sprintf(buff, " -resv0_sl 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].time_stamp) + sprintf(buff, " -time_stamp 0x%08x", cl_ntoh32($source[i].time_stamp)); + else + sprintf(buff, " -time_stamp 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + +typedef struct _ib_cong_log_event_sw { + ib_net16_t slid; + ib_net16_t dlid; + uint8_t resv0_sl; + uint8_t resv1; + ib_net16_t resv2; + ib_net32_t time_stamp; +} ib_cong_log_event_sw_t; + +/**************************************************/ + +%typemap(in) ib_cong_log_event_ca_t[ANY] (ib_cong_log_event_ca_t entrys[$dim0]) { + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, $source, &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj($source,NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure log_event: + * 1. Configure the whole list by providing list of value groups: + * ccCACongestionLogMad configure -log_event {{1 2 3 4} {5 6 7 8}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccCACongestionLogMad configure -log_event {{4 1 2 3 4} {8 5 6 7 8}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 4) + option = 1; + else if (numElements == 5) + option = 2; + else { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].resv0_local_qp = cl_hton32(value); break; + case 1: entrys[i].remote_qp_sl_service_type = cl_hton32(value); break; + case 2: entrys[i].remote_lid = cl_hton16(value); break; + case 3: entrys[i++].time_stamp = cl_hton32(value); break; + default: break; + } + } + } + + for (; i < $dim0; i++) { + entrys[i].resv0_local_qp = 0; + entrys[i].remote_qp_sl_service_type = 0; + entrys[i].remote_lid = 0; + entrys[i].time_stamp = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 5) { + printf("Error: wrong number of elements for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 5; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].resv0_local_qp = cl_hton32(value); break; + case 2: entrys[entry_index].remote_qp_sl_service_type = cl_hton32(value); break; + case 3: entrys[entry_index].remote_lid = cl_hton16(value); break; + case 4: entrys[entry_index].time_stamp = cl_hton32(value); break; + default: break; + } + } + + } + } + + $target = entrys; +} + +%typemap(memberin) ib_cong_log_event_ca_t[ANY] { + int i; + for (i=0; i <$dim0 ; i++) { + $target[i] = *($source+i); + } +} + +%typemap(tcl8,out) ib_cong_log_event_ca_t[ANY] { + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <$dim0 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].resv0_local_qp) + sprintf(buff, " -resv0_local_qp 0x%08x", cl_ntoh32($source[i].resv0_local_qp)); + else + sprintf(buff, " -resv0_local_qp 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].remote_qp_sl_service_type) + sprintf(buff, " -remote_qp_sl_service_type 0x%08x", cl_ntoh32($source[i].remote_qp_sl_service_type)); + else + sprintf(buff, " -remote_qp_sl_service_type 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].remote_lid) + sprintf(buff, " -remote_lid 0x%04x", cl_ntoh16($source[i].remote_lid)); + else + sprintf(buff, " -remote_lid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].time_stamp) + sprintf(buff, " -time_stamp 0x%08x", cl_ntoh32($source[i].time_stamp)); + else + sprintf(buff, " -time_stamp 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + +typedef struct _ib_cong_log_event_ca { + ib_net32_t resv0_local_qp; + ib_net32_t remote_qp_sl_service_type; + ib_net16_t remote_lid; + ib_net16_t resv1; + ib_net32_t time_stamp; +} ib_cong_log_event_ca_t; + +/**************************************************/ + +typedef struct _ib_sw_cong_log { + uint8_t log_type; + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net32_t time_stamp; + uint8_array_t port_map[32]; + ib_cong_log_event_sw_t entry_list[15]; +} ccSWCongestionLog; + +%addmethods ccSWCongestionLog { + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + (uint8_t *)self, // log data + sizeof(ccSWCongestionLog), // log data size + NULL, // mgt data + 0, // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_LOG), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +} + +typedef struct _ib_ca_cong_log { + uint8_t log_type; + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net16_t event_map; + ib_net16_t resv; + ib_net32_t time_stamp; + ib_cong_log_event_ca_t log_event[13]; +} ccCACongestionLog; + +%addmethods ccCACongestionLog { + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + (uint8_t *)self, // log data + sizeof(ccCACongestionLog), // log data size + NULL, // mgt data + 0, // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_LOG), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +} + + +/************************************************** + *** SwitchCongestionSetting (A10.4.3.6) *** + **************************************************/ + +typedef struct _ib_sw_cong_setting { + ib_net32_t control_map; + uint8_array_t victim_mask[32]; + uint8_array_t credit_mask[32]; + uint8_t threshold; + uint8_t packet_size; + uint8_t cs_threshold; + uint8_t resv0; + ib_net16_t cs_return_delay; + ib_net16_t marking_rate; +} ccSWCongestionSetting; + + +%addmethods ccSWCongestionSetting { + + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } + + int setByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } + +} + +/************************************************** + *** SwitchPortCongestionSetting (A10.4.3.7) *** + **************************************************/ + +%typemap(in) ib_sw_port_cong_setting_element_t[ANY] (ib_sw_port_cong_setting_element_t entrys[$dim0]) { + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, $source, &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj($source,NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure entry_list: + * 1. Configure the whole list by providing list of value groups: + * ccSWPortCongestionSettingMad configure -block {{1 2 3} {4 5 6}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccSWPortCongestionSettingMad configure -block {{4 1 2 3} {8 4 5 6}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 3) + option = 1; + else if (numElements == 4) + option = 2; + else { + printf("Error: wrong number of elements for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 3) { + printf("Error: wrong number of elements for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 3; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].valid_ctrl_type_res_threshold = value; break; + case 1: entrys[i].packet_size = value; break; + case 2: entrys[i++].cong_param = cl_hton16(value); break; + default: break; + } + } + } + + for (; i < $dim0; i++) { + entrys[i].valid_ctrl_type_res_threshold = 0; + entrys[i].packet_size = 0; + entrys[i].cong_param = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].valid_ctrl_type_res_threshold = value; break; + case 2: entrys[entry_index].packet_size = value; break; + case 3: entrys[entry_index].cong_param = cl_hton16(value); break; + default: break; + } + } + + } + } + + $target = entrys; +} + +%typemap(memberin) ib_sw_port_cong_setting_element_t[ANY] { + int i; + for (i=0; i <$dim0 ; i++) { + $target[i] = *($source+i); + } +} + +%typemap(tcl8,out) ib_sw_port_cong_setting_element_t[ANY] { + int i; + char buff[99]; + + sprintf(buff, "-block "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <$dim0 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].valid_ctrl_type_res_threshold) + sprintf(buff, " -valid_ctrl_type_res_threshold 0x%02x", $source[i].valid_ctrl_type_res_threshold); + else + sprintf(buff, " -valid_ctrl_type_res_threshold 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].packet_size) + sprintf(buff, " -packet_size 0x%02x", $source[i].packet_size); + else + sprintf(buff, " -packet_size 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].cong_param) + sprintf(buff, " -cong_param 0x%04x", cl_ntoh16($source[i].cong_param)); + else + sprintf(buff, " -cong_param 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + +typedef struct _ib_sw_port_cong_setting_element { + uint8_t valid_ctrl_type_res_threshold; + uint8_t packet_size; + ib_net16_t cong_param; +} ib_sw_port_cong_setting_element_t; + +typedef struct _ib_sw_port_cong_setting { + ib_sw_port_cong_setting_element_t block[32]; +} ccSWPortCongestionSetting; + +%addmethods ccSWPortCongestionSetting { + + int getByLid(uint16_t lid, uint8_t block) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWPortCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_PORT_CONG_SETTING), + block, // attribute modifier + IB_MAD_METHOD_GET); + } + + int setByLid(uint16_t lid, uint8_t block) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWPortCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_PORT_CONG_SETTING), + block, // attribute modifier + IB_MAD_METHOD_SET); + } + +} + +/************************************************** + *** CACongestionSetting (A10.4.3.8) *** + **************************************************/ + +%typemap(in) ib_ca_cong_entry_t[ANY] (ib_ca_cong_entry_t entrys[$dim0]) { + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, $source, &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for CA Congestion Setting: %s\n", + Tcl_GetStringFromObj($source,NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure entry_list: + * 1. Configure the whole list by providing list of value groups: + * ccCACongestionSetting configure -entry_list {{1 2 3 4} {5 6 7 8}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccCACongestionSetting configure -entry_list {{4 1 2 3 4} {8 5 6 7 8}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 4) + option = 1; + else if (numElements == 5) + option = 2; + else { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].ccti_timer = cl_hton16(value); break; + case 1: entrys[i].ccti_increase = value; break; + case 2: entrys[i].trigger_threshold = value; break; + case 3: entrys[i++].ccti_min = value; break; + default: break; + } + } + } + + for (; i < $dim0; i++) { + entrys[i].ccti_timer = 0; + entrys[i].ccti_increase = 0; + entrys[i].trigger_threshold = 0; + entrys[i].ccti_min = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 5) { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 5; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].ccti_timer = cl_hton16(value); break; + case 2: entrys[entry_index].ccti_increase = value; break; + case 3: entrys[entry_index].trigger_threshold = value; break; + case 4: entrys[entry_index].ccti_min = value; break; + default: break; + } + } + + } + } + + $target = entrys; +} + +%typemap(memberin) ib_ca_cong_entry_t[ANY] { + int i; + for (i=0; i <$dim0 ; i++) { + $target[i] = *($source+i); + } +} + +%typemap(tcl8,out) ib_ca_cong_entry_t[ANY] { + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <$dim0 ; i++) { + sprintf(buff, "{SL%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].ccti_timer) + sprintf(buff, " -ccti_timer 0x%04x", cl_ntoh16($source[i].ccti_timer)); + else + sprintf(buff, " -ccti_timer 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].ccti_increase) + sprintf(buff, " -ccti_increase 0x%02x", $source[i].ccti_increase); + else + sprintf(buff, " -ccti_increase 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].trigger_threshold) + sprintf(buff, " -trigger_threshold 0x%02x", $source[i].trigger_threshold); + else + sprintf(buff, " -trigger_threshold 0"); + Tcl_AppendResult(interp, buff, NULL); + + if ($source[i].ccti_min) + sprintf(buff, " -ccti_min 0x%02x", $source[i].ccti_min); + else + sprintf(buff, " -ccti_min 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + +typedef struct _ib_ca_cong_entry { + ib_net16_t ccti_timer; + uint8_t ccti_increase; + uint8_t trigger_threshold; + uint8_t ccti_min; + uint8_t resv0; + ib_net16_t resv1; +} ib_ca_cong_entry_t; + +typedef struct _ib_ca_cong_setting { + uint8_t port_control; + uint8_t resv0; + ib_net16_t control_map; + ib_ca_cong_entry_t entry_list[16]; +} ccCACongestionSetting; + + +%addmethods ccCACongestionSetting { + + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCACongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CA_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } + + int setByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCACongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CA_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } + +} + +/************************************************** + *** CongestionControlTable (A10.4.3.9) *** + **************************************************/ + +%typemap(in) ib_cc_tbl_entry_t[ANY] (ib_cc_tbl_entry_t entrys[$dim0]) { + long int value; + long int entry_index = 0; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, $source, &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for CC Table: %s\n", + Tcl_GetStringFromObj($source,NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure entry_list: + * 1. Configure the whole list by providing list of values: + * ccTableMad configure -entry_list {1 2 3 4 5 6} + * 2. Configure specific items from the list by providing index and value: + * ccTableMad configure -entry_list {{1 3} {2 4} {7 16}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 1) + option = 1; + else if (numElements == 2) + option = 2; + else { + printf("Error: wrong number of elements for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 1) { + printf("Error: wrong number of elements for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], 0, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + + entrys[i++].shift_multiplier = cl_hton16(value); + } + + for (; i < $dim0; i++) + entrys[i].shift_multiplier = 0; + } + else { + /* + * second option - index and value + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 2) { + printf("Error: wrong number of elements for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], 0, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + entry_index = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], 1, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + + entrys[entry_index].shift_multiplier = cl_hton16(value); + } + } + + $target = entrys; +} + +%typemap(memberin) ib_cc_tbl_entry_t[ANY] { + int i; + for (i=0; i <$dim0 ; i++) { + $target[i] = *($source+i); + } +} + +%typemap(tcl8,out) ib_cc_tbl_entry_t[ANY] { + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <$dim0 ; i++) { + if ($source[i].shift_multiplier) + sprintf(buff, "{#%u: 0x%04x} ", i, cl_ntoh16($source[i].shift_multiplier)); + else + sprintf(buff, "{#%u: 0} ", i); + Tcl_AppendResult(interp, buff, NULL); + } +} + +typedef struct _ibcc_tbl_entry { + ib_net16_t shift_multiplier; +} ib_cc_tbl_entry_t; + +typedef struct _ib_cc_tbl { + ib_net16_t ccti_limit; + ib_net16_t resv; + ib_cc_tbl_entry_t entry_list[64]; +} ccTable; + +%addmethods ccTable { + + int getByLid(uint16_t lid, uint8_t sn) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccTable), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CC_TBL), + sn, // attribute modifier + IB_MAD_METHOD_GET); + } + + int setByLid(uint16_t lid, uint8_t sn) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccTable), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CC_TBL), + sn, // attribute modifier + IB_MAD_METHOD_SET); + } + +} + +/************************************************** + *** TimeStamp (A10.4.3.10) *** + **************************************************/ + +typedef struct _ib_time_stamp { + ib_net32_t value; +} ccTimeStamp; + +%addmethods ccTimeStamp { + + int getByLid(uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccTimeStamp), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_TIME_STAMP), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } + +} -- 1.5.1.4 From kliteyn at dev.mellanox.co.il Mon Jun 30 01:33:35 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 11:33:35 +0300 Subject: [ofa-general] [PATCH 2/3] ibutils/ibis: adding support for Congestion Control Message-ID: <48689A5F.8050700@dev.mellanox.co.il> Hi Oren, This patch integrates new CC interface into ibis and exposes all the CC attributes and methods. Signed-off-by: Yevgeny Kliteynik --- ibis/src/Makefile.am | 8 ++- ibis/src/ibis.i | 100 +++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 103 insertions(+), 5 deletions(-) diff --git a/ibis/src/Makefile.am b/ibis/src/Makefile.am index 7166906..a45b1ec 100644 --- a/ibis/src/Makefile.am +++ b/ibis/src/Makefile.am @@ -57,11 +57,11 @@ LIB_FILE_TRIPLET=1.0.0 lib_LTLIBRARIES = libibiscom.la libibis.la libibiscom_la_SOURCES = ibbbm.c ibcr.c ibis.c ibis_gsi_mad_ctrl.c \ - ibpm.c ibsac.c ibsm.c ibvs.c + ibpm.c ibsac.c ibsm.c ibvs.c ibcc.c # client library to be used by IBIS TCL package: libibis_la_SOURCES = ibis_wrap.c ibbbm.c ibcr.c ibis.c ibis_gsi_mad_ctrl.c \ - ibpm.c ibsac.c ibsm.c ibvs.c + ibpm.c ibsac.c ibsm.c ibvs.c ibcc.c libibis_la_LDFLAGS = -version-info $(LIB_VER_TRIPLET) -no-undefined \ $(OSM_LDFLAGS) $(TCL_LIBS) @@ -87,7 +87,8 @@ SWIG_IFC_FILES= $(srcdir)/ibbbm.i \ $(srcdir)/ibpm.i \ $(srcdir)/ibsac.i \ $(srcdir)/ibsm.i \ - $(srcdir)/ibvs.i + $(srcdir)/ibvs.i \ + $(srcdir)/ibcc.i ibis.lo: $(srcdir)/git_version.h @@ -150,6 +151,7 @@ EXTRA_DIST = swig_extended_obj.c fixSwigWrapper pkgIndex.tcl \ ibsm.h \ ibvs_base.h \ ibvs.h \ + ibcc.h \ git_version.h # we do not want the temporary and the archive libs installed: diff --git a/ibis/src/ibis.i b/ibis/src/ibis.i index a3c9b45..12e2115 100644 --- a/ibis/src/ibis.i +++ b/ibis/src/ibis.i @@ -64,6 +64,7 @@ BEGIN_C_DECLS #include "ibbbm.h" #include "ibsac.h" #include "ibsm.h" +#include "ibcc.h" END_C_DECLS @@ -149,7 +150,7 @@ ibisp_is_debug(void) %include ibcr.i // -// IBCR Interfaces and C Code +// IBPM Interfaces and C Code // %include ibpm.i @@ -173,6 +174,11 @@ ibisp_is_debug(void) // %include ibsm.i +// +// IBCC Interfaces and C Code +// +%include ibcc.i + %{ /* globals */ ibis_t IbisObj; @@ -231,11 +237,19 @@ ibisp_is_debug(void) status = ibsm_init(gp_ibsm); if( status != IB_SUCCESS ) { - printf("-E- Fail to init ibbbm_init.\n"); + printf("-E- Fail to init ibsm_init.\n"); ibsm_destroy( gp_ibsm ); exit(1); } + status = ibcc_init(gp_ibcc); + if( status != IB_SUCCESS ) + { + printf("-E- Fail to init ibcc_init.\n"); + ibcc_destroy( gp_ibcc ); + exit(1); + } + return 0; } @@ -249,6 +263,7 @@ ibisp_is_debug(void) ibvs_destroy(p_ibvs_global); ibbbm_destroy(p_ibbbm_global); ibsm_destroy(gp_ibsm); + ibcc_destroy(gp_ibcc); ibis_destroy(); usleep(100); @@ -321,6 +336,14 @@ ibisp_is_debug(void) exit(1); } + status = ibcc_bind(gp_ibcc); + if( status != IB_SUCCESS ) + { + printf("-E- Fail to ibcc_bind.\n"); + ibcc_destroy( gp_ibcc ); + exit(1); + } + if (ibsac_bind(&IbisObj)) { printf("-E- Fail to ibsac_bind.\n"); @@ -592,6 +615,28 @@ extern char * ibisSourceVersion; memset(&ibsm_pkey_table_obj, 0, sizeof(ib_pkey_table_t)); memset(&ibsm_sm_info_obj, 0, sizeof(ib_sm_info_t)); + /* ------------------ IBCC ---------------------- */ + + gp_ibcc = ibcc_construct(); + + if (gp_ibcc == NULL) { + printf("-E- Error from ibcc_construct.\n"); + exit(1); + } + + /* Initialize global records */ + memset(&ibcc_class_port_info_obj, 0, sizeof(ib_class_port_info_t)); + memset(&ibcc_notice_obj, 0, sizeof(ib_mad_notice_attr_t)); + memset(&ibcc_cong_info_obj, 0, sizeof(ib_cong_info_t)); + memset(&ibcc_cong_key_info_obj, 0, sizeof(ib_cong_key_info_t)); + memset(&ibcc_ca_cong_log_obj, 0, sizeof(ibcc_ca_cong_log_t)); + memset(&ibcc_sw_cong_log_obj, 0, sizeof(ibcc_sw_cong_log_t)); + memset(&ibcc_sw_cong_setting_obj, 0, sizeof(ib_sw_cong_setting_t)); + memset(&ibcc_sw_port_cong_setting_obj, 0, sizeof(ib_sw_port_cong_setting_t)); + memset(&ibcc_ca_cong_setting_obj, 0, sizeof(ib_ca_cong_setting_t)); + memset(&ibcc_table_obj, 0, sizeof(ib_cc_tbl_t)); + memset(&ibcc_time_stamp_obj, 0, sizeof(ib_time_stamp_t)); + /* ------------------ IBSAC ---------------------- */ /* Initialize global records */ @@ -697,6 +742,9 @@ extern char * ibisSourceVersion; (ClientData)ibis_opt_p, 0); /* add commands for accessing the global query records */ + + /* ------------------ IBSM ---------------------- */ + Tcl_CreateObjCommand(interp,"smNodeInfoMad", TclsmNodeInfoMethodCmd, (ClientData)&ibsm_node_info_obj, 0); @@ -745,6 +793,54 @@ extern char * ibisSourceVersion; TclsmNoticeMethodCmd, (ClientData)&ibsm_notice_obj, 0); + /* ------------------ IBCC ---------------------- */ + + Tcl_CreateObjCommand(interp,"ccClassPortInfoMad", + TclccClassPortInfoMethodCmd, + (ClientData)&ibcc_class_port_info_obj, 0); + + Tcl_CreateObjCommand(interp,"ccNoticeMad", + TclccNoticeMethodCmd, + (ClientData)&ibcc_notice_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCongestionInfoMad", + TclccCongestionInfoMethodCmd, + (ClientData)&ibcc_cong_info_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCongestionKeyInfoMad", + TclccCongestionKeyInfoMethodCmd, + (ClientData)&ibcc_cong_key_info_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCACongestionLogMad", + TclccCACongestionLogMethodCmd, + (ClientData)&ibcc_ca_cong_log_obj, 0); + + Tcl_CreateObjCommand(interp,"ccSWCongestionLogMad", + TclccSWCongestionLogMethodCmd, + (ClientData)&ibcc_sw_cong_log_obj, 0); + + Tcl_CreateObjCommand(interp,"ccSWCongestionSettingMad", + TclccSWCongestionSettingMethodCmd, + (ClientData)&ibcc_sw_cong_setting_obj, 0); + + Tcl_CreateObjCommand(interp,"ccSWPortCongestionSettingMad", + TclccSWPortCongestionSettingMethodCmd, + (ClientData)&ibcc_sw_port_cong_setting_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCACongestionSettingMad", + TclccCACongestionSettingMethodCmd, + (ClientData)&ibcc_ca_cong_setting_obj, 0); + + Tcl_CreateObjCommand(interp,"ccTableMad", + TclccTableMethodCmd, + (ClientData)&ibcc_table_obj, 0); + + Tcl_CreateObjCommand(interp,"ccTimeStampMad", + TclccTimeStampMethodCmd, + (ClientData)&ibcc_time_stamp_obj, 0); + + /* ------------------ IBSAC --------------------- */ + Tcl_CreateObjCommand(interp,"sacNodeQuery", TclsacNodeRecMethodCmd, (ClientData)&ibsac_node_rec, 0); -- 1.5.1.4 From kliteyn at dev.mellanox.co.il Mon Jun 30 01:35:06 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 11:35:06 +0300 Subject: [ofa-general] [PATCH 3/3] ibutils/ibis: adding support for Congestion Control Message-ID: <48689ABA.8000600@dev.mellanox.co.il> Hi Oren, Last patch, only generated files, nothing fancy. Signed-off-by: Yevgeny Kliteynik --- ibdm/doc/ibdmsh.1 | 2 +- ibdm/doc/ibdmtr.1 | 2 +- ibis/src/ibissh_wrap.cpp |18197 +++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 18198 insertions(+), 3 deletions(-) diff --git a/ibdm/doc/ibdmsh.1 b/ibdm/doc/ibdmsh.1 index 0461ab2..0278a54 100644 --- a/ibdm/doc/ibdmsh.1 +++ b/ibdm/doc/ibdmsh.1 @@ -129,7 +129,7 @@ .\" ======================================================================== .\" .IX Title "IBDMSH 1" -.TH IBDMSH 1 "2008-06-16" "IBDM 1.0" "IB DATA MODEL PACKAGE" +.TH IBDMSH 1 "2008-06-17" "IBDM 1.0" "IB DATA MODEL PACKAGE" .SH "NAME" \&\fBibdmsh\fR IB DATA MODEL \- Extentended TCL shell .SH "DESCRIPTION" diff --git a/ibdm/doc/ibdmtr.1 b/ibdm/doc/ibdmtr.1 index 9442c4a..79dbed4 100644 --- a/ibdm/doc/ibdmtr.1 +++ b/ibdm/doc/ibdmtr.1 @@ -129,7 +129,7 @@ .\" ======================================================================== .\" .IX Title "IBDMTR 1" -.TH IBDMTR 1 "2008-06-16" "IBDM 1.0" "IB DATA MODEL PACKAGE" +.TH IBDMTR 1 "2008-06-17" "IBDM 1.0" "IB DATA MODEL PACKAGE" .SH "NAME" \&\fBibdmtr\fR \- Fabric Trace Route .SH "SYNOPSYS" diff --git a/ibis/src/ibissh_wrap.cpp b/ibis/src/ibissh_wrap.cpp index 3ea92cf..2d06be0 100644 --- a/ibis/src/ibissh_wrap.cpp +++ b/ibis/src/ibissh_wrap.cpp @@ -329,6 +329,7 @@ BEGIN_C_DECLS #include "ibbbm.h" #include "ibsac.h" #include "ibsm.h" +#include "ibcc.h" END_C_DECLS @@ -2890,6 +2891,104 @@ typedef struct { } smNotice_data_details_ntc_259; +#include "stdio.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include "ibis_api.h" + + /* the global pointer to this CC mads manager */ + static ibcc_t *gp_ibcc; + + /* we probably want to use our own naming for classes */ + typedef ib_class_port_info_t ccClassPortInfo; + typedef ib_mad_notice_attr_t ccNotice; + typedef ib_cong_info_t ccCongestionInfo; + typedef ib_cong_key_info_t ccCongestionKeyInfo; + typedef ibcc_ca_cong_log_t ccCACongestionLog; + typedef ibcc_sw_cong_log_t ccSWCongestionLog; + typedef ib_sw_cong_setting_t ccSWCongestionSetting; + typedef ib_sw_port_cong_setting_t ccSWPortCongestionSetting; + typedef ib_ca_cong_setting_t ccCACongestionSetting; + typedef ib_cc_tbl_t ccTable; + typedef ib_time_stamp_t ccTimeStamp; + + /* these are the global objects to be used + for set/get (one for each attribute) */ + ib_class_port_info_t ibcc_class_port_info_obj; + ib_mad_notice_attr_t ibcc_notice_obj; + ib_cong_info_t ibcc_cong_info_obj; + ib_cong_key_info_t ibcc_cong_key_info_obj; + ibcc_ca_cong_log_t ibcc_ca_cong_log_obj; + ibcc_sw_cong_log_t ibcc_sw_cong_log_obj; + ib_sw_cong_setting_t ibcc_sw_cong_setting_obj; + ib_sw_port_cong_setting_t ibcc_sw_port_cong_setting_obj; + ib_ca_cong_setting_t ibcc_ca_cong_setting_obj; + ib_cc_tbl_t ibcc_table_obj; + ib_time_stamp_t ibcc_time_stamp_obj; + + +typedef union { + struct { + uint8_t prod_type_msb; + ib_net16_t prod_type_lsb; + ib_net16_t trap_num; + } generic; + } ccNotice_g_or_v; + + +typedef union { + struct _cc_raw_data { + uint8_array_t details[54]; + } raw_data; + + struct { + ib_net16_t source_lid; // Source LID from offending packet LRH + uint8_t method; // Method, from common MAD header + uint8_t resv0; + ib_net16_t attr_id; // Attribute ID, from common MAD header + ib_net16_t resv1; + ib_net32_t attr_mod; // Attribute Modif, from common MAD header + ib_net32_t qp; // 8b pad, 24b dest QP from BTH + ib_net64_t cc_key; // CC key of the offending packet + ib_gid_t source_gid; // GID from GRH of the offending packet + uint8_array_t padding[14];// Padding - ignored on read + } ntc_0; + + } ccNotice_data_details; + + +typedef struct { + uint8_t prod_type_msb; + ib_net16_t prod_type_lsb; + ib_net16_t trap_num; + } ccNotice_g_or_v_generic; + + +typedef struct { + uint8_array_t details[54]; + } ccNotice_data_details_raw_data; + + +typedef struct { + ib_net16_t source_lid; // Source LID from offending packet LRH + uint8_t method; // Method, from common MAD header + uint8_t resv0; + ib_net16_t attr_id; // Attribute ID, from common MAD header + ib_net16_t resv1; + ib_net32_t attr_mod; // Attribute Modif, from common MAD header + ib_net32_t qp; // 8b pad, 24b dest QP from BTH + ib_net64_t cc_key; // CC key of the offending packet + ib_gid_t source_gid; // GID from GRH of the offending packet + uint8_array_t padding[14];// Padding - ignored on read + } ccNotice_data_details_ntc_0; + + /* globals */ ibis_t IbisObj; static ibis_opt_t *ibis_opt_p; @@ -2947,11 +3046,19 @@ typedef struct { status = ibsm_init(gp_ibsm); if( status != IB_SUCCESS ) { - printf("-E- Fail to init ibbbm_init.\n"); + printf("-E- Fail to init ibsm_init.\n"); ibsm_destroy( gp_ibsm ); exit(1); } + status = ibcc_init(gp_ibcc); + if( status != IB_SUCCESS ) + { + printf("-E- Fail to init ibcc_init.\n"); + ibcc_destroy( gp_ibcc ); + exit(1); + } + return 0; } @@ -2965,6 +3072,7 @@ typedef struct { ibvs_destroy(p_ibvs_global); ibbbm_destroy(p_ibbbm_global); ibsm_destroy(gp_ibsm); + ibcc_destroy(gp_ibcc); ibis_destroy(); usleep(100); @@ -3037,6 +3145,14 @@ typedef struct { exit(1); } + status = ibcc_bind(gp_ibcc); + if( status != IB_SUCCESS ) + { + printf("-E- Fail to ibcc_bind.\n"); + ibcc_destroy( gp_ibcc ); + exit(1); + } + if (ibsac_bind(&IbisObj)) { printf("-E- Fail to ibsac_bind.\n"); @@ -56847,6 +56963,17685 @@ static int TclsmNotice_data_details_ntc_259Cmd(ClientData clientData, Tcl_Interp } +#define _ibcc_class_port_info_base_ver_set(_swigobj,_swigval) (_swigobj->base_ver = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_base_ver_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccClassPortInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_base_ver_set { ccClassPortInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_base_ver_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ibcc_class_port_info_base_ver_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_base_ver_get(_swigobj) (&_swigobj->base_ver) +static int _wrap_ccClassPortInfo_base_ver_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_base_ver_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_base_ver_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ibcc_class_port_info_base_ver_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_class_ver_set(_swigobj,_swigval) (_swigobj->class_ver = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_class_ver_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccClassPortInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_class_ver_set { ccClassPortInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_class_ver_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ibcc_class_port_info_class_ver_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_class_ver_get(_swigobj) (&_swigobj->class_ver) +static int _wrap_ccClassPortInfo_class_ver_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_class_ver_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_class_ver_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ibcc_class_port_info_class_ver_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_cap_mask_set(_swigobj,_swigval) (_swigobj->cap_mask = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_cap_mask_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_cap_mask_set { ccClassPortInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_cap_mask_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_cap_mask_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_cap_mask_get(_swigobj) (&_swigobj->cap_mask) +static int _wrap_ccClassPortInfo_cap_mask_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_cap_mask_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_cap_mask_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_cap_mask_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_cap_mask2_resp_time_set(_swigobj,_swigval) (_swigobj->cap_mask2_resp_time = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_cap_mask2_resp_time_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_cap_mask2_resp_time_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_cap_mask2_resp_time_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_cap_mask2_resp_time_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_cap_mask2_resp_time_get(_swigobj) (&_swigobj->cap_mask2_resp_time) +static int _wrap_ccClassPortInfo_cap_mask2_resp_time_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_cap_mask2_resp_time_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_cap_mask2_resp_time_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_cap_mask2_resp_time_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_gid_set(_swigobj,_swigval) (_swigobj->redir_gid = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_redir_gid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccClassPortInfo * _arg0; + ib_gid_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_gid_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_gid_set { ccClassPortInfo * } { ib_gid_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_gid_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char buf[38]; + char *p_prefix, *p_guid; + char *str_token; + + strcpy(buf, Tcl_GetStringFromObj(objv[2],NULL)); + p_prefix = strtok_r(buf,":", &str_token); + p_guid = strtok_r(NULL, " ", &str_token); + temp.unicast.prefix = cl_hton64(strtoull(p_prefix, NULL, 16)); + errno = 0; + if (errno) { + printf("Wrong format for gid prefix:%s\n", p_prefix); + return TCL_ERROR; + } + + temp.unicast.interface_id = cl_hton64(strtoull(p_guid, NULL, 16)); + if (errno) { + printf("Wrong format for gid guid:%s\n", p_guid); + return TCL_ERROR; + } + + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)_ibcc_class_port_info_redir_gid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_gid_get(_swigobj) (&_swigobj->redir_gid) +static int _wrap_ccClassPortInfo_redir_gid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_gid_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_gid_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)_ibcc_class_port_info_redir_gid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_tc_sl_fl_set(_swigobj,_swigval) (_swigobj->redir_tc_sl_fl = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_redir_tc_sl_fl_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_tc_sl_fl_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_tc_sl_fl_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_redir_tc_sl_fl_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_tc_sl_fl_get(_swigobj) (&_swigobj->redir_tc_sl_fl) +static int _wrap_ccClassPortInfo_redir_tc_sl_fl_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_tc_sl_fl_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_tc_sl_fl_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_redir_tc_sl_fl_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_lid_set(_swigobj,_swigval) (_swigobj->redir_lid = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_redir_lid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_lid_set { ccClassPortInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_lid_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_redir_lid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_lid_get(_swigobj) (&_swigobj->redir_lid) +static int _wrap_ccClassPortInfo_redir_lid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_lid_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_lid_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_redir_lid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_pkey_set(_swigobj,_swigval) (_swigobj->redir_pkey = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_redir_pkey_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_pkey_set { ccClassPortInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_pkey_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_redir_pkey_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_pkey_get(_swigobj) (&_swigobj->redir_pkey) +static int _wrap_ccClassPortInfo_redir_pkey_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_pkey_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_pkey_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_redir_pkey_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_qp_set(_swigobj,_swigval) (_swigobj->redir_qp = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_redir_qp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_qp_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_qp_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_redir_qp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_qp_get(_swigobj) (&_swigobj->redir_qp) +static int _wrap_ccClassPortInfo_redir_qp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_qp_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_qp_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_redir_qp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_qkey_set(_swigobj,_swigval) (_swigobj->redir_qkey = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_redir_qkey_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_qkey_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_qkey_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_redir_qkey_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_redir_qkey_get(_swigobj) (&_swigobj->redir_qkey) +static int _wrap_ccClassPortInfo_redir_qkey_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_redir_qkey_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_redir_qkey_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_redir_qkey_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_gid_set(_swigobj,_swigval) (_swigobj->trap_gid = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_trap_gid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccClassPortInfo * _arg0; + ib_gid_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_gid_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_gid_set { ccClassPortInfo * } { ib_gid_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_gid_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char buf[38]; + char *p_prefix, *p_guid; + char *str_token; + + strcpy(buf, Tcl_GetStringFromObj(objv[2],NULL)); + p_prefix = strtok_r(buf,":", &str_token); + p_guid = strtok_r(NULL, " ", &str_token); + temp.unicast.prefix = cl_hton64(strtoull(p_prefix, NULL, 16)); + errno = 0; + if (errno) { + printf("Wrong format for gid prefix:%s\n", p_prefix); + return TCL_ERROR; + } + + temp.unicast.interface_id = cl_hton64(strtoull(p_guid, NULL, 16)); + if (errno) { + printf("Wrong format for gid guid:%s\n", p_guid); + return TCL_ERROR; + } + + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)_ibcc_class_port_info_trap_gid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_gid_get(_swigobj) (&_swigobj->trap_gid) +static int _wrap_ccClassPortInfo_trap_gid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_gid_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_gid_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)_ibcc_class_port_info_trap_gid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_tc_sl_fl_set(_swigobj,_swigval) (_swigobj->trap_tc_sl_fl = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_trap_tc_sl_fl_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_tc_sl_fl_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_tc_sl_fl_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_trap_tc_sl_fl_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_tc_sl_fl_get(_swigobj) (&_swigobj->trap_tc_sl_fl) +static int _wrap_ccClassPortInfo_trap_tc_sl_fl_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_tc_sl_fl_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_tc_sl_fl_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_trap_tc_sl_fl_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_lid_set(_swigobj,_swigval) (_swigobj->trap_lid = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_trap_lid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_lid_set { ccClassPortInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_lid_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_trap_lid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_lid_get(_swigobj) (&_swigobj->trap_lid) +static int _wrap_ccClassPortInfo_trap_lid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_lid_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_lid_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_trap_lid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_pkey_set(_swigobj,_swigval) (_swigobj->trap_pkey = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_trap_pkey_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_pkey_set { ccClassPortInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_pkey_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_trap_pkey_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_pkey_get(_swigobj) (&_swigobj->trap_pkey) +static int _wrap_ccClassPortInfo_trap_pkey_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_pkey_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_pkey_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_class_port_info_trap_pkey_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_hop_qp_set(_swigobj,_swigval) (_swigobj->trap_hop_qp = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_trap_hop_qp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_hop_qp_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_hop_qp_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_trap_hop_qp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_hop_qp_get(_swigobj) (&_swigobj->trap_hop_qp) +static int _wrap_ccClassPortInfo_trap_hop_qp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_hop_qp_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_hop_qp_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_trap_hop_qp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_qkey_set(_swigobj,_swigval) (_swigobj->trap_qkey = *(_swigval),_swigval) +static int _wrap_ccClassPortInfo_trap_qkey_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_qkey_set { ccClassPortInfo * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_qkey_set. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_trap_qkey_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_class_port_info_trap_qkey_get(_swigobj) (&_swigobj->trap_qkey) +static int _wrap_ccClassPortInfo_trap_qkey_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccClassPortInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_trap_qkey_get { ccClassPortInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_trap_qkey_get. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ibcc_class_port_info_trap_qkey_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static int ccClassPortInfo_getByLid(ccClassPortInfo *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccClassPortInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CLASS_PORT_INFO), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccClassPortInfo_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccClassPortInfo * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_getByLid { ccClassPortInfo * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_getByLid. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccClassPortInfo_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +static int ccClassPortInfo_setByLid(ccClassPortInfo *self,uint16_t lid) { + return ibcc_send_mad_by_lid( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccClassPortInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CLASS_PORT_INFO), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } +static int _wrap_ccClassPortInfo_setByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccClassPortInfo * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccClassPortInfo_setByLid { ccClassPortInfo * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccClassPortInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccClassPortInfo_setByLid. Expected _ccClassPortInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccClassPortInfo_setByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccClassPortInfoMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccClassPortInfo methods : { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccClassPortInfo_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccClassPortInfo_getByLid; + } else if (strcmp(_str,"setByLid") == 0) { + cmd = _wrap_ccClassPortInfo_setByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-base_ver") == 0) { + cmd = _wrap_ccClassPortInfo_base_ver_set; + } else if (strcmp(_str,"-class_ver") == 0) { + cmd = _wrap_ccClassPortInfo_class_ver_set; + } else if (strcmp(_str,"-cap_mask") == 0) { + cmd = _wrap_ccClassPortInfo_cap_mask_set; + } else if (strcmp(_str,"-cap_mask2_resp_time") == 0) { + cmd = _wrap_ccClassPortInfo_cap_mask2_resp_time_set; + } else if (strcmp(_str,"-redir_gid") == 0) { + cmd = _wrap_ccClassPortInfo_redir_gid_set; + } else if (strcmp(_str,"-redir_tc_sl_fl") == 0) { + cmd = _wrap_ccClassPortInfo_redir_tc_sl_fl_set; + } else if (strcmp(_str,"-redir_lid") == 0) { + cmd = _wrap_ccClassPortInfo_redir_lid_set; + } else if (strcmp(_str,"-redir_pkey") == 0) { + cmd = _wrap_ccClassPortInfo_redir_pkey_set; + } else if (strcmp(_str,"-redir_qp") == 0) { + cmd = _wrap_ccClassPortInfo_redir_qp_set; + } else if (strcmp(_str,"-redir_qkey") == 0) { + cmd = _wrap_ccClassPortInfo_redir_qkey_set; + } else if (strcmp(_str,"-trap_gid") == 0) { + cmd = _wrap_ccClassPortInfo_trap_gid_set; + } else if (strcmp(_str,"-trap_tc_sl_fl") == 0) { + cmd = _wrap_ccClassPortInfo_trap_tc_sl_fl_set; + } else if (strcmp(_str,"-trap_lid") == 0) { + cmd = _wrap_ccClassPortInfo_trap_lid_set; + } else if (strcmp(_str,"-trap_pkey") == 0) { + cmd = _wrap_ccClassPortInfo_trap_pkey_set; + } else if (strcmp(_str,"-trap_hop_qp") == 0) { + cmd = _wrap_ccClassPortInfo_trap_hop_qp_set; + } else if (strcmp(_str,"-trap_qkey") == 0) { + cmd = _wrap_ccClassPortInfo_trap_qkey_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -base_ver -class_ver -cap_mask -cap_mask2_resp_time -redir_gid -redir_tc_sl_fl -redir_lid -redir_pkey -redir_qp -redir_qkey -trap_gid -trap_tc_sl_fl -trap_lid -trap_pkey -trap_hop_qp -trap_qkey }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -base_ver -class_ver -cap_mask -cap_mask2_resp_time -redir_gid -redir_tc_sl_fl -redir_lid -redir_pkey -redir_qp -redir_qkey -trap_gid -trap_tc_sl_fl -trap_lid -trap_pkey -trap_hop_qp -trap_qkey }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-base_ver") == 0) { + cmd = _wrap_ccClassPortInfo_base_ver_get; + } else if (strcmp(_str,"-class_ver") == 0) { + cmd = _wrap_ccClassPortInfo_class_ver_get; + } else if (strcmp(_str,"-cap_mask") == 0) { + cmd = _wrap_ccClassPortInfo_cap_mask_get; + } else if (strcmp(_str,"-cap_mask2_resp_time") == 0) { + cmd = _wrap_ccClassPortInfo_cap_mask2_resp_time_get; + } else if (strcmp(_str,"-redir_gid") == 0) { + cmd = _wrap_ccClassPortInfo_redir_gid_get; + } else if (strcmp(_str,"-redir_tc_sl_fl") == 0) { + cmd = _wrap_ccClassPortInfo_redir_tc_sl_fl_get; + } else if (strcmp(_str,"-redir_lid") == 0) { + cmd = _wrap_ccClassPortInfo_redir_lid_get; + } else if (strcmp(_str,"-redir_pkey") == 0) { + cmd = _wrap_ccClassPortInfo_redir_pkey_get; + } else if (strcmp(_str,"-redir_qp") == 0) { + cmd = _wrap_ccClassPortInfo_redir_qp_get; + } else if (strcmp(_str,"-redir_qkey") == 0) { + cmd = _wrap_ccClassPortInfo_redir_qkey_get; + } else if (strcmp(_str,"-trap_gid") == 0) { + cmd = _wrap_ccClassPortInfo_trap_gid_get; + } else if (strcmp(_str,"-trap_tc_sl_fl") == 0) { + cmd = _wrap_ccClassPortInfo_trap_tc_sl_fl_get; + } else if (strcmp(_str,"-trap_lid") == 0) { + cmd = _wrap_ccClassPortInfo_trap_lid_get; + } else if (strcmp(_str,"-trap_pkey") == 0) { + cmd = _wrap_ccClassPortInfo_trap_pkey_get; + } else if (strcmp(_str,"-trap_hop_qp") == 0) { + cmd = _wrap_ccClassPortInfo_trap_hop_qp_get; + } else if (strcmp(_str,"-trap_qkey") == 0) { + cmd = _wrap_ccClassPortInfo_trap_qkey_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccClassPortInfo_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -base_ver -class_ver -cap_mask -cap_mask2_resp_time -redir_gid -redir_tc_sl_fl -redir_lid -redir_pkey -redir_qp -redir_qkey -trap_gid -trap_tc_sl_fl -trap_lid -trap_pkey -trap_hop_qp -trap_qkey }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -base_ver -class_ver -cap_mask -cap_mask2_resp_time -redir_gid -redir_tc_sl_fl -redir_lid -redir_pkey -redir_qp -redir_qkey -trap_gid -trap_tc_sl_fl -trap_lid -trap_pkey -trap_hop_qp -trap_qkey }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccClassPortInfo_base_ver_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-base_ver ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_class_ver_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-class_ver ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_cap_mask_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cap_mask ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_cap_mask2_resp_time_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cap_mask2_resp_time ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_redir_gid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-redir_gid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_redir_tc_sl_fl_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-redir_tc_sl_fl ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_redir_lid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-redir_lid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_redir_pkey_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-redir_pkey ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_redir_qp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-redir_qp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_redir_qkey_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-redir_qkey ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_trap_gid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_gid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_trap_tc_sl_fl_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_tc_sl_fl ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_trap_lid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_lid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_trap_pkey_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_pkey ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_trap_hop_qp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_hop_qp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccClassPortInfo_trap_qkey_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_qkey ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccClassPortInfoCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccClassPortInfo * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccClassPortInfo_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccClassPortInfo_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccClassPortInfo object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccClassPortInfoMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ibcc_notice_generic_type_set(_swigobj,_swigval) (_swigobj->generic_type = *(_swigval),_swigval) +static int _wrap_ccNotice_generic_type_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_generic_type_set { ccNotice * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_generic_type_set. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ibcc_notice_generic_type_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_generic_type_get(_swigobj) (&_swigobj->generic_type) +static int _wrap_ccNotice_generic_type_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_generic_type_get { ccNotice * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_generic_type_get. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ibcc_notice_generic_type_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_issuer_lid_set(_swigobj,_swigval) (_swigobj->issuer_lid = *(_swigval),_swigval) +static int _wrap_ccNotice_issuer_lid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_issuer_lid_set { ccNotice * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_issuer_lid_set. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_notice_issuer_lid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_issuer_lid_get(_swigobj) (&_swigobj->issuer_lid) +static int _wrap_ccNotice_issuer_lid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_issuer_lid_get { ccNotice * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_issuer_lid_get. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_notice_issuer_lid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_toggle_count_set(_swigobj,_swigval) (_swigobj->toggle_count = *(_swigval),_swigval) +static int _wrap_ccNotice_toggle_count_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_toggle_count_set { ccNotice * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_toggle_count_set. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_notice_toggle_count_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_toggle_count_get(_swigobj) (&_swigobj->toggle_count) +static int _wrap_ccNotice_toggle_count_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_toggle_count_get { ccNotice * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_toggle_count_get. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_notice_toggle_count_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_issuer_gid_set(_swigobj,_swigval) (_swigobj->issuer_gid = *(_swigval),_swigval) +static int _wrap_ccNotice_issuer_gid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccNotice * _arg0; + ib_gid_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_gid_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_issuer_gid_set { ccNotice * } { ib_gid_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_issuer_gid_set. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char buf[38]; + char *p_prefix, *p_guid; + char *str_token; + + strcpy(buf, Tcl_GetStringFromObj(objv[2],NULL)); + p_prefix = strtok_r(buf,":", &str_token); + p_guid = strtok_r(NULL, " ", &str_token); + temp.unicast.prefix = cl_hton64(strtoull(p_prefix, NULL, 16)); + errno = 0; + if (errno) { + printf("Wrong format for gid prefix:%s\n", p_prefix); + return TCL_ERROR; + } + + temp.unicast.interface_id = cl_hton64(strtoull(p_guid, NULL, 16)); + if (errno) { + printf("Wrong format for gid guid:%s\n", p_guid); + return TCL_ERROR; + } + + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)_ibcc_notice_issuer_gid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_issuer_gid_get(_swigobj) (&_swigobj->issuer_gid) +static int _wrap_ccNotice_issuer_gid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccNotice * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_issuer_gid_get { ccNotice * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_issuer_gid_get. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)_ibcc_notice_issuer_gid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_notice_g_or_v_get(_swigobj) (&_swigobj->g_or_v) +static int _wrap_ccNotice_g_or_v_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ccNotice_g_or_v * _result; + ccNotice * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_get { ccNotice * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_get. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ccNotice_g_or_v *)_ibcc_notice_g_or_v_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + SWIG_SetPointerObj(tcl_result,(void *) _result,"_ccNotice_g_or_v_p"); + return TCL_OK; +} +#define _ibcc_notice_data_details_get(_swigobj) (&_swigobj->data_details) +static int _wrap_ccNotice_data_details_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ccNotice_data_details * _result; + ccNotice * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_get { ccNotice * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_get. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ccNotice_data_details *)_ibcc_notice_data_details_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + SWIG_SetPointerObj(tcl_result,(void *) _result,"_ccNotice_data_details_p"); + return TCL_OK; +} +static int ccNotice_trapByLid(ccNotice *self,uint16_t lid) { + return ibcc_send_mad_by_lid( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccNotice), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_NOTICE), + 0, // attribute modifier + IB_MAD_METHOD_TRAP); + } +static int _wrap_ccNotice_trapByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccNotice * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_trapByLid { ccNotice * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_trapByLid. Expected _ccNotice_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccNotice_trapByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccNoticeMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccNotice methods : { dump cget configure trapByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccNotice_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"trapByLid") == 0) { + cmd = _wrap_ccNotice_trapByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-generic_type") == 0) { + cmd = _wrap_ccNotice_generic_type_set; + } else if (strcmp(_str,"-issuer_lid") == 0) { + cmd = _wrap_ccNotice_issuer_lid_set; + } else if (strcmp(_str,"-toggle_count") == 0) { + cmd = _wrap_ccNotice_toggle_count_set; + } else if (strcmp(_str,"-issuer_gid") == 0) { + cmd = _wrap_ccNotice_issuer_gid_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -generic_type -issuer_lid -toggle_count -issuer_gid }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -generic_type -issuer_lid -toggle_count -issuer_gid }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-generic_type") == 0) { + cmd = _wrap_ccNotice_generic_type_get; + } else if (strcmp(_str,"-issuer_lid") == 0) { + cmd = _wrap_ccNotice_issuer_lid_get; + } else if (strcmp(_str,"-toggle_count") == 0) { + cmd = _wrap_ccNotice_toggle_count_get; + } else if (strcmp(_str,"-issuer_gid") == 0) { + cmd = _wrap_ccNotice_issuer_gid_get; + } else if (strcmp(_str,"-g_or_v") == 0) { + cmd = _wrap_ccNotice_g_or_v_get; + } else if (strcmp(_str,"-data_details") == 0) { + cmd = _wrap_ccNotice_data_details_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccNotice_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -generic_type -issuer_lid -toggle_count -issuer_gid -g_or_v -data_details }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -generic_type -issuer_lid -toggle_count -issuer_gid -g_or_v -data_details }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccNotice_generic_type_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-generic_type ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_issuer_lid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-issuer_lid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_toggle_count_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-toggle_count ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_issuer_gid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-issuer_gid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_g_or_v_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-g_or_v ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-data_details ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure trapByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccNoticeCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccNotice * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccNotice_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccNotice_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccNotice object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccNoticeMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define ccNotice_g_or_v_generic_get(_swigobj) (&_swigobj->generic) +static int _wrap_ccNotice_g_or_v_generic_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ccNotice_g_or_v_generic * _result; + ccNotice_g_or_v * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_get { ccNotice_g_or_v * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_get. Expected _ccNotice_g_or_v_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ccNotice_g_or_v_generic *)ccNotice_g_or_v_generic_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + SWIG_SetPointerObj(tcl_result,(void *) _result,"_ccNotice_g_or_v_generic_p"); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccNotice_g_or_vMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccNotice_g_or_v methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccNotice_g_or_v_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-generic") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccNotice_g_or_v_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -generic }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -generic }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccNotice_g_or_v_generic_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-generic ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccNotice_g_or_vCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccNotice_g_or_v * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccNotice_g_or_v_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccNotice_g_or_v_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccNotice_g_or_v object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccNotice_g_or_vMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define ccNotice_data_details_raw_data_get(_swigobj) (&_swigobj->raw_data) +static int _wrap_ccNotice_data_details_raw_data_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ccNotice_data_details_raw_data * _result; + ccNotice_data_details * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_raw_data_get { ccNotice_data_details * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_raw_data_get. Expected _ccNotice_data_details_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ccNotice_data_details_raw_data *)ccNotice_data_details_raw_data_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + SWIG_SetPointerObj(tcl_result,(void *) _result,"_ccNotice_data_details_raw_data_p"); + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_get(_swigobj) (&_swigobj->ntc_0) +static int _wrap_ccNotice_data_details_ntc_0_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ccNotice_data_details_ntc_0 * _result; + ccNotice_data_details * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_get { ccNotice_data_details * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_get. Expected _ccNotice_data_details_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ccNotice_data_details_ntc_0 *)ccNotice_data_details_ntc_0_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + SWIG_SetPointerObj(tcl_result,(void *) _result,"_ccNotice_data_details_ntc_0_p"); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccNotice_data_detailsMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccNotice_data_details methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccNotice_data_details_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-raw_data") == 0) { + cmd = _wrap_ccNotice_data_details_raw_data_get; + } else if (strcmp(_str,"-ntc_0") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccNotice_data_details_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -raw_data -ntc_0 }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -raw_data -ntc_0 }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccNotice_data_details_raw_data_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-raw_data ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-ntc_0 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccNotice_data_detailsCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccNotice_data_details * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccNotice_data_details_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccNotice_data_details_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccNotice_data_details object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccNotice_data_detailsMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define ccNotice_g_or_v_generic_prod_type_msb_set(_swigobj,_swigval) (_swigobj->prod_type_msb = *(_swigval),_swigval) +static int _wrap_ccNotice_g_or_v_generic_prod_type_msb_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice_g_or_v_generic * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_prod_type_msb_set { ccNotice_g_or_v_generic * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_generic_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_prod_type_msb_set. Expected _ccNotice_g_or_v_generic_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)ccNotice_g_or_v_generic_prod_type_msb_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_g_or_v_generic_prod_type_msb_get(_swigobj) (&_swigobj->prod_type_msb) +static int _wrap_ccNotice_g_or_v_generic_prod_type_msb_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice_g_or_v_generic * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_prod_type_msb_get { ccNotice_g_or_v_generic * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_generic_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_prod_type_msb_get. Expected _ccNotice_g_or_v_generic_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)ccNotice_g_or_v_generic_prod_type_msb_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_g_or_v_generic_prod_type_lsb_set(_swigobj,_swigval) (_swigobj->prod_type_lsb = *(_swigval),_swigval) +static int _wrap_ccNotice_g_or_v_generic_prod_type_lsb_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_g_or_v_generic * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_prod_type_lsb_set { ccNotice_g_or_v_generic * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_generic_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_prod_type_lsb_set. Expected _ccNotice_g_or_v_generic_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_g_or_v_generic_prod_type_lsb_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_g_or_v_generic_prod_type_lsb_get(_swigobj) (&_swigobj->prod_type_lsb) +static int _wrap_ccNotice_g_or_v_generic_prod_type_lsb_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_g_or_v_generic * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_prod_type_lsb_get { ccNotice_g_or_v_generic * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_generic_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_prod_type_lsb_get. Expected _ccNotice_g_or_v_generic_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_g_or_v_generic_prod_type_lsb_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_g_or_v_generic_trap_num_set(_swigobj,_swigval) (_swigobj->trap_num = *(_swigval),_swigval) +static int _wrap_ccNotice_g_or_v_generic_trap_num_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_g_or_v_generic * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_trap_num_set { ccNotice_g_or_v_generic * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_generic_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_trap_num_set. Expected _ccNotice_g_or_v_generic_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_g_or_v_generic_trap_num_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_g_or_v_generic_trap_num_get(_swigobj) (&_swigobj->trap_num) +static int _wrap_ccNotice_g_or_v_generic_trap_num_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_g_or_v_generic * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_g_or_v_generic_trap_num_get { ccNotice_g_or_v_generic * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_g_or_v_generic_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_g_or_v_generic_trap_num_get. Expected _ccNotice_g_or_v_generic_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_g_or_v_generic_trap_num_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccNotice_g_or_v_genericMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccNotice_g_or_v_generic methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccNotice_g_or_v_generic_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-prod_type_msb") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_prod_type_msb_set; + } else if (strcmp(_str,"-prod_type_lsb") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_prod_type_lsb_set; + } else if (strcmp(_str,"-trap_num") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_trap_num_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -prod_type_msb -prod_type_lsb -trap_num }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -prod_type_msb -prod_type_lsb -trap_num }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-prod_type_msb") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_prod_type_msb_get; + } else if (strcmp(_str,"-prod_type_lsb") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_prod_type_lsb_get; + } else if (strcmp(_str,"-trap_num") == 0) { + cmd = _wrap_ccNotice_g_or_v_generic_trap_num_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccNotice_g_or_v_generic_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -prod_type_msb -prod_type_lsb -trap_num }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -prod_type_msb -prod_type_lsb -trap_num }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccNotice_g_or_v_generic_prod_type_msb_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-prod_type_msb ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_g_or_v_generic_prod_type_lsb_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-prod_type_lsb ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_g_or_v_generic_trap_num_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trap_num ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccNotice_g_or_v_genericCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccNotice_g_or_v_generic * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccNotice_g_or_v_generic_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccNotice_g_or_v_generic_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccNotice_g_or_v_generic object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccNotice_g_or_v_genericMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +static uint8_array_t * ccNotice_data_details_raw_data_details_set(ccNotice_data_details_raw_data *obj, uint8_array_t val[54]) { +{ + int i; + for (i=0; i <54 ; i++) { + obj->details[i] = *(val+i); + } +} + return (uint8_array_t *) val; +} +static int _wrap_ccNotice_data_details_raw_data_details_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccNotice_data_details_raw_data * _arg0; + uint8_array_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t entrys[54]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_raw_data_details_set { ccNotice_data_details_raw_data * } { uint8_array_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_raw_data_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_raw_data_details_set. Expected _ccNotice_data_details_raw_data_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char *buff; + char *p_ch; + char *last; + long int entry; + + int i = 0; + buff = (char *)malloc((strlen(Tcl_GetStringFromObj(objv[2],NULL))+1)*sizeof(char)); + strcpy(buff, Tcl_GetStringFromObj(objv[2],NULL)); + p_ch = strtok_r(buff, " \t",&last); + while (p_ch && (i < 54)) + { + entry = strtol(p_ch, NULL, 0); + if (entry > 0xff) + { + printf("Error: wrong format or out of range value for expected uint8_t entry: %s\n", p_ch); + return TCL_ERROR; + } + entrys[i++] = entry; + p_ch = strtok_r(NULL, " \t", &last); + } + for (; i < 54; i++) entrys[i] = 0; + + free(buff); + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)ccNotice_data_details_raw_data_details_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <54 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define ccNotice_data_details_raw_data_details_get(_swigobj) ((uint8_array_t *) _swigobj->details) +static int _wrap_ccNotice_data_details_raw_data_details_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccNotice_data_details_raw_data * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_raw_data_details_get { ccNotice_data_details_raw_data * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_raw_data_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_raw_data_details_get. Expected _ccNotice_data_details_raw_data_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)ccNotice_data_details_raw_data_details_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <54 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccNotice_data_details_raw_dataMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccNotice_data_details_raw_data methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccNotice_data_details_raw_data_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-details") == 0) { + cmd = _wrap_ccNotice_data_details_raw_data_details_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -details }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -details }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-details") == 0) { + cmd = _wrap_ccNotice_data_details_raw_data_details_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccNotice_data_details_raw_data_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -details }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -details }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccNotice_data_details_raw_data_details_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-details ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccNotice_data_details_raw_dataCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccNotice_data_details_raw_data * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccNotice_data_details_raw_data_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccNotice_data_details_raw_data_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccNotice_data_details_raw_data object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccNotice_data_details_raw_dataMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define ccNotice_data_details_ntc_0_source_lid_set(_swigobj,_swigval) (_swigobj->source_lid = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_source_lid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_source_lid_set { ccNotice_data_details_ntc_0 * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_source_lid_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_data_details_ntc_0_source_lid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_source_lid_get(_swigobj) (&_swigobj->source_lid) +static int _wrap_ccNotice_data_details_ntc_0_source_lid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_source_lid_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_source_lid_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_data_details_ntc_0_source_lid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_method_set(_swigobj,_swigval) (_swigobj->method = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_method_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_method_set { ccNotice_data_details_ntc_0 * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_method_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)ccNotice_data_details_ntc_0_method_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_method_get(_swigobj) (&_swigobj->method) +static int _wrap_ccNotice_data_details_ntc_0_method_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_method_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_method_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)ccNotice_data_details_ntc_0_method_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_resv0_set(_swigobj,_swigval) (_swigobj->resv0 = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_resv0_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_resv0_set { ccNotice_data_details_ntc_0 * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_resv0_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)ccNotice_data_details_ntc_0_resv0_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_resv0_get(_swigobj) (&_swigobj->resv0) +static int _wrap_ccNotice_data_details_ntc_0_resv0_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_resv0_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_resv0_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)ccNotice_data_details_ntc_0_resv0_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_attr_id_set(_swigobj,_swigval) (_swigobj->attr_id = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_attr_id_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_attr_id_set { ccNotice_data_details_ntc_0 * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_attr_id_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_data_details_ntc_0_attr_id_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_attr_id_get(_swigobj) (&_swigobj->attr_id) +static int _wrap_ccNotice_data_details_ntc_0_attr_id_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_attr_id_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_attr_id_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_data_details_ntc_0_attr_id_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_resv1_set(_swigobj,_swigval) (_swigobj->resv1 = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_resv1_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_resv1_set { ccNotice_data_details_ntc_0 * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_resv1_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_data_details_ntc_0_resv1_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_resv1_get(_swigobj) (&_swigobj->resv1) +static int _wrap_ccNotice_data_details_ntc_0_resv1_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_resv1_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_resv1_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)ccNotice_data_details_ntc_0_resv1_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_attr_mod_set(_swigobj,_swigval) (_swigobj->attr_mod = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_attr_mod_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_attr_mod_set { ccNotice_data_details_ntc_0 * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_attr_mod_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)ccNotice_data_details_ntc_0_attr_mod_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_attr_mod_get(_swigobj) (&_swigobj->attr_mod) +static int _wrap_ccNotice_data_details_ntc_0_attr_mod_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_attr_mod_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_attr_mod_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)ccNotice_data_details_ntc_0_attr_mod_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_qp_set(_swigobj,_swigval) (_swigobj->qp = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_qp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_qp_set { ccNotice_data_details_ntc_0 * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_qp_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)ccNotice_data_details_ntc_0_qp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_qp_get(_swigobj) (&_swigobj->qp) +static int _wrap_ccNotice_data_details_ntc_0_qp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_qp_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_qp_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)ccNotice_data_details_ntc_0_qp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_cc_key_set(_swigobj,_swigval) (_swigobj->cc_key = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_cc_key_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net64_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_net64_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint64_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_cc_key_set { ccNotice_data_details_ntc_0 * } { ib_net64_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_cc_key_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton64(strtoull(Tcl_GetStringFromObj(objv[2],NULL), NULL, 16)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net64_t *)ccNotice_data_details_ntc_0_cc_key_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "0x%016" PRIx64, cl_ntoh64(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_cc_key_get(_swigobj) (&_swigobj->cc_key) +static int _wrap_ccNotice_data_details_ntc_0_cc_key_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net64_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_cc_key_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_cc_key_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net64_t *)ccNotice_data_details_ntc_0_cc_key_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "0x%016" PRIx64, cl_ntoh64(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_source_gid_set(_swigobj,_swigval) (_swigobj->source_gid = *(_swigval),_swigval) +static int _wrap_ccNotice_data_details_ntc_0_source_gid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + ib_gid_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_gid_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_source_gid_set { ccNotice_data_details_ntc_0 * } { ib_gid_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_source_gid_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char buf[38]; + char *p_prefix, *p_guid; + char *str_token; + + strcpy(buf, Tcl_GetStringFromObj(objv[2],NULL)); + p_prefix = strtok_r(buf,":", &str_token); + p_guid = strtok_r(NULL, " ", &str_token); + temp.unicast.prefix = cl_hton64(strtoull(p_prefix, NULL, 16)); + errno = 0; + if (errno) { + printf("Wrong format for gid prefix:%s\n", p_prefix); + return TCL_ERROR; + } + + temp.unicast.interface_id = cl_hton64(strtoull(p_guid, NULL, 16)); + if (errno) { + printf("Wrong format for gid guid:%s\n", p_guid); + return TCL_ERROR; + } + + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)ccNotice_data_details_ntc_0_source_gid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_source_gid_get(_swigobj) (&_swigobj->source_gid) +static int _wrap_ccNotice_data_details_ntc_0_source_gid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_gid_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_source_gid_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_source_gid_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_gid_t *)ccNotice_data_details_ntc_0_source_gid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[38]; + sprintf(buff, "0x%016" PRIx64 ":0x%016" PRIx64, + cl_ntoh64(_result->unicast.prefix), + cl_ntoh64(_result->unicast.interface_id) + ); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static uint8_array_t * ccNotice_data_details_ntc_0_padding_set(ccNotice_data_details_ntc_0 *obj, uint8_array_t val[14]) { +{ + int i; + for (i=0; i <14 ; i++) { + obj->padding[i] = *(val+i); + } +} + return (uint8_array_t *) val; +} +static int _wrap_ccNotice_data_details_ntc_0_padding_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + uint8_array_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t entrys[14]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_padding_set { ccNotice_data_details_ntc_0 * } { uint8_array_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_padding_set. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char *buff; + char *p_ch; + char *last; + long int entry; + + int i = 0; + buff = (char *)malloc((strlen(Tcl_GetStringFromObj(objv[2],NULL))+1)*sizeof(char)); + strcpy(buff, Tcl_GetStringFromObj(objv[2],NULL)); + p_ch = strtok_r(buff, " \t",&last); + while (p_ch && (i < 14)) + { + entry = strtol(p_ch, NULL, 0); + if (entry > 0xff) + { + printf("Error: wrong format or out of range value for expected uint8_t entry: %s\n", p_ch); + return TCL_ERROR; + } + entrys[i++] = entry; + p_ch = strtok_r(NULL, " \t", &last); + } + for (; i < 14; i++) entrys[i] = 0; + + free(buff); + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)ccNotice_data_details_ntc_0_padding_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <14 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define ccNotice_data_details_ntc_0_padding_get(_swigobj) ((uint8_array_t *) _swigobj->padding) +static int _wrap_ccNotice_data_details_ntc_0_padding_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccNotice_data_details_ntc_0 * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccNotice_data_details_ntc_0_padding_get { ccNotice_data_details_ntc_0 * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccNotice_data_details_ntc_0_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccNotice_data_details_ntc_0_padding_get. Expected _ccNotice_data_details_ntc_0_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)ccNotice_data_details_ntc_0_padding_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <14 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccNotice_data_details_ntc_0MethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccNotice_data_details_ntc_0 methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccNotice_data_details_ntc_0_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-source_lid") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_source_lid_set; + } else if (strcmp(_str,"-method") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_method_set; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_resv0_set; + } else if (strcmp(_str,"-attr_id") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_attr_id_set; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_resv1_set; + } else if (strcmp(_str,"-attr_mod") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_attr_mod_set; + } else if (strcmp(_str,"-qp") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_qp_set; + } else if (strcmp(_str,"-cc_key") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_cc_key_set; + } else if (strcmp(_str,"-source_gid") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_source_gid_set; + } else if (strcmp(_str,"-padding") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_padding_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -source_lid -method -resv0 -attr_id -resv1 -attr_mod -qp -cc_key -source_gid -padding }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -source_lid -method -resv0 -attr_id -resv1 -attr_mod -qp -cc_key -source_gid -padding }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-source_lid") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_source_lid_get; + } else if (strcmp(_str,"-method") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_method_get; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_resv0_get; + } else if (strcmp(_str,"-attr_id") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_attr_id_get; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_resv1_get; + } else if (strcmp(_str,"-attr_mod") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_attr_mod_get; + } else if (strcmp(_str,"-qp") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_qp_get; + } else if (strcmp(_str,"-cc_key") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_cc_key_get; + } else if (strcmp(_str,"-source_gid") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_source_gid_get; + } else if (strcmp(_str,"-padding") == 0) { + cmd = _wrap_ccNotice_data_details_ntc_0_padding_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccNotice_data_details_ntc_0_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -source_lid -method -resv0 -attr_id -resv1 -attr_mod -qp -cc_key -source_gid -padding }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -source_lid -method -resv0 -attr_id -resv1 -attr_mod -qp -cc_key -source_gid -padding }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccNotice_data_details_ntc_0_source_lid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-source_lid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_method_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-method ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_resv0_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv0 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_attr_id_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-attr_id ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_resv1_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv1 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_attr_mod_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-attr_mod ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_qp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-qp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_cc_key_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cc_key ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_source_gid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-source_gid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccNotice_data_details_ntc_0_padding_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-padding ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccNotice_data_details_ntc_0Cmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccNotice_data_details_ntc_0 * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccNotice_data_details_ntc_0_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccNotice_data_details_ntc_0_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccNotice_data_details_ntc_0 object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccNotice_data_details_ntc_0MethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_cong_info_cong_info_set(_swigobj,_swigval) (_swigobj->cong_info = *(_swigval),_swigval) +static int _wrap_ccCongestionInfo_cong_info_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_cong_info_set { ccCongestionInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_cong_info_set. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_info_cong_info_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_info_cong_info_get(_swigobj) (&_swigobj->cong_info) +static int _wrap_ccCongestionInfo_cong_info_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_cong_info_get { ccCongestionInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_cong_info_get. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_info_cong_info_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_info_resv_set(_swigobj,_swigval) (_swigobj->resv = *(_swigval),_swigval) +static int _wrap_ccCongestionInfo_resv_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_resv_set { ccCongestionInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_resv_set. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_info_resv_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_info_resv_get(_swigobj) (&_swigobj->resv) +static int _wrap_ccCongestionInfo_resv_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_resv_get { ccCongestionInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_resv_get. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_info_resv_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_info_ctrl_table_cap_set(_swigobj,_swigval) (_swigobj->ctrl_table_cap = *(_swigval),_swigval) +static int _wrap_ccCongestionInfo_ctrl_table_cap_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_ctrl_table_cap_set { ccCongestionInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_ctrl_table_cap_set. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_info_ctrl_table_cap_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_info_ctrl_table_cap_get(_swigobj) (&_swigobj->ctrl_table_cap) +static int _wrap_ccCongestionInfo_ctrl_table_cap_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_ctrl_table_cap_get { ccCongestionInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_ctrl_table_cap_get. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_info_ctrl_table_cap_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static int ccCongestionInfo_getByLid(ccCongestionInfo *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCongestionInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_INFO), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccCongestionInfo_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccCongestionInfo * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionInfo_getByLid { ccCongestionInfo * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionInfo_getByLid. Expected _ccCongestionInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccCongestionInfo_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccCongestionInfoMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccCongestionInfo methods : { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccCongestionInfo_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccCongestionInfo_getByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-cong_info") == 0) { + cmd = _wrap_ccCongestionInfo_cong_info_set; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccCongestionInfo_resv_set; + } else if (strcmp(_str,"-ctrl_table_cap") == 0) { + cmd = _wrap_ccCongestionInfo_ctrl_table_cap_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -cong_info -resv -ctrl_table_cap }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -cong_info -resv -ctrl_table_cap }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-cong_info") == 0) { + cmd = _wrap_ccCongestionInfo_cong_info_get; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccCongestionInfo_resv_get; + } else if (strcmp(_str,"-ctrl_table_cap") == 0) { + cmd = _wrap_ccCongestionInfo_ctrl_table_cap_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccCongestionInfo_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -cong_info -resv -ctrl_table_cap }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -cong_info -resv -ctrl_table_cap }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccCongestionInfo_cong_info_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cong_info ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCongestionInfo_resv_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCongestionInfo_ctrl_table_cap_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-ctrl_table_cap ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccCongestionInfoCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccCongestionInfo * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccCongestionInfo_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccCongestionInfo_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccCongestionInfo object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccCongestionInfoMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_cong_key_info_cc_key_set(_swigobj,_swigval) (_swigobj->cc_key = *(_swigval),_swigval) +static int _wrap_ccCongestionKeyInfo_cc_key_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net64_t * _result; + ccCongestionKeyInfo * _arg0; + ib_net64_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint64_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_cc_key_set { ccCongestionKeyInfo * } { ib_net64_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_cc_key_set. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton64(strtoull(Tcl_GetStringFromObj(objv[2],NULL), NULL, 16)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net64_t *)_ib_cong_key_info_cc_key_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "0x%016" PRIx64, cl_ntoh64(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_cc_key_get(_swigobj) (&_swigobj->cc_key) +static int _wrap_ccCongestionKeyInfo_cc_key_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net64_t * _result; + ccCongestionKeyInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_cc_key_get { ccCongestionKeyInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_cc_key_get. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net64_t *)_ib_cong_key_info_cc_key_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "0x%016" PRIx64, cl_ntoh64(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_protect_bit_set(_swigobj,_swigval) (_swigobj->protect_bit = *(_swigval),_swigval) +static int _wrap_ccCongestionKeyInfo_protect_bit_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionKeyInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_protect_bit_set { ccCongestionKeyInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_protect_bit_set. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_key_info_protect_bit_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_protect_bit_get(_swigobj) (&_swigobj->protect_bit) +static int _wrap_ccCongestionKeyInfo_protect_bit_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionKeyInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_protect_bit_get { ccCongestionKeyInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_protect_bit_get. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_key_info_protect_bit_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_resv_set(_swigobj,_swigval) (_swigobj->resv = *(_swigval),_swigval) +static int _wrap_ccCongestionKeyInfo_resv_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionKeyInfo * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_resv_set { ccCongestionKeyInfo * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_resv_set. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_key_info_resv_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_resv_get(_swigobj) (&_swigobj->resv) +static int _wrap_ccCongestionKeyInfo_resv_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCongestionKeyInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_resv_get { ccCongestionKeyInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_resv_get. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_key_info_resv_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_lease_period_set(_swigobj,_swigval) (_swigobj->lease_period = *(_swigval),_swigval) +static int _wrap_ccCongestionKeyInfo_lease_period_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCongestionKeyInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_lease_period_set { ccCongestionKeyInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_lease_period_set. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_key_info_lease_period_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_lease_period_get(_swigobj) (&_swigobj->lease_period) +static int _wrap_ccCongestionKeyInfo_lease_period_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCongestionKeyInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_lease_period_get { ccCongestionKeyInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_lease_period_get. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_key_info_lease_period_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_violations_set(_swigobj,_swigval) (_swigobj->violations = *(_swigval),_swigval) +static int _wrap_ccCongestionKeyInfo_violations_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCongestionKeyInfo * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_violations_set { ccCongestionKeyInfo * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_violations_set. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_key_info_violations_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_key_info_violations_get(_swigobj) (&_swigobj->violations) +static int _wrap_ccCongestionKeyInfo_violations_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCongestionKeyInfo * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_violations_get { ccCongestionKeyInfo * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_violations_get. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_key_info_violations_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static int ccCongestionKeyInfo_getByLid(ccCongestionKeyInfo *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCongestionKeyInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_KEY_INFO), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccCongestionKeyInfo_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccCongestionKeyInfo * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_getByLid { ccCongestionKeyInfo * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_getByLid. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccCongestionKeyInfo_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +static int ccCongestionKeyInfo_setByLid(ccCongestionKeyInfo *self,uint16_t lid) { + return ibcc_send_mad_by_lid( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCongestionKeyInfo), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_KEY_INFO), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } +static int _wrap_ccCongestionKeyInfo_setByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccCongestionKeyInfo * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCongestionKeyInfo_setByLid { ccCongestionKeyInfo * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCongestionKeyInfo_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCongestionKeyInfo_setByLid. Expected _ccCongestionKeyInfo_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccCongestionKeyInfo_setByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccCongestionKeyInfoMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccCongestionKeyInfo methods : { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccCongestionKeyInfo_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccCongestionKeyInfo_getByLid; + } else if (strcmp(_str,"setByLid") == 0) { + cmd = _wrap_ccCongestionKeyInfo_setByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-cc_key") == 0) { + cmd = _wrap_ccCongestionKeyInfo_cc_key_set; + } else if (strcmp(_str,"-protect_bit") == 0) { + cmd = _wrap_ccCongestionKeyInfo_protect_bit_set; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccCongestionKeyInfo_resv_set; + } else if (strcmp(_str,"-lease_period") == 0) { + cmd = _wrap_ccCongestionKeyInfo_lease_period_set; + } else if (strcmp(_str,"-violations") == 0) { + cmd = _wrap_ccCongestionKeyInfo_violations_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -cc_key -protect_bit -resv -lease_period -violations }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -cc_key -protect_bit -resv -lease_period -violations }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-cc_key") == 0) { + cmd = _wrap_ccCongestionKeyInfo_cc_key_get; + } else if (strcmp(_str,"-protect_bit") == 0) { + cmd = _wrap_ccCongestionKeyInfo_protect_bit_get; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccCongestionKeyInfo_resv_get; + } else if (strcmp(_str,"-lease_period") == 0) { + cmd = _wrap_ccCongestionKeyInfo_lease_period_get; + } else if (strcmp(_str,"-violations") == 0) { + cmd = _wrap_ccCongestionKeyInfo_violations_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccCongestionKeyInfo_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -cc_key -protect_bit -resv -lease_period -violations }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -cc_key -protect_bit -resv -lease_period -violations }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccCongestionKeyInfo_cc_key_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cc_key ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCongestionKeyInfo_protect_bit_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-protect_bit ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCongestionKeyInfo_resv_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCongestionKeyInfo_lease_period_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-lease_period ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCongestionKeyInfo_violations_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-violations ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccCongestionKeyInfoCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccCongestionKeyInfo * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccCongestionKeyInfo_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccCongestionKeyInfo_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccCongestionKeyInfo object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccCongestionKeyInfoMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_cong_log_event_sw_slid_set(_swigobj,_swigval) (_swigobj->slid = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_sw_t_slid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_sw_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_slid_set { ib_cong_log_event_sw_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_slid_set. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_sw_slid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_slid_get(_swigobj) (&_swigobj->slid) +static int _wrap_ib_cong_log_event_sw_t_slid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_sw_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_slid_get { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_slid_get. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_sw_slid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_dlid_set(_swigobj,_swigval) (_swigobj->dlid = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_sw_t_dlid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_sw_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_dlid_set { ib_cong_log_event_sw_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_dlid_set. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_sw_dlid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_dlid_get(_swigobj) (&_swigobj->dlid) +static int _wrap_ib_cong_log_event_sw_t_dlid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_sw_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_dlid_get { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_dlid_get. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_sw_dlid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_resv0_sl_set(_swigobj,_swigval) (_swigobj->resv0_sl = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_sw_t_resv0_sl_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_cong_log_event_sw_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_resv0_sl_set { ib_cong_log_event_sw_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_resv0_sl_set. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_log_event_sw_resv0_sl_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_resv0_sl_get(_swigobj) (&_swigobj->resv0_sl) +static int _wrap_ib_cong_log_event_sw_t_resv0_sl_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_cong_log_event_sw_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_resv0_sl_get { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_resv0_sl_get. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_log_event_sw_resv0_sl_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_resv1_set(_swigobj,_swigval) (_swigobj->resv1 = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_sw_t_resv1_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_cong_log_event_sw_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_resv1_set { ib_cong_log_event_sw_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_resv1_set. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_log_event_sw_resv1_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_resv1_get(_swigobj) (&_swigobj->resv1) +static int _wrap_ib_cong_log_event_sw_t_resv1_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_cong_log_event_sw_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_resv1_get { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_resv1_get. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_cong_log_event_sw_resv1_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_resv2_set(_swigobj,_swigval) (_swigobj->resv2 = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_sw_t_resv2_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_sw_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_resv2_set { ib_cong_log_event_sw_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_resv2_set. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_sw_resv2_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_resv2_get(_swigobj) (&_swigobj->resv2) +static int _wrap_ib_cong_log_event_sw_t_resv2_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_sw_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_resv2_get { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_resv2_get. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_sw_resv2_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_time_stamp_set(_swigobj,_swigval) (_swigobj->time_stamp = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_sw_t_time_stamp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_sw_t * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_time_stamp_set { ib_cong_log_event_sw_t * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_time_stamp_set. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_sw_time_stamp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_sw_time_stamp_get(_swigobj) (&_swigobj->time_stamp) +static int _wrap_ib_cong_log_event_sw_t_time_stamp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_sw_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_sw_t_time_stamp_get { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_sw_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_sw_t_time_stamp_get. Expected _ib_cong_log_event_sw_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_sw_time_stamp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int Tclib_cong_log_event_sw_tMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ib_cong_log_event_sw_t methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ib_cong_log_event_sw_t_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-slid") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_slid_set; + } else if (strcmp(_str,"-dlid") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_dlid_set; + } else if (strcmp(_str,"-resv0_sl") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_resv0_sl_set; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_resv1_set; + } else if (strcmp(_str,"-resv2") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_resv2_set; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_time_stamp_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -slid -dlid -resv0_sl -resv1 -resv2 -time_stamp }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -slid -dlid -resv0_sl -resv1 -resv2 -time_stamp }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-slid") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_slid_get; + } else if (strcmp(_str,"-dlid") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_dlid_get; + } else if (strcmp(_str,"-resv0_sl") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_resv0_sl_get; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_resv1_get; + } else if (strcmp(_str,"-resv2") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_resv2_get; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ib_cong_log_event_sw_t_time_stamp_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ib_cong_log_event_sw_t_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -slid -dlid -resv0_sl -resv1 -resv2 -time_stamp }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -slid -dlid -resv0_sl -resv1 -resv2 -time_stamp }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ib_cong_log_event_sw_t_slid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-slid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_sw_t_dlid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-dlid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_sw_t_resv0_sl_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv0_sl ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_sw_t_resv1_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv1 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_sw_t_resv2_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv2 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_sw_t_time_stamp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-time_stamp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int Tclib_cong_log_event_sw_tCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ib_cong_log_event_sw_t * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ib_cong_log_event_sw_t_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ib_cong_log_event_sw_t_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ib_cong_log_event_sw_t object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, Tclib_cong_log_event_sw_tMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_cong_log_event_ca_resv0_local_qp_set(_swigobj,_swigval) (_swigobj->resv0_local_qp = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_ca_t_resv0_local_qp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_ca_t * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_resv0_local_qp_set { ib_cong_log_event_ca_t * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_resv0_local_qp_set. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_ca_resv0_local_qp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_resv0_local_qp_get(_swigobj) (&_swigobj->resv0_local_qp) +static int _wrap_ib_cong_log_event_ca_t_resv0_local_qp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_ca_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_resv0_local_qp_get { ib_cong_log_event_ca_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_resv0_local_qp_get. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_ca_resv0_local_qp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_remote_qp_sl_service_type_set(_swigobj,_swigval) (_swigobj->remote_qp_sl_service_type = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_ca_t * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_remote_qp_sl_service_type_set { ib_cong_log_event_ca_t * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_remote_qp_sl_service_type_set. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_ca_remote_qp_sl_service_type_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_remote_qp_sl_service_type_get(_swigobj) (&_swigobj->remote_qp_sl_service_type) +static int _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_ca_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_remote_qp_sl_service_type_get { ib_cong_log_event_ca_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_remote_qp_sl_service_type_get. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_ca_remote_qp_sl_service_type_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_remote_lid_set(_swigobj,_swigval) (_swigobj->remote_lid = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_ca_t_remote_lid_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_ca_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_remote_lid_set { ib_cong_log_event_ca_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_remote_lid_set. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_ca_remote_lid_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_remote_lid_get(_swigobj) (&_swigobj->remote_lid) +static int _wrap_ib_cong_log_event_ca_t_remote_lid_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_ca_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_remote_lid_get { ib_cong_log_event_ca_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_remote_lid_get. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_ca_remote_lid_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_resv1_set(_swigobj,_swigval) (_swigobj->resv1 = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_ca_t_resv1_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_ca_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_resv1_set { ib_cong_log_event_ca_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_resv1_set. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_ca_resv1_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_resv1_get(_swigobj) (&_swigobj->resv1) +static int _wrap_ib_cong_log_event_ca_t_resv1_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cong_log_event_ca_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_resv1_get { ib_cong_log_event_ca_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_resv1_get. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cong_log_event_ca_resv1_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_time_stamp_set(_swigobj,_swigval) (_swigobj->time_stamp = *(_swigval),_swigval) +static int _wrap_ib_cong_log_event_ca_t_time_stamp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_ca_t * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_time_stamp_set { ib_cong_log_event_ca_t * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_time_stamp_set. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_ca_time_stamp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cong_log_event_ca_time_stamp_get(_swigobj) (&_swigobj->time_stamp) +static int _wrap_ib_cong_log_event_ca_t_time_stamp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ib_cong_log_event_ca_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cong_log_event_ca_t_time_stamp_get { ib_cong_log_event_ca_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cong_log_event_ca_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cong_log_event_ca_t_time_stamp_get. Expected _ib_cong_log_event_ca_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_cong_log_event_ca_time_stamp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int Tclib_cong_log_event_ca_tMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ib_cong_log_event_ca_t methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ib_cong_log_event_ca_t_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-resv0_local_qp") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_resv0_local_qp_set; + } else if (strcmp(_str,"-remote_qp_sl_service_type") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_set; + } else if (strcmp(_str,"-remote_lid") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_remote_lid_set; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_resv1_set; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_time_stamp_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -resv0_local_qp -remote_qp_sl_service_type -remote_lid -resv1 -time_stamp }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -resv0_local_qp -remote_qp_sl_service_type -remote_lid -resv1 -time_stamp }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-resv0_local_qp") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_resv0_local_qp_get; + } else if (strcmp(_str,"-remote_qp_sl_service_type") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_get; + } else if (strcmp(_str,"-remote_lid") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_remote_lid_get; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_resv1_get; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ib_cong_log_event_ca_t_time_stamp_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ib_cong_log_event_ca_t_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -resv0_local_qp -remote_qp_sl_service_type -remote_lid -resv1 -time_stamp }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -resv0_local_qp -remote_qp_sl_service_type -remote_lid -resv1 -time_stamp }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ib_cong_log_event_ca_t_resv0_local_qp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv0_local_qp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-remote_qp_sl_service_type ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_ca_t_remote_lid_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-remote_lid ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_ca_t_resv1_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv1 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_cong_log_event_ca_t_time_stamp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-time_stamp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int Tclib_cong_log_event_ca_tCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ib_cong_log_event_ca_t * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ib_cong_log_event_ca_t_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ib_cong_log_event_ca_t_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ib_cong_log_event_ca_t object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, Tclib_cong_log_event_ca_tMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_sw_cong_log_log_type_set(_swigobj,_swigval) (_swigobj->log_type = *(_swigval),_swigval) +static int _wrap_ccSWCongestionLog_log_type_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionLog * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_log_type_set { ccSWCongestionLog * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_log_type_set. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_log_log_type_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_log_type_get(_swigobj) (&_swigobj->log_type) +static int _wrap_ccSWCongestionLog_log_type_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_log_type_get { ccSWCongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_log_type_get. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_log_log_type_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_cong_flags_set(_swigobj,_swigval) (_swigobj->cong_flags = *(_swigval),_swigval) +static int _wrap_ccSWCongestionLog_cong_flags_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionLog * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_cong_flags_set { ccSWCongestionLog * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_cong_flags_set. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_log_cong_flags_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_cong_flags_get(_swigobj) (&_swigobj->cong_flags) +static int _wrap_ccSWCongestionLog_cong_flags_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_cong_flags_get { ccSWCongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_cong_flags_get. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_log_cong_flags_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_event_counter_set(_swigobj,_swigval) (_swigobj->event_counter = *(_swigval),_swigval) +static int _wrap_ccSWCongestionLog_event_counter_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccSWCongestionLog * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_event_counter_set { ccSWCongestionLog * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_event_counter_set. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_cong_log_event_counter_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_event_counter_get(_swigobj) (&_swigobj->event_counter) +static int _wrap_ccSWCongestionLog_event_counter_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccSWCongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_event_counter_get { ccSWCongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_event_counter_get. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_cong_log_event_counter_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_time_stamp_set(_swigobj,_swigval) (_swigobj->time_stamp = *(_swigval),_swigval) +static int _wrap_ccSWCongestionLog_time_stamp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccSWCongestionLog * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_time_stamp_set { ccSWCongestionLog * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_time_stamp_set. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_sw_cong_log_time_stamp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_log_time_stamp_get(_swigobj) (&_swigobj->time_stamp) +static int _wrap_ccSWCongestionLog_time_stamp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccSWCongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_time_stamp_get { ccSWCongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_time_stamp_get. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_sw_cong_log_time_stamp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static uint8_array_t * _ib_sw_cong_log_port_map_set(ccSWCongestionLog *obj, uint8_array_t val[32]) { +{ + int i; + for (i=0; i <32 ; i++) { + obj->port_map[i] = *(val+i); + } +} + return (uint8_array_t *) val; +} +static int _wrap_ccSWCongestionLog_port_map_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccSWCongestionLog * _arg0; + uint8_array_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t entrys[32]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_port_map_set { ccSWCongestionLog * } { uint8_array_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_port_map_set. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char *buff; + char *p_ch; + char *last; + long int entry; + + int i = 0; + buff = (char *)malloc((strlen(Tcl_GetStringFromObj(objv[2],NULL))+1)*sizeof(char)); + strcpy(buff, Tcl_GetStringFromObj(objv[2],NULL)); + p_ch = strtok_r(buff, " \t",&last); + while (p_ch && (i < 32)) + { + entry = strtol(p_ch, NULL, 0); + if (entry > 0xff) + { + printf("Error: wrong format or out of range value for expected uint8_t entry: %s\n", p_ch); + return TCL_ERROR; + } + entrys[i++] = entry; + p_ch = strtok_r(NULL, " \t", &last); + } + for (; i < 32; i++) entrys[i] = 0; + + free(buff); + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)_ib_sw_cong_log_port_map_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <32 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_sw_cong_log_port_map_get(_swigobj) ((uint8_array_t *) _swigobj->port_map) +static int _wrap_ccSWCongestionLog_port_map_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccSWCongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_port_map_get { ccSWCongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_port_map_get. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)_ib_sw_cong_log_port_map_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <32 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static ib_cong_log_event_sw_t * _ib_sw_cong_log_entry_list_set(ccSWCongestionLog *obj, ib_cong_log_event_sw_t val[15]) { +{ + int i; + for (i=0; i <15 ; i++) { + obj->entry_list[i] = *(val+i); + } +} + return (ib_cong_log_event_sw_t *) val; +} +static int _wrap_ccSWCongestionLog_entry_list_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_cong_log_event_sw_t * _result; + ccSWCongestionLog * _arg0; + ib_cong_log_event_sw_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_cong_log_event_sw_t entrys[15]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_entry_list_set { ccSWCongestionLog * } { ib_cong_log_event_sw_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_entry_list_set. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, objv[2], &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(objv[2],NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure log_event: + * 1. Configure the whole list by providing list of value groups: + * ccSWCongestionLogMad configure -log_event {{1 2 3 4} {5 6 7 8}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccSWCongestionLogMad configure -log_event {{4 1 2 3 4} {8 5 6 7 8}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 4) + option = 1; + else if (numElements == 5) + option = 2; + else { + printf("Error: wrong number of elements for SW Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].slid = cl_hton16(value); break; + case 1: entrys[i].dlid = cl_hton16(value); break; + case 2: entrys[i].resv0_sl = value; break; + case 3: entrys[i++].time_stamp = cl_hton32(value); break; + default: break; + } + } + } + + for (; i < 15; i++) { + entrys[i].slid = 0; + entrys[i].dlid = 0; + entrys[i].resv0_sl = 0; + entrys[i].time_stamp = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 5) { + printf("Error: wrong number of elements for SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 5; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].slid = cl_hton16(value); break; + case 2: entrys[entry_index].dlid = cl_hton16(value); break; + case 3: entrys[entry_index].resv0_sl = value; break; + case 4: entrys[entry_index].time_stamp = cl_hton32(value); break; + default: break; + } + } + + } + } + + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_cong_log_event_sw_t *)_ib_sw_cong_log_entry_list_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <15 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].slid) + sprintf(buff, " -slid 0x%04x", cl_ntoh16(_result[i].slid)); + else + sprintf(buff, " -slid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].dlid) + sprintf(buff, " -dlid 0x%04x", cl_ntoh16(_result[i].dlid)); + else + sprintf(buff, " -dlid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].resv0_sl) + sprintf(buff, " -resv0_sl 0x%02x", _result[i].resv0_sl); + else + sprintf(buff, " -resv0_sl 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].time_stamp) + sprintf(buff, " -time_stamp 0x%08x", cl_ntoh32(_result[i].time_stamp)); + else + sprintf(buff, " -time_stamp 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_sw_cong_log_entry_list_get(_swigobj) ((ib_cong_log_event_sw_t *) _swigobj->entry_list) +static int _wrap_ccSWCongestionLog_entry_list_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_cong_log_event_sw_t * _result; + ccSWCongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_entry_list_get { ccSWCongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_entry_list_get. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_cong_log_event_sw_t *)_ib_sw_cong_log_entry_list_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <15 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].slid) + sprintf(buff, " -slid 0x%04x", cl_ntoh16(_result[i].slid)); + else + sprintf(buff, " -slid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].dlid) + sprintf(buff, " -dlid 0x%04x", cl_ntoh16(_result[i].dlid)); + else + sprintf(buff, " -dlid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].resv0_sl) + sprintf(buff, " -resv0_sl 0x%02x", _result[i].resv0_sl); + else + sprintf(buff, " -resv0_sl 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].time_stamp) + sprintf(buff, " -time_stamp 0x%08x", cl_ntoh32(_result[i].time_stamp)); + else + sprintf(buff, " -time_stamp 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static int ccSWCongestionLog_getByLid(ccSWCongestionLog *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + (uint8_t *)self, // log data + sizeof(ccSWCongestionLog), // log data size + NULL, // mgt data + 0, // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_LOG), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccSWCongestionLog_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccSWCongestionLog * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionLog_getByLid { ccSWCongestionLog * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionLog_getByLid. Expected _ccSWCongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccSWCongestionLog_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccSWCongestionLogMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccSWCongestionLog methods : { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccSWCongestionLog_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccSWCongestionLog_getByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-log_type") == 0) { + cmd = _wrap_ccSWCongestionLog_log_type_set; + } else if (strcmp(_str,"-cong_flags") == 0) { + cmd = _wrap_ccSWCongestionLog_cong_flags_set; + } else if (strcmp(_str,"-event_counter") == 0) { + cmd = _wrap_ccSWCongestionLog_event_counter_set; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ccSWCongestionLog_time_stamp_set; + } else if (strcmp(_str,"-port_map") == 0) { + cmd = _wrap_ccSWCongestionLog_port_map_set; + } else if (strcmp(_str,"-entry_list") == 0) { + cmd = _wrap_ccSWCongestionLog_entry_list_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -log_type -cong_flags -event_counter -time_stamp -port_map -entry_list }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -log_type -cong_flags -event_counter -time_stamp -port_map -entry_list }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-log_type") == 0) { + cmd = _wrap_ccSWCongestionLog_log_type_get; + } else if (strcmp(_str,"-cong_flags") == 0) { + cmd = _wrap_ccSWCongestionLog_cong_flags_get; + } else if (strcmp(_str,"-event_counter") == 0) { + cmd = _wrap_ccSWCongestionLog_event_counter_get; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ccSWCongestionLog_time_stamp_get; + } else if (strcmp(_str,"-port_map") == 0) { + cmd = _wrap_ccSWCongestionLog_port_map_get; + } else if (strcmp(_str,"-entry_list") == 0) { + cmd = _wrap_ccSWCongestionLog_entry_list_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccSWCongestionLog_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -log_type -cong_flags -event_counter -time_stamp -port_map -entry_list }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -log_type -cong_flags -event_counter -time_stamp -port_map -entry_list }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccSWCongestionLog_log_type_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-log_type ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionLog_cong_flags_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cong_flags ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionLog_event_counter_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-event_counter ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionLog_time_stamp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-time_stamp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionLog_port_map_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-port_map ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionLog_entry_list_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-entry_list ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccSWCongestionLogCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccSWCongestionLog * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccSWCongestionLog_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccSWCongestionLog_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccSWCongestionLog object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccSWCongestionLogMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_ca_cong_log_log_type_set(_swigobj,_swigval) (_swigobj->log_type = *(_swigval),_swigval) +static int _wrap_ccCACongestionLog_log_type_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionLog * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_log_type_set { ccCACongestionLog * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_log_type_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_log_log_type_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_log_type_get(_swigobj) (&_swigobj->log_type) +static int _wrap_ccCACongestionLog_log_type_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_log_type_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_log_type_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_log_log_type_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_cong_flags_set(_swigobj,_swigval) (_swigobj->cong_flags = *(_swigval),_swigval) +static int _wrap_ccCACongestionLog_cong_flags_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionLog * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_cong_flags_set { ccCACongestionLog * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_cong_flags_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_log_cong_flags_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_cong_flags_get(_swigobj) (&_swigobj->cong_flags) +static int _wrap_ccCACongestionLog_cong_flags_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_cong_flags_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_cong_flags_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_log_cong_flags_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_event_counter_set(_swigobj,_swigval) (_swigobj->event_counter = *(_swigval),_swigval) +static int _wrap_ccCACongestionLog_event_counter_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionLog * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_event_counter_set { ccCACongestionLog * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_event_counter_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_log_event_counter_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_event_counter_get(_swigobj) (&_swigobj->event_counter) +static int _wrap_ccCACongestionLog_event_counter_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_event_counter_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_event_counter_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_log_event_counter_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_event_map_set(_swigobj,_swigval) (_swigobj->event_map = *(_swigval),_swigval) +static int _wrap_ccCACongestionLog_event_map_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionLog * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_event_map_set { ccCACongestionLog * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_event_map_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_log_event_map_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_event_map_get(_swigobj) (&_swigobj->event_map) +static int _wrap_ccCACongestionLog_event_map_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_event_map_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_event_map_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_log_event_map_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_resv_set(_swigobj,_swigval) (_swigobj->resv = *(_swigval),_swigval) +static int _wrap_ccCACongestionLog_resv_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionLog * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_resv_set { ccCACongestionLog * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_resv_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_log_resv_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_resv_get(_swigobj) (&_swigobj->resv) +static int _wrap_ccCACongestionLog_resv_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_resv_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_resv_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_log_resv_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_time_stamp_set(_swigobj,_swigval) (_swigobj->time_stamp = *(_swigval),_swigval) +static int _wrap_ccCACongestionLog_time_stamp_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccCACongestionLog * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_time_stamp_set { ccCACongestionLog * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_time_stamp_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_ca_cong_log_time_stamp_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_log_time_stamp_get(_swigobj) (&_swigobj->time_stamp) +static int _wrap_ccCACongestionLog_time_stamp_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_time_stamp_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_time_stamp_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_ca_cong_log_time_stamp_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static ib_cong_log_event_ca_t * _ib_ca_cong_log_log_event_set(ccCACongestionLog *obj, ib_cong_log_event_ca_t val[13]) { +{ + int i; + for (i=0; i <13 ; i++) { + obj->log_event[i] = *(val+i); + } +} + return (ib_cong_log_event_ca_t *) val; +} +static int _wrap_ccCACongestionLog_log_event_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_cong_log_event_ca_t * _result; + ccCACongestionLog * _arg0; + ib_cong_log_event_ca_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_cong_log_event_ca_t entrys[13]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_log_event_set { ccCACongestionLog * } { ib_cong_log_event_ca_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_log_event_set. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, objv[2], &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(objv[2],NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure log_event: + * 1. Configure the whole list by providing list of value groups: + * ccCACongestionLogMad configure -log_event {{1 2 3 4} {5 6 7 8}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccCACongestionLogMad configure -log_event {{4 1 2 3 4} {8 5 6 7 8}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 4) + option = 1; + else if (numElements == 5) + option = 2; + else { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].resv0_local_qp = cl_hton32(value); break; + case 1: entrys[i].remote_qp_sl_service_type = cl_hton32(value); break; + case 2: entrys[i].remote_lid = cl_hton16(value); break; + case 3: entrys[i++].time_stamp = cl_hton32(value); break; + default: break; + } + } + } + + for (; i < 13; i++) { + entrys[i].resv0_local_qp = 0; + entrys[i].remote_qp_sl_service_type = 0; + entrys[i].remote_lid = 0; + entrys[i].time_stamp = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 5) { + printf("Error: wrong number of elements for CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 5; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Log Event: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].resv0_local_qp = cl_hton32(value); break; + case 2: entrys[entry_index].remote_qp_sl_service_type = cl_hton32(value); break; + case 3: entrys[entry_index].remote_lid = cl_hton16(value); break; + case 4: entrys[entry_index].time_stamp = cl_hton32(value); break; + default: break; + } + } + + } + } + + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_cong_log_event_ca_t *)_ib_ca_cong_log_log_event_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <13 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].resv0_local_qp) + sprintf(buff, " -resv0_local_qp 0x%08x", cl_ntoh32(_result[i].resv0_local_qp)); + else + sprintf(buff, " -resv0_local_qp 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].remote_qp_sl_service_type) + sprintf(buff, " -remote_qp_sl_service_type 0x%08x", cl_ntoh32(_result[i].remote_qp_sl_service_type)); + else + sprintf(buff, " -remote_qp_sl_service_type 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].remote_lid) + sprintf(buff, " -remote_lid 0x%04x", cl_ntoh16(_result[i].remote_lid)); + else + sprintf(buff, " -remote_lid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].time_stamp) + sprintf(buff, " -time_stamp 0x%08x", cl_ntoh32(_result[i].time_stamp)); + else + sprintf(buff, " -time_stamp 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_ca_cong_log_log_event_get(_swigobj) ((ib_cong_log_event_ca_t *) _swigobj->log_event) +static int _wrap_ccCACongestionLog_log_event_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_cong_log_event_ca_t * _result; + ccCACongestionLog * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_log_event_get { ccCACongestionLog * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_log_event_get. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_cong_log_event_ca_t *)_ib_ca_cong_log_log_event_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <13 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].resv0_local_qp) + sprintf(buff, " -resv0_local_qp 0x%08x", cl_ntoh32(_result[i].resv0_local_qp)); + else + sprintf(buff, " -resv0_local_qp 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].remote_qp_sl_service_type) + sprintf(buff, " -remote_qp_sl_service_type 0x%08x", cl_ntoh32(_result[i].remote_qp_sl_service_type)); + else + sprintf(buff, " -remote_qp_sl_service_type 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].remote_lid) + sprintf(buff, " -remote_lid 0x%04x", cl_ntoh16(_result[i].remote_lid)); + else + sprintf(buff, " -remote_lid 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].time_stamp) + sprintf(buff, " -time_stamp 0x%08x", cl_ntoh32(_result[i].time_stamp)); + else + sprintf(buff, " -time_stamp 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static int ccCACongestionLog_getByLid(ccCACongestionLog *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + (uint8_t *)self, // log data + sizeof(ccCACongestionLog), // log data size + NULL, // mgt data + 0, // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CONG_LOG), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccCACongestionLog_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccCACongestionLog * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionLog_getByLid { ccCACongestionLog * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionLog_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionLog_getByLid. Expected _ccCACongestionLog_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccCACongestionLog_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccCACongestionLogMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccCACongestionLog methods : { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccCACongestionLog_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccCACongestionLog_getByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-log_type") == 0) { + cmd = _wrap_ccCACongestionLog_log_type_set; + } else if (strcmp(_str,"-cong_flags") == 0) { + cmd = _wrap_ccCACongestionLog_cong_flags_set; + } else if (strcmp(_str,"-event_counter") == 0) { + cmd = _wrap_ccCACongestionLog_event_counter_set; + } else if (strcmp(_str,"-event_map") == 0) { + cmd = _wrap_ccCACongestionLog_event_map_set; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccCACongestionLog_resv_set; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ccCACongestionLog_time_stamp_set; + } else if (strcmp(_str,"-log_event") == 0) { + cmd = _wrap_ccCACongestionLog_log_event_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -log_type -cong_flags -event_counter -event_map -resv -time_stamp -log_event }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -log_type -cong_flags -event_counter -event_map -resv -time_stamp -log_event }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-log_type") == 0) { + cmd = _wrap_ccCACongestionLog_log_type_get; + } else if (strcmp(_str,"-cong_flags") == 0) { + cmd = _wrap_ccCACongestionLog_cong_flags_get; + } else if (strcmp(_str,"-event_counter") == 0) { + cmd = _wrap_ccCACongestionLog_event_counter_get; + } else if (strcmp(_str,"-event_map") == 0) { + cmd = _wrap_ccCACongestionLog_event_map_get; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccCACongestionLog_resv_get; + } else if (strcmp(_str,"-time_stamp") == 0) { + cmd = _wrap_ccCACongestionLog_time_stamp_get; + } else if (strcmp(_str,"-log_event") == 0) { + cmd = _wrap_ccCACongestionLog_log_event_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccCACongestionLog_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -log_type -cong_flags -event_counter -event_map -resv -time_stamp -log_event }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -log_type -cong_flags -event_counter -event_map -resv -time_stamp -log_event }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccCACongestionLog_log_type_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-log_type ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionLog_cong_flags_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cong_flags ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionLog_event_counter_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-event_counter ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionLog_event_map_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-event_map ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionLog_resv_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionLog_time_stamp_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-time_stamp ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionLog_log_event_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-log_event ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccCACongestionLogCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccCACongestionLog * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccCACongestionLog_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccCACongestionLog_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccCACongestionLog object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccCACongestionLogMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_sw_cong_setting_control_map_set(_swigobj,_swigval) (_swigobj->control_map = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_control_map_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccSWCongestionSetting * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_control_map_set { ccSWCongestionSetting * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_control_map_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_sw_cong_setting_control_map_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_control_map_get(_swigobj) (&_swigobj->control_map) +static int _wrap_ccSWCongestionSetting_control_map_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_control_map_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_control_map_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_sw_cong_setting_control_map_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static uint8_array_t * _ib_sw_cong_setting_victim_mask_set(ccSWCongestionSetting *obj, uint8_array_t val[32]) { +{ + int i; + for (i=0; i <32 ; i++) { + obj->victim_mask[i] = *(val+i); + } +} + return (uint8_array_t *) val; +} +static int _wrap_ccSWCongestionSetting_victim_mask_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccSWCongestionSetting * _arg0; + uint8_array_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t entrys[32]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_victim_mask_set { ccSWCongestionSetting * } { uint8_array_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_victim_mask_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char *buff; + char *p_ch; + char *last; + long int entry; + + int i = 0; + buff = (char *)malloc((strlen(Tcl_GetStringFromObj(objv[2],NULL))+1)*sizeof(char)); + strcpy(buff, Tcl_GetStringFromObj(objv[2],NULL)); + p_ch = strtok_r(buff, " \t",&last); + while (p_ch && (i < 32)) + { + entry = strtol(p_ch, NULL, 0); + if (entry > 0xff) + { + printf("Error: wrong format or out of range value for expected uint8_t entry: %s\n", p_ch); + return TCL_ERROR; + } + entrys[i++] = entry; + p_ch = strtok_r(NULL, " \t", &last); + } + for (; i < 32; i++) entrys[i] = 0; + + free(buff); + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)_ib_sw_cong_setting_victim_mask_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <32 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_sw_cong_setting_victim_mask_get(_swigobj) ((uint8_array_t *) _swigobj->victim_mask) +static int _wrap_ccSWCongestionSetting_victim_mask_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_victim_mask_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_victim_mask_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)_ib_sw_cong_setting_victim_mask_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <32 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static uint8_array_t * _ib_sw_cong_setting_credit_mask_set(ccSWCongestionSetting *obj, uint8_array_t val[32]) { +{ + int i; + for (i=0; i <32 ; i++) { + obj->credit_mask[i] = *(val+i); + } +} + return (uint8_array_t *) val; +} +static int _wrap_ccSWCongestionSetting_credit_mask_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccSWCongestionSetting * _arg0; + uint8_array_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t entrys[32]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_credit_mask_set { ccSWCongestionSetting * } { uint8_array_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_credit_mask_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + char *buff; + char *p_ch; + char *last; + long int entry; + + int i = 0; + buff = (char *)malloc((strlen(Tcl_GetStringFromObj(objv[2],NULL))+1)*sizeof(char)); + strcpy(buff, Tcl_GetStringFromObj(objv[2],NULL)); + p_ch = strtok_r(buff, " \t",&last); + while (p_ch && (i < 32)) + { + entry = strtol(p_ch, NULL, 0); + if (entry > 0xff) + { + printf("Error: wrong format or out of range value for expected uint8_t entry: %s\n", p_ch); + return TCL_ERROR; + } + entrys[i++] = entry; + p_ch = strtok_r(NULL, " \t", &last); + } + for (; i < 32; i++) entrys[i] = 0; + + free(buff); + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)_ib_sw_cong_setting_credit_mask_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <32 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_sw_cong_setting_credit_mask_get(_swigobj) ((uint8_array_t *) _swigobj->credit_mask) +static int _wrap_ccSWCongestionSetting_credit_mask_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_array_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_credit_mask_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_credit_mask_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_array_t *)_ib_sw_cong_setting_credit_mask_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[8]; + for (i=0; i <32 ; i++) { + sprintf(buff, "0x%02x ", *(_result+i)); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_sw_cong_setting_threshold_set(_swigobj,_swigval) (_swigobj->threshold = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_threshold_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_threshold_set { ccSWCongestionSetting * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_threshold_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_threshold_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_threshold_get(_swigobj) (&_swigobj->threshold) +static int _wrap_ccSWCongestionSetting_threshold_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_threshold_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_threshold_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_threshold_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_packet_size_set(_swigobj,_swigval) (_swigobj->packet_size = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_packet_size_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_packet_size_set { ccSWCongestionSetting * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_packet_size_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_packet_size_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_packet_size_get(_swigobj) (&_swigobj->packet_size) +static int _wrap_ccSWCongestionSetting_packet_size_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_packet_size_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_packet_size_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_packet_size_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_cs_threshold_set(_swigobj,_swigval) (_swigobj->cs_threshold = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_cs_threshold_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_cs_threshold_set { ccSWCongestionSetting * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_cs_threshold_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_cs_threshold_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_cs_threshold_get(_swigobj) (&_swigobj->cs_threshold) +static int _wrap_ccSWCongestionSetting_cs_threshold_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_cs_threshold_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_cs_threshold_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_cs_threshold_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_resv0_set(_swigobj,_swigval) (_swigobj->resv0 = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_resv0_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_resv0_set { ccSWCongestionSetting * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_resv0_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_resv0_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_resv0_get(_swigobj) (&_swigobj->resv0) +static int _wrap_ccSWCongestionSetting_resv0_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_resv0_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_resv0_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_cong_setting_resv0_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_cs_return_delay_set(_swigobj,_swigval) (_swigobj->cs_return_delay = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_cs_return_delay_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccSWCongestionSetting * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_cs_return_delay_set { ccSWCongestionSetting * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_cs_return_delay_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_cong_setting_cs_return_delay_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_cs_return_delay_get(_swigobj) (&_swigobj->cs_return_delay) +static int _wrap_ccSWCongestionSetting_cs_return_delay_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_cs_return_delay_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_cs_return_delay_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_cong_setting_cs_return_delay_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_marking_rate_set(_swigobj,_swigval) (_swigobj->marking_rate = *(_swigval),_swigval) +static int _wrap_ccSWCongestionSetting_marking_rate_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccSWCongestionSetting * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_marking_rate_set { ccSWCongestionSetting * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_marking_rate_set. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_cong_setting_marking_rate_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_cong_setting_marking_rate_get(_swigobj) (&_swigobj->marking_rate) +static int _wrap_ccSWCongestionSetting_marking_rate_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccSWCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_marking_rate_get { ccSWCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_marking_rate_get. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_cong_setting_marking_rate_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static int ccSWCongestionSetting_getByLid(ccSWCongestionSetting *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccSWCongestionSetting_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccSWCongestionSetting * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_getByLid { ccSWCongestionSetting * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_getByLid. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccSWCongestionSetting_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +static int ccSWCongestionSetting_setByLid(ccSWCongestionSetting *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } +static int _wrap_ccSWCongestionSetting_setByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccSWCongestionSetting * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWCongestionSetting_setByLid { ccSWCongestionSetting * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWCongestionSetting_setByLid. Expected _ccSWCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccSWCongestionSetting_setByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccSWCongestionSettingMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccSWCongestionSetting methods : { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccSWCongestionSetting_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccSWCongestionSetting_getByLid; + } else if (strcmp(_str,"setByLid") == 0) { + cmd = _wrap_ccSWCongestionSetting_setByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-control_map") == 0) { + cmd = _wrap_ccSWCongestionSetting_control_map_set; + } else if (strcmp(_str,"-victim_mask") == 0) { + cmd = _wrap_ccSWCongestionSetting_victim_mask_set; + } else if (strcmp(_str,"-credit_mask") == 0) { + cmd = _wrap_ccSWCongestionSetting_credit_mask_set; + } else if (strcmp(_str,"-threshold") == 0) { + cmd = _wrap_ccSWCongestionSetting_threshold_set; + } else if (strcmp(_str,"-packet_size") == 0) { + cmd = _wrap_ccSWCongestionSetting_packet_size_set; + } else if (strcmp(_str,"-cs_threshold") == 0) { + cmd = _wrap_ccSWCongestionSetting_cs_threshold_set; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ccSWCongestionSetting_resv0_set; + } else if (strcmp(_str,"-cs_return_delay") == 0) { + cmd = _wrap_ccSWCongestionSetting_cs_return_delay_set; + } else if (strcmp(_str,"-marking_rate") == 0) { + cmd = _wrap_ccSWCongestionSetting_marking_rate_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -control_map -victim_mask -credit_mask -threshold -packet_size -cs_threshold -resv0 -cs_return_delay -marking_rate }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -control_map -victim_mask -credit_mask -threshold -packet_size -cs_threshold -resv0 -cs_return_delay -marking_rate }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-control_map") == 0) { + cmd = _wrap_ccSWCongestionSetting_control_map_get; + } else if (strcmp(_str,"-victim_mask") == 0) { + cmd = _wrap_ccSWCongestionSetting_victim_mask_get; + } else if (strcmp(_str,"-credit_mask") == 0) { + cmd = _wrap_ccSWCongestionSetting_credit_mask_get; + } else if (strcmp(_str,"-threshold") == 0) { + cmd = _wrap_ccSWCongestionSetting_threshold_get; + } else if (strcmp(_str,"-packet_size") == 0) { + cmd = _wrap_ccSWCongestionSetting_packet_size_get; + } else if (strcmp(_str,"-cs_threshold") == 0) { + cmd = _wrap_ccSWCongestionSetting_cs_threshold_get; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ccSWCongestionSetting_resv0_get; + } else if (strcmp(_str,"-cs_return_delay") == 0) { + cmd = _wrap_ccSWCongestionSetting_cs_return_delay_get; + } else if (strcmp(_str,"-marking_rate") == 0) { + cmd = _wrap_ccSWCongestionSetting_marking_rate_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccSWCongestionSetting_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -control_map -victim_mask -credit_mask -threshold -packet_size -cs_threshold -resv0 -cs_return_delay -marking_rate }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -control_map -victim_mask -credit_mask -threshold -packet_size -cs_threshold -resv0 -cs_return_delay -marking_rate }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccSWCongestionSetting_control_map_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-control_map ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_victim_mask_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-victim_mask ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_credit_mask_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-credit_mask ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_threshold_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-threshold ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_packet_size_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-packet_size ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_cs_threshold_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cs_threshold ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_resv0_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv0 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_cs_return_delay_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cs_return_delay ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccSWCongestionSetting_marking_rate_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-marking_rate ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccSWCongestionSettingCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccSWCongestionSetting * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccSWCongestionSetting_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccSWCongestionSetting_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccSWCongestionSetting object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccSWCongestionSettingMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_sw_port_cong_setting_element_valid_ctrl_type_res_threshold_set(_swigobj,_swigval) (_swigobj->valid_ctrl_type_res_threshold = *(_swigval),_swigval) +static int _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_sw_port_cong_setting_element_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_set { ib_sw_port_cong_setting_element_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_sw_port_cong_setting_element_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_set. Expected _ib_sw_port_cong_setting_element_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_port_cong_setting_element_valid_ctrl_type_res_threshold_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_port_cong_setting_element_valid_ctrl_type_res_threshold_get(_swigobj) (&_swigobj->valid_ctrl_type_res_threshold) +static int _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_sw_port_cong_setting_element_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get { ib_sw_port_cong_setting_element_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_sw_port_cong_setting_element_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get. Expected _ib_sw_port_cong_setting_element_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_port_cong_setting_element_valid_ctrl_type_res_threshold_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_port_cong_setting_element_packet_size_set(_swigobj,_swigval) (_swigobj->packet_size = *(_swigval),_swigval) +static int _wrap_ib_sw_port_cong_setting_element_t_packet_size_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_sw_port_cong_setting_element_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_sw_port_cong_setting_element_t_packet_size_set { ib_sw_port_cong_setting_element_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_sw_port_cong_setting_element_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_sw_port_cong_setting_element_t_packet_size_set. Expected _ib_sw_port_cong_setting_element_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_port_cong_setting_element_packet_size_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_port_cong_setting_element_packet_size_get(_swigobj) (&_swigobj->packet_size) +static int _wrap_ib_sw_port_cong_setting_element_t_packet_size_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_sw_port_cong_setting_element_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_sw_port_cong_setting_element_t_packet_size_get { ib_sw_port_cong_setting_element_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_sw_port_cong_setting_element_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_sw_port_cong_setting_element_t_packet_size_get. Expected _ib_sw_port_cong_setting_element_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_sw_port_cong_setting_element_packet_size_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_port_cong_setting_element_cong_param_set(_swigobj,_swigval) (_swigobj->cong_param = *(_swigval),_swigval) +static int _wrap_ib_sw_port_cong_setting_element_t_cong_param_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_sw_port_cong_setting_element_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_sw_port_cong_setting_element_t_cong_param_set { ib_sw_port_cong_setting_element_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_sw_port_cong_setting_element_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_sw_port_cong_setting_element_t_cong_param_set. Expected _ib_sw_port_cong_setting_element_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_port_cong_setting_element_cong_param_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_sw_port_cong_setting_element_cong_param_get(_swigobj) (&_swigobj->cong_param) +static int _wrap_ib_sw_port_cong_setting_element_t_cong_param_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_sw_port_cong_setting_element_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_sw_port_cong_setting_element_t_cong_param_get { ib_sw_port_cong_setting_element_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_sw_port_cong_setting_element_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_sw_port_cong_setting_element_t_cong_param_get. Expected _ib_sw_port_cong_setting_element_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_sw_port_cong_setting_element_cong_param_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int Tclib_sw_port_cong_setting_element_tMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ib_sw_port_cong_setting_element_t methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ib_sw_port_cong_setting_element_t_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-valid_ctrl_type_res_threshold") == 0) { + cmd = _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_set; + } else if (strcmp(_str,"-packet_size") == 0) { + cmd = _wrap_ib_sw_port_cong_setting_element_t_packet_size_set; + } else if (strcmp(_str,"-cong_param") == 0) { + cmd = _wrap_ib_sw_port_cong_setting_element_t_cong_param_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -valid_ctrl_type_res_threshold -packet_size -cong_param }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -valid_ctrl_type_res_threshold -packet_size -cong_param }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-valid_ctrl_type_res_threshold") == 0) { + cmd = _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get; + } else if (strcmp(_str,"-packet_size") == 0) { + cmd = _wrap_ib_sw_port_cong_setting_element_t_packet_size_get; + } else if (strcmp(_str,"-cong_param") == 0) { + cmd = _wrap_ib_sw_port_cong_setting_element_t_cong_param_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ib_sw_port_cong_setting_element_t_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -valid_ctrl_type_res_threshold -packet_size -cong_param }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -valid_ctrl_type_res_threshold -packet_size -cong_param }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-valid_ctrl_type_res_threshold ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_sw_port_cong_setting_element_t_packet_size_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-packet_size ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_sw_port_cong_setting_element_t_cong_param_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-cong_param ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int Tclib_sw_port_cong_setting_element_tCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ib_sw_port_cong_setting_element_t * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ib_sw_port_cong_setting_element_t_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ib_sw_port_cong_setting_element_t_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ib_sw_port_cong_setting_element_t object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, Tclib_sw_port_cong_setting_element_tMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +static ib_sw_port_cong_setting_element_t * _ib_sw_port_cong_setting_block_set(ccSWPortCongestionSetting *obj, ib_sw_port_cong_setting_element_t val[32]) { +{ + int i; + for (i=0; i <32 ; i++) { + obj->block[i] = *(val+i); + } +} + return (ib_sw_port_cong_setting_element_t *) val; +} +static int _wrap_ccSWPortCongestionSetting_block_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_sw_port_cong_setting_element_t * _result; + ccSWPortCongestionSetting * _arg0; + ib_sw_port_cong_setting_element_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_sw_port_cong_setting_element_t entrys[32]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWPortCongestionSetting_block_set { ccSWPortCongestionSetting * } { ib_sw_port_cong_setting_element_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWPortCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWPortCongestionSetting_block_set. Expected _ccSWPortCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, objv[2], &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(objv[2],NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure entry_list: + * 1. Configure the whole list by providing list of value groups: + * ccSWPortCongestionSettingMad configure -block {{1 2 3} {4 5 6}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccSWPortCongestionSettingMad configure -block {{4 1 2 3} {8 4 5 6}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 3) + option = 1; + else if (numElements == 4) + option = 2; + else { + printf("Error: wrong number of elements for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 3) { + printf("Error: wrong number of elements for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 3; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].valid_ctrl_type_res_threshold = value; break; + case 1: entrys[i].packet_size = value; break; + case 2: entrys[i++].cong_param = cl_hton16(value); break; + default: break; + } + } + } + + for (; i < 32; i++) { + entrys[i].valid_ctrl_type_res_threshold = 0; + entrys[i].packet_size = 0; + entrys[i].cong_param = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of SW Port Congestion Setting Element: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].valid_ctrl_type_res_threshold = value; break; + case 2: entrys[entry_index].packet_size = value; break; + case 3: entrys[entry_index].cong_param = cl_hton16(value); break; + default: break; + } + } + + } + } + + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_sw_port_cong_setting_element_t *)_ib_sw_port_cong_setting_block_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-block "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <32 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].valid_ctrl_type_res_threshold) + sprintf(buff, " -valid_ctrl_type_res_threshold 0x%02x", _result[i].valid_ctrl_type_res_threshold); + else + sprintf(buff, " -valid_ctrl_type_res_threshold 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].packet_size) + sprintf(buff, " -packet_size 0x%02x", _result[i].packet_size); + else + sprintf(buff, " -packet_size 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].cong_param) + sprintf(buff, " -cong_param 0x%04x", cl_ntoh16(_result[i].cong_param)); + else + sprintf(buff, " -cong_param 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_sw_port_cong_setting_block_get(_swigobj) ((ib_sw_port_cong_setting_element_t *) _swigobj->block) +static int _wrap_ccSWPortCongestionSetting_block_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_sw_port_cong_setting_element_t * _result; + ccSWPortCongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWPortCongestionSetting_block_get { ccSWPortCongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWPortCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWPortCongestionSetting_block_get. Expected _ccSWPortCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_sw_port_cong_setting_element_t *)_ib_sw_port_cong_setting_block_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-block "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <32 ; i++) { + sprintf(buff, "{#%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].valid_ctrl_type_res_threshold) + sprintf(buff, " -valid_ctrl_type_res_threshold 0x%02x", _result[i].valid_ctrl_type_res_threshold); + else + sprintf(buff, " -valid_ctrl_type_res_threshold 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].packet_size) + sprintf(buff, " -packet_size 0x%02x", _result[i].packet_size); + else + sprintf(buff, " -packet_size 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].cong_param) + sprintf(buff, " -cong_param 0x%04x", cl_ntoh16(_result[i].cong_param)); + else + sprintf(buff, " -cong_param 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static int ccSWPortCongestionSetting_getByLid(ccSWPortCongestionSetting *self,uint16_t lid,uint8_t block) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWPortCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_PORT_CONG_SETTING), + block, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccSWPortCongestionSetting_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccSWPortCongestionSetting * _arg0; + uint16_t * _arg1; + uint8_t * _arg2; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + uint8_t temp0; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 4) || (objc > 4)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWPortCongestionSetting_getByLid { ccSWPortCongestionSetting * } lid block ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWPortCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWPortCongestionSetting_getByLid. Expected _ccSWPortCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + temp0 = strtoul(Tcl_GetStringFromObj(objv[3],NULL), NULL, 0); + _arg2 = &temp0; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccSWPortCongestionSetting_getByLid(_arg0,*_arg1,*_arg2); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +static int ccSWPortCongestionSetting_setByLid(ccSWPortCongestionSetting *self,uint16_t lid,uint8_t block) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccSWPortCongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_SW_PORT_CONG_SETTING), + block, // attribute modifier + IB_MAD_METHOD_SET); + } +static int _wrap_ccSWPortCongestionSetting_setByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccSWPortCongestionSetting * _arg0; + uint16_t * _arg1; + uint8_t * _arg2; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + uint8_t temp0; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 4) || (objc > 4)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccSWPortCongestionSetting_setByLid { ccSWPortCongestionSetting * } lid block ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccSWPortCongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccSWPortCongestionSetting_setByLid. Expected _ccSWPortCongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + temp0 = strtoul(Tcl_GetStringFromObj(objv[3],NULL), NULL, 0); + _arg2 = &temp0; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccSWPortCongestionSetting_setByLid(_arg0,*_arg1,*_arg2); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccSWPortCongestionSettingMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccSWPortCongestionSetting methods : { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccSWPortCongestionSetting_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccSWPortCongestionSetting_getByLid; + } else if (strcmp(_str,"setByLid") == 0) { + cmd = _wrap_ccSWPortCongestionSetting_setByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-block") == 0) { + cmd = _wrap_ccSWPortCongestionSetting_block_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -block }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -block }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-block") == 0) { + cmd = _wrap_ccSWPortCongestionSetting_block_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccSWPortCongestionSetting_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -block }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -block }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccSWPortCongestionSetting_block_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-block ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccSWPortCongestionSettingCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccSWPortCongestionSetting * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccSWPortCongestionSetting_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccSWPortCongestionSetting_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccSWPortCongestionSetting object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccSWPortCongestionSettingMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_ca_cong_entry_ccti_timer_set(_swigobj,_swigval) (_swigobj->ccti_timer = *(_swigval),_swigval) +static int _wrap_ib_ca_cong_entry_t_ccti_timer_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_ca_cong_entry_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_ccti_timer_set { ib_ca_cong_entry_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_ccti_timer_set. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_entry_ccti_timer_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_ccti_timer_get(_swigobj) (&_swigobj->ccti_timer) +static int _wrap_ib_ca_cong_entry_t_ccti_timer_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_ca_cong_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_ccti_timer_get { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_ccti_timer_get. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_entry_ccti_timer_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_ccti_increase_set(_swigobj,_swigval) (_swigobj->ccti_increase = *(_swigval),_swigval) +static int _wrap_ib_ca_cong_entry_t_ccti_increase_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_ccti_increase_set { ib_ca_cong_entry_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_ccti_increase_set. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_ccti_increase_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_ccti_increase_get(_swigobj) (&_swigobj->ccti_increase) +static int _wrap_ib_ca_cong_entry_t_ccti_increase_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_ccti_increase_get { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_ccti_increase_get. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_ccti_increase_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_trigger_threshold_set(_swigobj,_swigval) (_swigobj->trigger_threshold = *(_swigval),_swigval) +static int _wrap_ib_ca_cong_entry_t_trigger_threshold_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_trigger_threshold_set { ib_ca_cong_entry_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_trigger_threshold_set. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_trigger_threshold_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_trigger_threshold_get(_swigobj) (&_swigobj->trigger_threshold) +static int _wrap_ib_ca_cong_entry_t_trigger_threshold_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_trigger_threshold_get { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_trigger_threshold_get. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_trigger_threshold_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_ccti_min_set(_swigobj,_swigval) (_swigobj->ccti_min = *(_swigval),_swigval) +static int _wrap_ib_ca_cong_entry_t_ccti_min_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_ccti_min_set { ib_ca_cong_entry_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_ccti_min_set. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_ccti_min_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_ccti_min_get(_swigobj) (&_swigobj->ccti_min) +static int _wrap_ib_ca_cong_entry_t_ccti_min_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_ccti_min_get { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_ccti_min_get. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_ccti_min_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_resv0_set(_swigobj,_swigval) (_swigobj->resv0 = *(_swigval),_swigval) +static int _wrap_ib_ca_cong_entry_t_resv0_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_resv0_set { ib_ca_cong_entry_t * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_resv0_set. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_resv0_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_resv0_get(_swigobj) (&_swigobj->resv0) +static int _wrap_ib_ca_cong_entry_t_resv0_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ib_ca_cong_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_resv0_get { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_resv0_get. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_entry_resv0_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_resv1_set(_swigobj,_swigval) (_swigobj->resv1 = *(_swigval),_swigval) +static int _wrap_ib_ca_cong_entry_t_resv1_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_ca_cong_entry_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_resv1_set { ib_ca_cong_entry_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_resv1_set. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_entry_resv1_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_entry_resv1_get(_swigobj) (&_swigobj->resv1) +static int _wrap_ib_ca_cong_entry_t_resv1_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_ca_cong_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_ca_cong_entry_t_resv1_get { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_ca_cong_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_ca_cong_entry_t_resv1_get. Expected _ib_ca_cong_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_entry_resv1_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int Tclib_ca_cong_entry_tMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ib_ca_cong_entry_t methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ib_ca_cong_entry_t_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-ccti_timer") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_ccti_timer_set; + } else if (strcmp(_str,"-ccti_increase") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_ccti_increase_set; + } else if (strcmp(_str,"-trigger_threshold") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_trigger_threshold_set; + } else if (strcmp(_str,"-ccti_min") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_ccti_min_set; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_resv0_set; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_resv1_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -ccti_timer -ccti_increase -trigger_threshold -ccti_min -resv0 -resv1 }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -ccti_timer -ccti_increase -trigger_threshold -ccti_min -resv0 -resv1 }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-ccti_timer") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_ccti_timer_get; + } else if (strcmp(_str,"-ccti_increase") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_ccti_increase_get; + } else if (strcmp(_str,"-trigger_threshold") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_trigger_threshold_get; + } else if (strcmp(_str,"-ccti_min") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_ccti_min_get; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_resv0_get; + } else if (strcmp(_str,"-resv1") == 0) { + cmd = _wrap_ib_ca_cong_entry_t_resv1_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ib_ca_cong_entry_t_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -ccti_timer -ccti_increase -trigger_threshold -ccti_min -resv0 -resv1 }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -ccti_timer -ccti_increase -trigger_threshold -ccti_min -resv0 -resv1 }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ib_ca_cong_entry_t_ccti_timer_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-ccti_timer ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_ca_cong_entry_t_ccti_increase_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-ccti_increase ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_ca_cong_entry_t_trigger_threshold_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-trigger_threshold ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_ca_cong_entry_t_ccti_min_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-ccti_min ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_ca_cong_entry_t_resv0_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv0 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ib_ca_cong_entry_t_resv1_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv1 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int Tclib_ca_cong_entry_tCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ib_ca_cong_entry_t * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ib_ca_cong_entry_t_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ib_ca_cong_entry_t_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ib_ca_cong_entry_t object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, Tclib_ca_cong_entry_tMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_ca_cong_setting_port_control_set(_swigobj,_swigval) (_swigobj->port_control = *(_swigval),_swigval) +static int _wrap_ccCACongestionSetting_port_control_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionSetting * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_port_control_set { ccCACongestionSetting * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_port_control_set. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_setting_port_control_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_setting_port_control_get(_swigobj) (&_swigobj->port_control) +static int _wrap_ccCACongestionSetting_port_control_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_port_control_get { ccCACongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_port_control_get. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_setting_port_control_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_setting_resv0_set(_swigobj,_swigval) (_swigobj->resv0 = *(_swigval),_swigval) +static int _wrap_ccCACongestionSetting_resv0_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionSetting * _arg0; + uint8_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint8_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_resv0_set { ccCACongestionSetting * } { uint8_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_resv0_set. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_setting_resv0_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_setting_resv0_get(_swigobj) (&_swigobj->resv0) +static int _wrap_ccCACongestionSetting_resv0_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + uint8_t * _result; + ccCACongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_resv0_get { ccCACongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_resv0_get. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (uint8_t *)_ib_ca_cong_setting_resv0_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", *_result); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_setting_control_map_set(_swigobj,_swigval) (_swigobj->control_map = *(_swigval),_swigval) +static int _wrap_ccCACongestionSetting_control_map_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionSetting * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_control_map_set { ccCACongestionSetting * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_control_map_set. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_setting_control_map_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_ca_cong_setting_control_map_get(_swigobj) (&_swigobj->control_map) +static int _wrap_ccCACongestionSetting_control_map_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccCACongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_control_map_get { ccCACongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_control_map_get. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_ca_cong_setting_control_map_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static ib_ca_cong_entry_t * _ib_ca_cong_setting_entry_list_set(ccCACongestionSetting *obj, ib_ca_cong_entry_t val[16]) { +{ + int i; + for (i=0; i <16 ; i++) { + obj->entry_list[i] = *(val+i); + } +} + return (ib_ca_cong_entry_t *) val; +} +static int _wrap_ccCACongestionSetting_entry_list_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_ca_cong_entry_t * _result; + ccCACongestionSetting * _arg0; + ib_ca_cong_entry_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_ca_cong_entry_t entrys[16]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_entry_list_set { ccCACongestionSetting * } { ib_ca_cong_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_entry_list_set. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + long int value; + long int entry_index = 0; + int k; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, objv[2], &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for CA Congestion Setting: %s\n", + Tcl_GetStringFromObj(objv[2],NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure entry_list: + * 1. Configure the whole list by providing list of value groups: + * ccCACongestionSetting configure -entry_list {{1 2 3 4} {5 6 7 8}} + * 2. Configure specific items from the list by providing index in addition to the above: + * ccCACongestionSetting configure -entry_list {{4 1 2 3 4} {8 5 6 7 8}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 4) + option = 1; + else if (numElements == 5) + option = 2; + else { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of groups of four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 4) { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 4; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entrys[i].ccti_timer = cl_hton16(value); break; + case 1: entrys[i].ccti_increase = value; break; + case 2: entrys[i].trigger_threshold = value; break; + case 3: entrys[i++].ccti_min = value; break; + default: break; + } + } + } + + for (; i < 16; i++) { + entrys[i].ccti_timer = 0; + entrys[i].ccti_increase = 0; + entrys[i].trigger_threshold = 0; + entrys[i].ccti_min = 0; + } + } + else { + /* + * second option - index and four values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 5) { + printf("Error: wrong number of elements for CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + for (k = 0; k < 5; k++) { + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], k, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CA Congestion Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + switch (k) { + case 0: entry_index = value; break; + case 1: entrys[entry_index].ccti_timer = cl_hton16(value); break; + case 2: entrys[entry_index].ccti_increase = value; break; + case 3: entrys[entry_index].trigger_threshold = value; break; + case 4: entrys[entry_index].ccti_min = value; break; + default: break; + } + } + + } + } + + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_ca_cong_entry_t *)_ib_ca_cong_setting_entry_list_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <16 ; i++) { + sprintf(buff, "{SL%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].ccti_timer) + sprintf(buff, " -ccti_timer 0x%04x", cl_ntoh16(_result[i].ccti_timer)); + else + sprintf(buff, " -ccti_timer 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].ccti_increase) + sprintf(buff, " -ccti_increase 0x%02x", _result[i].ccti_increase); + else + sprintf(buff, " -ccti_increase 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].trigger_threshold) + sprintf(buff, " -trigger_threshold 0x%02x", _result[i].trigger_threshold); + else + sprintf(buff, " -trigger_threshold 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].ccti_min) + sprintf(buff, " -ccti_min 0x%02x", _result[i].ccti_min); + else + sprintf(buff, " -ccti_min 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_ca_cong_setting_entry_list_get(_swigobj) ((ib_ca_cong_entry_t *) _swigobj->entry_list) +static int _wrap_ccCACongestionSetting_entry_list_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_ca_cong_entry_t * _result; + ccCACongestionSetting * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_entry_list_get { ccCACongestionSetting * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_entry_list_get. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_ca_cong_entry_t *)_ib_ca_cong_setting_entry_list_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <16 ; i++) { + sprintf(buff, "{SL%u:", i); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].ccti_timer) + sprintf(buff, " -ccti_timer 0x%04x", cl_ntoh16(_result[i].ccti_timer)); + else + sprintf(buff, " -ccti_timer 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].ccti_increase) + sprintf(buff, " -ccti_increase 0x%02x", _result[i].ccti_increase); + else + sprintf(buff, " -ccti_increase 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].trigger_threshold) + sprintf(buff, " -trigger_threshold 0x%02x", _result[i].trigger_threshold); + else + sprintf(buff, " -trigger_threshold 0"); + Tcl_AppendResult(interp, buff, NULL); + + if (_result[i].ccti_min) + sprintf(buff, " -ccti_min 0x%02x", _result[i].ccti_min); + else + sprintf(buff, " -ccti_min 0"); + Tcl_AppendResult(interp, buff, NULL); + + sprintf(buff, "} "); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static int ccCACongestionSetting_getByLid(ccCACongestionSetting *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCACongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CA_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccCACongestionSetting_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccCACongestionSetting * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_getByLid { ccCACongestionSetting * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_getByLid. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccCACongestionSetting_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +static int ccCACongestionSetting_setByLid(ccCACongestionSetting *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccCACongestionSetting),// mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CA_CONG_SETTING), + 0, // attribute modifier + IB_MAD_METHOD_SET); + } +static int _wrap_ccCACongestionSetting_setByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccCACongestionSetting * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccCACongestionSetting_setByLid { ccCACongestionSetting * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccCACongestionSetting_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccCACongestionSetting_setByLid. Expected _ccCACongestionSetting_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccCACongestionSetting_setByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccCACongestionSettingMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccCACongestionSetting methods : { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccCACongestionSetting_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccCACongestionSetting_getByLid; + } else if (strcmp(_str,"setByLid") == 0) { + cmd = _wrap_ccCACongestionSetting_setByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-port_control") == 0) { + cmd = _wrap_ccCACongestionSetting_port_control_set; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ccCACongestionSetting_resv0_set; + } else if (strcmp(_str,"-control_map") == 0) { + cmd = _wrap_ccCACongestionSetting_control_map_set; + } else if (strcmp(_str,"-entry_list") == 0) { + cmd = _wrap_ccCACongestionSetting_entry_list_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -port_control -resv0 -control_map -entry_list }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -port_control -resv0 -control_map -entry_list }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-port_control") == 0) { + cmd = _wrap_ccCACongestionSetting_port_control_get; + } else if (strcmp(_str,"-resv0") == 0) { + cmd = _wrap_ccCACongestionSetting_resv0_get; + } else if (strcmp(_str,"-control_map") == 0) { + cmd = _wrap_ccCACongestionSetting_control_map_get; + } else if (strcmp(_str,"-entry_list") == 0) { + cmd = _wrap_ccCACongestionSetting_entry_list_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccCACongestionSetting_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -port_control -resv0 -control_map -entry_list }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -port_control -resv0 -control_map -entry_list }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccCACongestionSetting_port_control_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-port_control ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionSetting_resv0_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv0 ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionSetting_control_map_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-control_map ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccCACongestionSetting_entry_list_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-entry_list ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccCACongestionSettingCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccCACongestionSetting * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccCACongestionSetting_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccCACongestionSetting_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccCACongestionSetting object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccCACongestionSettingMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ibcc_tbl_entry_shift_multiplier_set(_swigobj,_swigval) (_swigobj->shift_multiplier = *(_swigval),_swigval) +static int _wrap_ib_cc_tbl_entry_t_shift_multiplier_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cc_tbl_entry_t * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cc_tbl_entry_t_shift_multiplier_set { ib_cc_tbl_entry_t * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cc_tbl_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cc_tbl_entry_t_shift_multiplier_set. Expected _ib_cc_tbl_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_tbl_entry_shift_multiplier_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ibcc_tbl_entry_shift_multiplier_get(_swigobj) (&_swigobj->shift_multiplier) +static int _wrap_ib_cc_tbl_entry_t_shift_multiplier_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ib_cc_tbl_entry_t * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ib_cc_tbl_entry_t_shift_multiplier_get { ib_cc_tbl_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ib_cc_tbl_entry_t_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ib_cc_tbl_entry_t_shift_multiplier_get. Expected _ib_cc_tbl_entry_t_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ibcc_tbl_entry_shift_multiplier_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int Tclib_cc_tbl_entry_tMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ib_cc_tbl_entry_t methods : { dump cget configure }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ib_cc_tbl_entry_t_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-shift_multiplier") == 0) { + cmd = _wrap_ib_cc_tbl_entry_t_shift_multiplier_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -shift_multiplier }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -shift_multiplier }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-shift_multiplier") == 0) { + cmd = _wrap_ib_cc_tbl_entry_t_shift_multiplier_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ib_cc_tbl_entry_t_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -shift_multiplier }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -shift_multiplier }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ib_cc_tbl_entry_t_shift_multiplier_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-shift_multiplier ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int Tclib_cc_tbl_entry_tCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ib_cc_tbl_entry_t * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ib_cc_tbl_entry_t_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ib_cc_tbl_entry_t_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ib_cc_tbl_entry_t object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, Tclib_cc_tbl_entry_tMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_cc_tbl_ccti_limit_set(_swigobj,_swigval) (_swigobj->ccti_limit = *(_swigval),_swigval) +static int _wrap_ccTable_ccti_limit_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccTable * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_ccti_limit_set { ccTable * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_ccti_limit_set. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cc_tbl_ccti_limit_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cc_tbl_ccti_limit_get(_swigobj) (&_swigobj->ccti_limit) +static int _wrap_ccTable_ccti_limit_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccTable * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_ccti_limit_get { ccTable * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_ccti_limit_get. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cc_tbl_ccti_limit_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cc_tbl_resv_set(_swigobj,_swigval) (_swigobj->resv = *(_swigval),_swigval) +static int _wrap_ccTable_resv_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccTable * _arg0; + ib_net16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_resv_set { ccTable * } { ib_net16_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_resv_set. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton16(atoi(Tcl_GetStringFromObj(objv[2],NULL))); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cc_tbl_resv_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_cc_tbl_resv_get(_swigobj) (&_swigobj->resv) +static int _wrap_ccTable_resv_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net16_t * _result; + ccTable * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_resv_get { ccTable * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_resv_get. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net16_t *)_ib_cc_tbl_resv_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_hton16(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static ib_cc_tbl_entry_t * _ib_cc_tbl_entry_list_set(ccTable *obj, ib_cc_tbl_entry_t val[64]) { +{ + int i; + for (i=0; i <64 ; i++) { + obj->entry_list[i] = *(val+i); + } +} + return (ib_cc_tbl_entry_t *) val; +} +static int _wrap_ccTable_entry_list_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_cc_tbl_entry_t * _result; + ccTable * _arg0; + ib_cc_tbl_entry_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_cc_tbl_entry_t entrys[64]; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_entry_list_set { ccTable * } { ib_cc_tbl_entry_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_entry_list_set. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + long int value; + long int entry_index = 0; + int countSubLists, idx, numElements; + int i = 0; + int option = 0; + Tcl_Obj ** subListObjArray; + Tcl_Obj * tclObj; + + if (Tcl_ListObjGetElements(interp, objv[2], &countSubLists, &subListObjArray) != TCL_OK) + { + printf("Error: wrong format for CC Table: %s\n", + Tcl_GetStringFromObj(objv[2],NULL)); + return TCL_ERROR; + } + + /* + * There are two options to configure entry_list: + * 1. Configure the whole list by providing list of values: + * ccTableMad configure -entry_list {1 2 3 4 5 6} + * 2. Configure specific items from the list by providing index and value: + * ccTableMad configure -entry_list {{1 3} {2 4} {7 16}} + */ + + if (countSubLists > 0) { + + /* check how many members does the first substring have */ + + if (Tcl_ListObjLength(interp, subListObjArray[0], &numElements) != TCL_OK) { + printf("Error: wrong format for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + + if (numElements == 1) + option = 1; + else if (numElements == 2) + option = 2; + else { + printf("Error: wrong number of elements for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[0],NULL)); + return TCL_ERROR; + } + } + else { + /* if the user didn't specify anything, use option 1, + wich will effectively clear the whole list values */ + option = 1; + } + + if (option == 1) { + /* + * first option - list of values + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 1) { + printf("Error: wrong number of elements for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], 0, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + + entrys[i++].shift_multiplier = cl_hton16(value); + } + + for (; i < 64; i++) + entrys[i].shift_multiplier = 0; + } + else { + /* + * second option - index and value + */ + + for (idx = 0; idx < countSubLists; idx++) { + + if (Tcl_ListObjLength(interp, subListObjArray[idx], &numElements) != TCL_OK) { + printf("Error: wrong format for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (numElements != 2) { + printf("Error: wrong number of elements for CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], 0, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + entry_index = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + + if (Tcl_ListObjIndex(interp, subListObjArray[idx], 1, &tclObj) != TCL_OK) { + printf("Error: Fail to obtain the element of CC Table Entry: %s\n", + Tcl_GetStringFromObj(subListObjArray[idx],NULL)); + return TCL_ERROR; + } + + value = strtol(Tcl_GetStringFromObj(tclObj, NULL), NULL, 0); + + entrys[entry_index].shift_multiplier = cl_hton16(value); + } + } + + _arg1 = entrys; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_cc_tbl_entry_t *)_ib_cc_tbl_entry_list_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <64 ; i++) { + if (_result[i].shift_multiplier) + sprintf(buff, "{#%u: 0x%04x} ", i, cl_ntoh16(_result[i].shift_multiplier)); + else + sprintf(buff, "{#%u: 0} ", i); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +#define _ib_cc_tbl_entry_list_get(_swigobj) ((ib_cc_tbl_entry_t *) _swigobj->entry_list) +static int _wrap_ccTable_entry_list_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_cc_tbl_entry_t * _result; + ccTable * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_entry_list_get { ccTable * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_entry_list_get. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_cc_tbl_entry_t *)_ib_cc_tbl_entry_list_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + int i; + char buff[99]; + + sprintf(buff, "-entry_list "); + Tcl_AppendResult(interp, buff, NULL); + + for (i=0; i <64 ; i++) { + if (_result[i].shift_multiplier) + sprintf(buff, "{#%u: 0x%04x} ", i, cl_ntoh16(_result[i].shift_multiplier)); + else + sprintf(buff, "{#%u: 0} ", i); + Tcl_AppendResult(interp, buff, NULL); + } +} + return TCL_OK; +} +static int ccTable_getByLid(ccTable *self,uint16_t lid,uint8_t sn) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccTable), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CC_TBL), + sn, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccTable_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccTable * _arg0; + uint16_t * _arg1; + uint8_t * _arg2; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + uint8_t temp0; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 4) || (objc > 4)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_getByLid { ccTable * } lid sn ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_getByLid. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + temp0 = strtoul(Tcl_GetStringFromObj(objv[3],NULL), NULL, 0); + _arg2 = &temp0; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccTable_getByLid(_arg0,*_arg1,*_arg2); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +static int ccTable_setByLid(ccTable *self,uint16_t lid,uint8_t sn) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccTable), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_CC_TBL), + sn, // attribute modifier + IB_MAD_METHOD_SET); + } +static int _wrap_ccTable_setByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccTable * _arg0; + uint16_t * _arg1; + uint8_t * _arg2; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + uint8_t temp0; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 4) || (objc > 4)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTable_setByLid { ccTable * } lid sn ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTable_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTable_setByLid. Expected _ccTable_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + temp0 = strtoul(Tcl_GetStringFromObj(objv[3],NULL), NULL, 0); + _arg2 = &temp0; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccTable_setByLid(_arg0,*_arg1,*_arg2); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccTableMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccTable methods : { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccTable_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccTable_getByLid; + } else if (strcmp(_str,"setByLid") == 0) { + cmd = _wrap_ccTable_setByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-ccti_limit") == 0) { + cmd = _wrap_ccTable_ccti_limit_set; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccTable_resv_set; + } else if (strcmp(_str,"-entry_list") == 0) { + cmd = _wrap_ccTable_entry_list_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -ccti_limit -resv -entry_list }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -ccti_limit -resv -entry_list }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-ccti_limit") == 0) { + cmd = _wrap_ccTable_ccti_limit_get; + } else if (strcmp(_str,"-resv") == 0) { + cmd = _wrap_ccTable_resv_get; + } else if (strcmp(_str,"-entry_list") == 0) { + cmd = _wrap_ccTable_entry_list_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccTable_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -ccti_limit -resv -entry_list }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -ccti_limit -resv -entry_list }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccTable_ccti_limit_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-ccti_limit ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccTable_resv_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-resv ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + cmd = _wrap_ccTable_entry_list_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-entry_list ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid setByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccTableCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccTable * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccTable_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccTable_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccTable object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccTableMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + +#define _ib_time_stamp_value_set(_swigobj,_swigval) (_swigobj->value = *(_swigval),_swigval) +static int _wrap_ccTimeStamp_value_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccTimeStamp * _arg0; + ib_net32_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + ib_net32_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTimeStamp_value_set { ccTimeStamp * } { ib_net32_t * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTimeStamp_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTimeStamp_value_set. Expected _ccTimeStamp_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = cl_hton32(strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0)); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_time_stamp_value_set(_arg0,_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +#define _ib_time_stamp_value_get(_swigobj) (&_swigobj->value) +static int _wrap_ccTimeStamp_value_get(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + ib_net32_t * _result; + ccTimeStamp * _arg0; + Tcl_Obj * tcl_result; + char * rettype; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 2) || (objc > 2)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTimeStamp_value_get { ccTimeStamp * } ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTimeStamp_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTimeStamp_value_get. Expected _ccTimeStamp_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (ib_net32_t *)_ib_time_stamp_value_get(_arg0); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); +{ + char buff[20]; + sprintf(buff, "%u", cl_ntoh32(*_result)); + Tcl_SetStringObj(tcl_result,buff,strlen(buff)); +} + return TCL_OK; +} +static int ccTimeStamp_getByLid(ccTimeStamp *self,uint16_t lid) { + return ibcc_send_mad_by_lid ( + gp_ibcc, + IBCC_DEAFULT_KEY, + NULL, // log data + 0, // log data size + (uint8_t *)self, // mgt data + sizeof(ccTimeStamp), // mgt data size + lid, + CL_NTOH16(IB_MAD_ATTR_TIME_STAMP), + 0, // attribute modifier + IB_MAD_METHOD_GET); + } +static int _wrap_ccTimeStamp_getByLid(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + + int _result; + ccTimeStamp * _arg0; + uint16_t * _arg1; + Tcl_Obj * tcl_result; + char * rettype; + uint16_t temp; + + clientData = clientData; objv = objv; + tcl_result = Tcl_GetObjResult(interp); + if ((objc < 3) || (objc > 3)) { + Tcl_SetStringObj(tcl_result,"Wrong # args. ccTimeStamp_getByLid { ccTimeStamp * } lid ",-1); + return TCL_ERROR; + } + if ((rettype = SWIG_GetPointerObj(interp,objv[1],(void **) &_arg0,"_ccTimeStamp_p"))) { + Tcl_SetStringObj(tcl_result, "Type error in argument 1 of ccTimeStamp_getByLid. Expected _ccTimeStamp_p, received ", -1); + Tcl_AppendToObj(tcl_result, rettype, -1); + return TCL_ERROR; + } +{ + temp = strtoul(Tcl_GetStringFromObj(objv[2],NULL), NULL, 0); + _arg1 = &temp; +} +{ + /* we can check if IBIS was initialized here */ + if (!IbisObj.initialized) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + "ibis was not yet initialized. please use ibis_init and then ibis_set_port before.", -1); + return TCL_ERROR; + } + + if (! IbisObj.port_guid) + { + Tcl_SetStringObj( + Tcl_GetObjResult(interp), + " ibis was not yet initialized. please use ibis_set_port before.", -1); + return TCL_ERROR; + } + + ibis_tcl_error = 0; + _result = (int )ccTimeStamp_getByLid(_arg0,*_arg1); +; + if (ibis_tcl_error) { + Tcl_SetStringObj(Tcl_GetObjResult(interp), ibis_tcl_error_msg, -1); + return TCL_ERROR; + } +} tcl_result = Tcl_GetObjResult(interp); + Tcl_SetIntObj(tcl_result,(long) _result); + return TCL_OK; +} +/* methodcmd8.swg : Tcl8.x method invocation */ + +static int TclccTimeStampMethodCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST _objv[]) { + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + char *_str; + int rcode; + Tcl_Obj **objv; + Tcl_Obj *oldarg,*tcl_result,*obj; + int length; + char c; + + tcl_result = Tcl_GetObjResult(interp); + objv = (Tcl_Obj **) _objv; + if (objc < 2) { + Tcl_SetStringObj(tcl_result,"ccTimeStamp methods : { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + obj = Tcl_NewObj(); + SWIG_SetPointerObj(obj,(void *) clientData,"_ccTimeStamp_p"); + _str = Tcl_GetStringFromObj(objv[1],&length); + c = *_str; + if (0); + if (strcmp(_str,"getByLid") == 0) { + cmd = _wrap_ccTimeStamp_getByLid; + } + else if ((c == 'c') && (strncmp(_str,"configure",length) == 0) && (length >= 2)) { + int i = 2; + cmd = 0; + while (i+1 < objc) { + _str = Tcl_GetStringFromObj(objv[i],&length); + if (strcmp(_str,"-value") == 0) { + cmd = _wrap_ccTimeStamp_value_set; + } + if (cmd) { + oldarg = objv[i]; + objv[i] = obj; + rcode = (*cmd)(clientData,interp,3,&objv[i-1]); + objv[i] = oldarg; + if (rcode == TCL_ERROR) return rcode; + cmd = 0; + } else { + Tcl_SetStringObj(tcl_result,"Invalid configure option. Must be { -value }",-1); + return TCL_ERROR; + } + i+=2; + } + if ((i < objc) || (i == 2)) { + Tcl_SetStringObj(tcl_result,"{ -value }",-1); + return TCL_ERROR; + } + return TCL_OK; + } else if ((c == 'c') && (strncmp(_str,"cget",length) == 0) && (length >= 2)) { + if (objc == 3) { + _str = Tcl_GetStringFromObj(objv[2],&length); + if (0) {} + if (strcmp(_str,"-value") == 0) { + cmd = _wrap_ccTimeStamp_value_get; + } + else if (strcmp(_str,"-this") == 0) { + SWIG_SetPointerObj(tcl_result,(void *) clientData, "_ccTimeStamp_p"); + return TCL_OK; + } + if (cmd) { + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc-1,&objv[1]); + objv[2] = oldarg; + return rcode; + } else { + Tcl_SetStringObj(tcl_result,"Invalid cget option. Must be { -this -value }",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"{ -this -value }", -1); + return TCL_ERROR; + } + } else if ((c == 'd') && (strncmp(_str,"dump",length) == 0) && (length >= 2)) { + if (objc == 2) { + Tcl_Obj *pDumpObj; + pDumpObj = Tcl_NewStringObj("",-1); + Tcl_IncrRefCount(pDumpObj); + cmd = _wrap_ccTimeStamp_value_get; + oldarg = objv[2]; + objv[2] = obj; + rcode = (*cmd)(clientData,interp,objc,&objv[1]); + objv[2] = oldarg; + Tcl_AppendStringsToObj(pDumpObj, "-value ", Tcl_GetStringFromObj(tcl_result, NULL), " ", NULL); + Tcl_SetStringObj(tcl_result, Tcl_GetStringFromObj(pDumpObj, NULL), -1); + + Tcl_DecrRefCount(pDumpObj); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"no parameters are allowed for dump", -1); + return TCL_ERROR; + } + } + if (!cmd) { + Tcl_SetStringObj(tcl_result,"Invalid Method. Must be { dump cget configure getByLid }",-1); + return TCL_ERROR; + } + oldarg = objv[1]; + objv[1] = obj; + rcode = (*cmd)(clientData,interp,objc,objv); + objv[1] = oldarg; + return rcode; +} + + + +/* objcmd8.swg : Tcl 8.x object creation */ + +static int TclccTimeStampCmd(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { + void (*del)(ClientData) = 0; + char *name = 0; + int (*cmd)(ClientData, Tcl_Interp *, int, Tcl_Obj *CONST*) = 0; + ccTimeStamp * newObj = 0; + int firstarg = 0; + int thisarg = 0; + int length; + char *_str; + Tcl_Obj *tcl_result; + + tcl_result = Tcl_GetObjResult(interp); + if (objc == 1) { + cmd = 0; + } else { + _str = Tcl_GetStringFromObj(objv[1],&length); + if (strcmp(_str,"-this") == 0) thisarg = 2; + else if (strcmp(_str,"-args") == 0) { + firstarg = 1; + cmd = 0; + } else if (objc == 2) { + firstarg = 1; + name = _str; + cmd = 0; + } else if (objc >= 3) { + name = _str; + _str = Tcl_GetStringFromObj(objv[2],&length); + if (strcmp(_str,"-this") == 0) thisarg = 3; + else { + firstarg = 1; + cmd = 0; + } + } + } + if (cmd) { + int result; + result = (*cmd)(clientData,interp,objc-firstarg,&objv[firstarg]); + if (result == TCL_OK) { + SWIG_GetPointerObj(interp,tcl_result,(void **) &newObj,"_ccTimeStamp_p"); + } else { return result; } + if (!name) name = Tcl_GetStringFromObj(tcl_result,&length); + del = 0; + } else if (thisarg > 0) { + if (thisarg < objc) { + char *r; + r = SWIG_GetPointerObj(interp,objv[thisarg],(void **) &newObj,"_ccTimeStamp_p"); + if (r) { + Tcl_SetStringObj(tcl_result,"Type error. not a ccTimeStamp object.",-1); + return TCL_ERROR; + } + if (!name) name = Tcl_GetStringFromObj(objv[thisarg],&length); + Tcl_SetStringObj(tcl_result,name,-1); + } else { + Tcl_SetStringObj(tcl_result,"wrong # args.",-1); + return TCL_ERROR; + } + } else { + Tcl_SetStringObj(tcl_result,"No constructor available.",-1); + return TCL_ERROR; + } + { + Tcl_CmdInfo dummy; + if (!Tcl_GetCommandInfo(interp,name,&dummy)) { + Tcl_CreateObjCommand(interp,name, TclccTimeStampMethodCmd, (ClientData) newObj, del); + return TCL_OK; + } else { + Tcl_SetStringObj(tcl_result,"Object name already exists!",-1); + return TCL_ERROR; + } + } +} + + #define _ibis_opt_single_thread_set(_swigobj,_swigval) (_swigobj->single_thread = *(_swigval),_swigval) static int _wrap_ibis_opt_t_single_thread_set(ClientData clientData, Tcl_Interp *interp, int objc, Tcl_Obj *CONST objv[]) { @@ -57863,6 +75658,28 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { memset(&ibsm_pkey_table_obj, 0, sizeof(ib_pkey_table_t)); memset(&ibsm_sm_info_obj, 0, sizeof(ib_sm_info_t)); + /* ------------------ IBCC ---------------------- */ + + gp_ibcc = ibcc_construct(); + + if (gp_ibcc == NULL) { + printf("-E- Error from ibcc_construct.\n"); + exit(1); + } + + /* Initialize global records */ + memset(&ibcc_class_port_info_obj, 0, sizeof(ib_class_port_info_t)); + memset(&ibcc_notice_obj, 0, sizeof(ib_mad_notice_attr_t)); + memset(&ibcc_cong_info_obj, 0, sizeof(ib_cong_info_t)); + memset(&ibcc_cong_key_info_obj, 0, sizeof(ib_cong_key_info_t)); + memset(&ibcc_ca_cong_log_obj, 0, sizeof(ibcc_ca_cong_log_t)); + memset(&ibcc_sw_cong_log_obj, 0, sizeof(ibcc_sw_cong_log_t)); + memset(&ibcc_sw_cong_setting_obj, 0, sizeof(ib_sw_cong_setting_t)); + memset(&ibcc_sw_port_cong_setting_obj, 0, sizeof(ib_sw_port_cong_setting_t)); + memset(&ibcc_ca_cong_setting_obj, 0, sizeof(ib_ca_cong_setting_t)); + memset(&ibcc_table_obj, 0, sizeof(ib_cc_tbl_t)); + memset(&ibcc_time_stamp_obj, 0, sizeof(ib_time_stamp_t)); + /* ------------------ IBSAC ---------------------- */ /* Initialize global records */ @@ -57968,6 +75785,9 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { (ClientData)ibis_opt_p, 0); /* add commands for accessing the global query records */ + + /* ------------------ IBSM ---------------------- */ + Tcl_CreateObjCommand(interp,"smNodeInfoMad", TclsmNodeInfoMethodCmd, (ClientData)&ibsm_node_info_obj, 0); @@ -58016,6 +75836,54 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { TclsmNoticeMethodCmd, (ClientData)&ibsm_notice_obj, 0); + /* ------------------ IBCC ---------------------- */ + + Tcl_CreateObjCommand(interp,"ccClassPortInfoMad", + TclccClassPortInfoMethodCmd, + (ClientData)&ibcc_class_port_info_obj, 0); + + Tcl_CreateObjCommand(interp,"ccNoticeMad", + TclccNoticeMethodCmd, + (ClientData)&ibcc_notice_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCongestionInfoMad", + TclccCongestionInfoMethodCmd, + (ClientData)&ibcc_cong_info_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCongestionKeyInfoMad", + TclccCongestionKeyInfoMethodCmd, + (ClientData)&ibcc_cong_key_info_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCACongestionLogMad", + TclccCACongestionLogMethodCmd, + (ClientData)&ibcc_ca_cong_log_obj, 0); + + Tcl_CreateObjCommand(interp,"ccSWCongestionLogMad", + TclccSWCongestionLogMethodCmd, + (ClientData)&ibcc_sw_cong_log_obj, 0); + + Tcl_CreateObjCommand(interp,"ccSWCongestionSettingMad", + TclccSWCongestionSettingMethodCmd, + (ClientData)&ibcc_sw_cong_setting_obj, 0); + + Tcl_CreateObjCommand(interp,"ccSWPortCongestionSettingMad", + TclccSWPortCongestionSettingMethodCmd, + (ClientData)&ibcc_sw_port_cong_setting_obj, 0); + + Tcl_CreateObjCommand(interp,"ccCACongestionSettingMad", + TclccCACongestionSettingMethodCmd, + (ClientData)&ibcc_ca_cong_setting_obj, 0); + + Tcl_CreateObjCommand(interp,"ccTableMad", + TclccTableMethodCmd, + (ClientData)&ibcc_table_obj, 0); + + Tcl_CreateObjCommand(interp,"ccTimeStampMad", + TclccTimeStampMethodCmd, + (ClientData)&ibcc_time_stamp_obj, 0); + + /* ------------------ IBSAC --------------------- */ + Tcl_CreateObjCommand(interp,"sacNodeQuery", TclsacNodeRecMethodCmd, (ClientData)&ibsac_node_rec, 0); @@ -58825,6 +76693,237 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { Tcl_CreateObjCommand(interp, SWIG_prefix "smNotice_data_details_ntc_259_port_no_set", _wrap_smNotice_data_details_ntc_259_port_no_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); Tcl_CreateObjCommand(interp, SWIG_prefix "smNotice_data_details_ntc_259_port_no_get", _wrap_smNotice_data_details_ntc_259_port_no_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); Tcl_CreateObjCommand(interp,SWIG_prefix "smNotice_data_details_ntc_259",TclsmNotice_data_details_ntc_259Cmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_base_ver_set", _wrap_ccClassPortInfo_base_ver_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_base_ver_get", _wrap_ccClassPortInfo_base_ver_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_class_ver_set", _wrap_ccClassPortInfo_class_ver_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_class_ver_get", _wrap_ccClassPortInfo_class_ver_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_cap_mask_set", _wrap_ccClassPortInfo_cap_mask_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_cap_mask_get", _wrap_ccClassPortInfo_cap_mask_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_cap_mask2_resp_time_set", _wrap_ccClassPortInfo_cap_mask2_resp_time_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_cap_mask2_resp_time_get", _wrap_ccClassPortInfo_cap_mask2_resp_time_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_gid_set", _wrap_ccClassPortInfo_redir_gid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_gid_get", _wrap_ccClassPortInfo_redir_gid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_tc_sl_fl_set", _wrap_ccClassPortInfo_redir_tc_sl_fl_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_tc_sl_fl_get", _wrap_ccClassPortInfo_redir_tc_sl_fl_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_lid_set", _wrap_ccClassPortInfo_redir_lid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_lid_get", _wrap_ccClassPortInfo_redir_lid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_pkey_set", _wrap_ccClassPortInfo_redir_pkey_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_pkey_get", _wrap_ccClassPortInfo_redir_pkey_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_qp_set", _wrap_ccClassPortInfo_redir_qp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_qp_get", _wrap_ccClassPortInfo_redir_qp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_qkey_set", _wrap_ccClassPortInfo_redir_qkey_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_redir_qkey_get", _wrap_ccClassPortInfo_redir_qkey_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_gid_set", _wrap_ccClassPortInfo_trap_gid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_gid_get", _wrap_ccClassPortInfo_trap_gid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_tc_sl_fl_set", _wrap_ccClassPortInfo_trap_tc_sl_fl_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_tc_sl_fl_get", _wrap_ccClassPortInfo_trap_tc_sl_fl_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_lid_set", _wrap_ccClassPortInfo_trap_lid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_lid_get", _wrap_ccClassPortInfo_trap_lid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_pkey_set", _wrap_ccClassPortInfo_trap_pkey_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_pkey_get", _wrap_ccClassPortInfo_trap_pkey_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_hop_qp_set", _wrap_ccClassPortInfo_trap_hop_qp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_hop_qp_get", _wrap_ccClassPortInfo_trap_hop_qp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_qkey_set", _wrap_ccClassPortInfo_trap_qkey_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_trap_qkey_get", _wrap_ccClassPortInfo_trap_qkey_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_getByLid", _wrap_ccClassPortInfo_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccClassPortInfo_setByLid", _wrap_ccClassPortInfo_setByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccClassPortInfo",TclccClassPortInfoCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_generic_type_set", _wrap_ccNotice_generic_type_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_generic_type_get", _wrap_ccNotice_generic_type_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_issuer_lid_set", _wrap_ccNotice_issuer_lid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_issuer_lid_get", _wrap_ccNotice_issuer_lid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_toggle_count_set", _wrap_ccNotice_toggle_count_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_toggle_count_get", _wrap_ccNotice_toggle_count_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_issuer_gid_set", _wrap_ccNotice_issuer_gid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_issuer_gid_get", _wrap_ccNotice_issuer_gid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_get", _wrap_ccNotice_g_or_v_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_get", _wrap_ccNotice_data_details_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_trapByLid", _wrap_ccNotice_trapByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccNotice",TclccNoticeCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_get", _wrap_ccNotice_g_or_v_generic_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccNotice_g_or_v",TclccNotice_g_or_vCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_raw_data_get", _wrap_ccNotice_data_details_raw_data_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_get", _wrap_ccNotice_data_details_ntc_0_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccNotice_data_details",TclccNotice_data_detailsCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_prod_type_msb_set", _wrap_ccNotice_g_or_v_generic_prod_type_msb_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_prod_type_msb_get", _wrap_ccNotice_g_or_v_generic_prod_type_msb_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_prod_type_lsb_set", _wrap_ccNotice_g_or_v_generic_prod_type_lsb_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_prod_type_lsb_get", _wrap_ccNotice_g_or_v_generic_prod_type_lsb_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_trap_num_set", _wrap_ccNotice_g_or_v_generic_trap_num_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_g_or_v_generic_trap_num_get", _wrap_ccNotice_g_or_v_generic_trap_num_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccNotice_g_or_v_generic",TclccNotice_g_or_v_genericCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_raw_data_details_set", _wrap_ccNotice_data_details_raw_data_details_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_raw_data_details_get", _wrap_ccNotice_data_details_raw_data_details_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccNotice_data_details_raw_data",TclccNotice_data_details_raw_dataCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_source_lid_set", _wrap_ccNotice_data_details_ntc_0_source_lid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_source_lid_get", _wrap_ccNotice_data_details_ntc_0_source_lid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_method_set", _wrap_ccNotice_data_details_ntc_0_method_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_method_get", _wrap_ccNotice_data_details_ntc_0_method_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_resv0_set", _wrap_ccNotice_data_details_ntc_0_resv0_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_resv0_get", _wrap_ccNotice_data_details_ntc_0_resv0_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_attr_id_set", _wrap_ccNotice_data_details_ntc_0_attr_id_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_attr_id_get", _wrap_ccNotice_data_details_ntc_0_attr_id_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_resv1_set", _wrap_ccNotice_data_details_ntc_0_resv1_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_resv1_get", _wrap_ccNotice_data_details_ntc_0_resv1_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_attr_mod_set", _wrap_ccNotice_data_details_ntc_0_attr_mod_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_attr_mod_get", _wrap_ccNotice_data_details_ntc_0_attr_mod_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_qp_set", _wrap_ccNotice_data_details_ntc_0_qp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_qp_get", _wrap_ccNotice_data_details_ntc_0_qp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_cc_key_set", _wrap_ccNotice_data_details_ntc_0_cc_key_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_cc_key_get", _wrap_ccNotice_data_details_ntc_0_cc_key_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_source_gid_set", _wrap_ccNotice_data_details_ntc_0_source_gid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_source_gid_get", _wrap_ccNotice_data_details_ntc_0_source_gid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_padding_set", _wrap_ccNotice_data_details_ntc_0_padding_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccNotice_data_details_ntc_0_padding_get", _wrap_ccNotice_data_details_ntc_0_padding_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccNotice_data_details_ntc_0",TclccNotice_data_details_ntc_0Cmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_cong_info_set", _wrap_ccCongestionInfo_cong_info_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_cong_info_get", _wrap_ccCongestionInfo_cong_info_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_resv_set", _wrap_ccCongestionInfo_resv_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_resv_get", _wrap_ccCongestionInfo_resv_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_ctrl_table_cap_set", _wrap_ccCongestionInfo_ctrl_table_cap_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_ctrl_table_cap_get", _wrap_ccCongestionInfo_ctrl_table_cap_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionInfo_getByLid", _wrap_ccCongestionInfo_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccCongestionInfo",TclccCongestionInfoCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_cc_key_set", _wrap_ccCongestionKeyInfo_cc_key_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_cc_key_get", _wrap_ccCongestionKeyInfo_cc_key_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_protect_bit_set", _wrap_ccCongestionKeyInfo_protect_bit_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_protect_bit_get", _wrap_ccCongestionKeyInfo_protect_bit_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_resv_set", _wrap_ccCongestionKeyInfo_resv_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_resv_get", _wrap_ccCongestionKeyInfo_resv_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_lease_period_set", _wrap_ccCongestionKeyInfo_lease_period_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_lease_period_get", _wrap_ccCongestionKeyInfo_lease_period_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_violations_set", _wrap_ccCongestionKeyInfo_violations_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_violations_get", _wrap_ccCongestionKeyInfo_violations_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_getByLid", _wrap_ccCongestionKeyInfo_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCongestionKeyInfo_setByLid", _wrap_ccCongestionKeyInfo_setByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccCongestionKeyInfo",TclccCongestionKeyInfoCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_slid_set", _wrap_ib_cong_log_event_sw_t_slid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_slid_get", _wrap_ib_cong_log_event_sw_t_slid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_dlid_set", _wrap_ib_cong_log_event_sw_t_dlid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_dlid_get", _wrap_ib_cong_log_event_sw_t_dlid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_resv0_sl_set", _wrap_ib_cong_log_event_sw_t_resv0_sl_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_resv0_sl_get", _wrap_ib_cong_log_event_sw_t_resv0_sl_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_resv1_set", _wrap_ib_cong_log_event_sw_t_resv1_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_resv1_get", _wrap_ib_cong_log_event_sw_t_resv1_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_resv2_set", _wrap_ib_cong_log_event_sw_t_resv2_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_resv2_get", _wrap_ib_cong_log_event_sw_t_resv2_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_time_stamp_set", _wrap_ib_cong_log_event_sw_t_time_stamp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_sw_t_time_stamp_get", _wrap_ib_cong_log_event_sw_t_time_stamp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ib_cong_log_event_sw_t",Tclib_cong_log_event_sw_tCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_resv0_local_qp_set", _wrap_ib_cong_log_event_ca_t_resv0_local_qp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_resv0_local_qp_get", _wrap_ib_cong_log_event_ca_t_resv0_local_qp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_remote_qp_sl_service_type_set", _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_remote_qp_sl_service_type_get", _wrap_ib_cong_log_event_ca_t_remote_qp_sl_service_type_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_remote_lid_set", _wrap_ib_cong_log_event_ca_t_remote_lid_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_remote_lid_get", _wrap_ib_cong_log_event_ca_t_remote_lid_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_resv1_set", _wrap_ib_cong_log_event_ca_t_resv1_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_resv1_get", _wrap_ib_cong_log_event_ca_t_resv1_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_time_stamp_set", _wrap_ib_cong_log_event_ca_t_time_stamp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cong_log_event_ca_t_time_stamp_get", _wrap_ib_cong_log_event_ca_t_time_stamp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ib_cong_log_event_ca_t",Tclib_cong_log_event_ca_tCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_log_type_set", _wrap_ccSWCongestionLog_log_type_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_log_type_get", _wrap_ccSWCongestionLog_log_type_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_cong_flags_set", _wrap_ccSWCongestionLog_cong_flags_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_cong_flags_get", _wrap_ccSWCongestionLog_cong_flags_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_event_counter_set", _wrap_ccSWCongestionLog_event_counter_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_event_counter_get", _wrap_ccSWCongestionLog_event_counter_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_time_stamp_set", _wrap_ccSWCongestionLog_time_stamp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_time_stamp_get", _wrap_ccSWCongestionLog_time_stamp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_port_map_set", _wrap_ccSWCongestionLog_port_map_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_port_map_get", _wrap_ccSWCongestionLog_port_map_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_entry_list_set", _wrap_ccSWCongestionLog_entry_list_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_entry_list_get", _wrap_ccSWCongestionLog_entry_list_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionLog_getByLid", _wrap_ccSWCongestionLog_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccSWCongestionLog",TclccSWCongestionLogCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_log_type_set", _wrap_ccCACongestionLog_log_type_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_log_type_get", _wrap_ccCACongestionLog_log_type_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_cong_flags_set", _wrap_ccCACongestionLog_cong_flags_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_cong_flags_get", _wrap_ccCACongestionLog_cong_flags_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_event_counter_set", _wrap_ccCACongestionLog_event_counter_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_event_counter_get", _wrap_ccCACongestionLog_event_counter_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_event_map_set", _wrap_ccCACongestionLog_event_map_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_event_map_get", _wrap_ccCACongestionLog_event_map_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_resv_set", _wrap_ccCACongestionLog_resv_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_resv_get", _wrap_ccCACongestionLog_resv_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_time_stamp_set", _wrap_ccCACongestionLog_time_stamp_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_time_stamp_get", _wrap_ccCACongestionLog_time_stamp_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_log_event_set", _wrap_ccCACongestionLog_log_event_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_log_event_get", _wrap_ccCACongestionLog_log_event_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionLog_getByLid", _wrap_ccCACongestionLog_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccCACongestionLog",TclccCACongestionLogCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_control_map_set", _wrap_ccSWCongestionSetting_control_map_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_control_map_get", _wrap_ccSWCongestionSetting_control_map_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_victim_mask_set", _wrap_ccSWCongestionSetting_victim_mask_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_victim_mask_get", _wrap_ccSWCongestionSetting_victim_mask_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_credit_mask_set", _wrap_ccSWCongestionSetting_credit_mask_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_credit_mask_get", _wrap_ccSWCongestionSetting_credit_mask_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_threshold_set", _wrap_ccSWCongestionSetting_threshold_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_threshold_get", _wrap_ccSWCongestionSetting_threshold_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_packet_size_set", _wrap_ccSWCongestionSetting_packet_size_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_packet_size_get", _wrap_ccSWCongestionSetting_packet_size_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_cs_threshold_set", _wrap_ccSWCongestionSetting_cs_threshold_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_cs_threshold_get", _wrap_ccSWCongestionSetting_cs_threshold_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_resv0_set", _wrap_ccSWCongestionSetting_resv0_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_resv0_get", _wrap_ccSWCongestionSetting_resv0_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_cs_return_delay_set", _wrap_ccSWCongestionSetting_cs_return_delay_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_cs_return_delay_get", _wrap_ccSWCongestionSetting_cs_return_delay_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_marking_rate_set", _wrap_ccSWCongestionSetting_marking_rate_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_marking_rate_get", _wrap_ccSWCongestionSetting_marking_rate_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_getByLid", _wrap_ccSWCongestionSetting_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWCongestionSetting_setByLid", _wrap_ccSWCongestionSetting_setByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccSWCongestionSetting",TclccSWCongestionSettingCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_set", _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get", _wrap_ib_sw_port_cong_setting_element_t_valid_ctrl_type_res_threshold_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_sw_port_cong_setting_element_t_packet_size_set", _wrap_ib_sw_port_cong_setting_element_t_packet_size_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_sw_port_cong_setting_element_t_packet_size_get", _wrap_ib_sw_port_cong_setting_element_t_packet_size_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_sw_port_cong_setting_element_t_cong_param_set", _wrap_ib_sw_port_cong_setting_element_t_cong_param_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_sw_port_cong_setting_element_t_cong_param_get", _wrap_ib_sw_port_cong_setting_element_t_cong_param_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ib_sw_port_cong_setting_element_t",Tclib_sw_port_cong_setting_element_tCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWPortCongestionSetting_block_set", _wrap_ccSWPortCongestionSetting_block_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWPortCongestionSetting_block_get", _wrap_ccSWPortCongestionSetting_block_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWPortCongestionSetting_getByLid", _wrap_ccSWPortCongestionSetting_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccSWPortCongestionSetting_setByLid", _wrap_ccSWPortCongestionSetting_setByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccSWPortCongestionSetting",TclccSWPortCongestionSettingCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_ccti_timer_set", _wrap_ib_ca_cong_entry_t_ccti_timer_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_ccti_timer_get", _wrap_ib_ca_cong_entry_t_ccti_timer_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_ccti_increase_set", _wrap_ib_ca_cong_entry_t_ccti_increase_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_ccti_increase_get", _wrap_ib_ca_cong_entry_t_ccti_increase_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_trigger_threshold_set", _wrap_ib_ca_cong_entry_t_trigger_threshold_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_trigger_threshold_get", _wrap_ib_ca_cong_entry_t_trigger_threshold_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_ccti_min_set", _wrap_ib_ca_cong_entry_t_ccti_min_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_ccti_min_get", _wrap_ib_ca_cong_entry_t_ccti_min_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_resv0_set", _wrap_ib_ca_cong_entry_t_resv0_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_resv0_get", _wrap_ib_ca_cong_entry_t_resv0_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_resv1_set", _wrap_ib_ca_cong_entry_t_resv1_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_ca_cong_entry_t_resv1_get", _wrap_ib_ca_cong_entry_t_resv1_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ib_ca_cong_entry_t",Tclib_ca_cong_entry_tCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_port_control_set", _wrap_ccCACongestionSetting_port_control_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_port_control_get", _wrap_ccCACongestionSetting_port_control_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_resv0_set", _wrap_ccCACongestionSetting_resv0_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_resv0_get", _wrap_ccCACongestionSetting_resv0_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_control_map_set", _wrap_ccCACongestionSetting_control_map_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_control_map_get", _wrap_ccCACongestionSetting_control_map_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_entry_list_set", _wrap_ccCACongestionSetting_entry_list_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_entry_list_get", _wrap_ccCACongestionSetting_entry_list_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_getByLid", _wrap_ccCACongestionSetting_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccCACongestionSetting_setByLid", _wrap_ccCACongestionSetting_setByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccCACongestionSetting",TclccCACongestionSettingCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cc_tbl_entry_t_shift_multiplier_set", _wrap_ib_cc_tbl_entry_t_shift_multiplier_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ib_cc_tbl_entry_t_shift_multiplier_get", _wrap_ib_cc_tbl_entry_t_shift_multiplier_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ib_cc_tbl_entry_t",Tclib_cc_tbl_entry_tCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_ccti_limit_set", _wrap_ccTable_ccti_limit_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_ccti_limit_get", _wrap_ccTable_ccti_limit_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_resv_set", _wrap_ccTable_resv_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_resv_get", _wrap_ccTable_resv_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_entry_list_set", _wrap_ccTable_entry_list_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_entry_list_get", _wrap_ccTable_entry_list_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_getByLid", _wrap_ccTable_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTable_setByLid", _wrap_ccTable_setByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccTable",TclccTableCmd, (ClientData) NULL, NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTimeStamp_value_set", _wrap_ccTimeStamp_value_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTimeStamp_value_get", _wrap_ccTimeStamp_value_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp, SWIG_prefix "ccTimeStamp_getByLid", _wrap_ccTimeStamp_getByLid, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); + Tcl_CreateObjCommand(interp,SWIG_prefix "ccTimeStamp",TclccTimeStampCmd, (ClientData) NULL, NULL); Tcl_CreateObjCommand(interp, SWIG_prefix "ibis_opt_t_single_thread_set", _wrap_ibis_opt_t_single_thread_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); Tcl_CreateObjCommand(interp, SWIG_prefix "ibis_opt_t_single_thread_get", _wrap_ibis_opt_t_single_thread_get, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); Tcl_CreateObjCommand(interp, SWIG_prefix "ibis_opt_t_force_log_flush_set", _wrap_ibis_opt_t_force_log_flush_set, (ClientData) NULL, (Tcl_CmdDeleteProc *) NULL); @@ -58846,6 +76945,10 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { */ SWIG_RegisterMapping("_struct__ibsm_sm_info","_smSMInfo",0); SWIG_RegisterMapping("_struct__ibsm_sm_info","__ibsm_sm_info",0); + SWIG_RegisterMapping("__ibcc_class_port_info","_ccClassPortInfo",0); + SWIG_RegisterMapping("__ibcc_class_port_info","_struct__ibcc_class_port_info",0); + SWIG_RegisterMapping("_struct__ibcc_notice","_ccNotice",0); + SWIG_RegisterMapping("_struct__ibcc_notice","__ibcc_notice",0); SWIG_RegisterMapping("_signed_long","_long",0); SWIG_RegisterMapping("_struct__ibsm_vl_arb_table","_smVlArbTable",0); SWIG_RegisterMapping("_struct__ibsm_vl_arb_table","__ibsm_vl_arb_table",0); @@ -58855,12 +76958,18 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("_sacPortInfo","__ib_port_info",0); SWIG_RegisterMapping("_struct__ib_pkey_table","_sacPKeyTbl",0); SWIG_RegisterMapping("_struct__ib_pkey_table","__ib_pkey_table",0); + SWIG_RegisterMapping("_ccTimeStamp","_struct__ib_time_stamp",0); + SWIG_RegisterMapping("_ccTimeStamp","__ib_time_stamp",0); SWIG_RegisterMapping("_sacNodeInfo","_struct__ib_node_info",0); SWIG_RegisterMapping("_sacNodeInfo","__ib_node_info",0); SWIG_RegisterMapping("_struct__ib_portinfo_record","_sacPortRec",0); SWIG_RegisterMapping("_struct__ib_portinfo_record","__ib_portinfo_record",0); + SWIG_RegisterMapping("__ibcc_notice","_ccNotice",0); + SWIG_RegisterMapping("__ibcc_notice","_struct__ibcc_notice",0); SWIG_RegisterMapping("_struct__ib_port_info","_sacPortInfo",0); SWIG_RegisterMapping("_struct__ib_port_info","__ib_port_info",0); + SWIG_RegisterMapping("_struct__ib_cong_log_event_ca","_ib_cong_log_event_ca_t",0); + SWIG_RegisterMapping("_struct__ib_cong_log_event_ca","__ib_cong_log_event_ca",0); SWIG_RegisterMapping("_struct__ib_switch_info","_sacSwInfo",0); SWIG_RegisterMapping("_struct__ib_switch_info","__ib_switch_info",0); SWIG_RegisterMapping("_sacNodeRec","_struct__ib_node_record_t",0); @@ -58869,14 +76978,22 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("__ib_switch_info","_struct__ib_switch_info",0); SWIG_RegisterMapping("_struct__ibsm_mft_block","_smMftBlock",0); SWIG_RegisterMapping("_struct__ibsm_mft_block","__ibsm_mft_block",0); + SWIG_RegisterMapping("_struct__ibcc_class_port_info","_ccClassPortInfo",0); + SWIG_RegisterMapping("_struct__ibcc_class_port_info","__ibcc_class_port_info",0); SWIG_RegisterMapping("__ibsm_mft_block","_smMftBlock",0); SWIG_RegisterMapping("__ibsm_mft_block","_struct__ibsm_mft_block",0); + SWIG_RegisterMapping("_ib_cong_log_event_sw_t","_struct__ib_cong_log_event_sw",0); + SWIG_RegisterMapping("_ib_cong_log_event_sw_t","__ib_cong_log_event_sw",0); SWIG_RegisterMapping("_long","_unsigned_long",0); SWIG_RegisterMapping("_long","_signed_long",0); + SWIG_RegisterMapping("_ib_cong_log_event_ca_t","_struct__ib_cong_log_event_ca",0); + SWIG_RegisterMapping("_ib_cong_log_event_ca_t","__ib_cong_log_event_ca",0); SWIG_RegisterMapping("_sacSlVlTbl","_struct__ib_slvl_table",0); SWIG_RegisterMapping("_sacSlVlTbl","__ib_slvl_table",0); SWIG_RegisterMapping("_smNotice","_struct__ibsm_notice",0); SWIG_RegisterMapping("_smNotice","__ibsm_notice",0); + SWIG_RegisterMapping("_struct__ib_sw_cong_setting","_ccSWCongestionSetting",0); + SWIG_RegisterMapping("_struct__ib_sw_cong_setting","__ib_sw_cong_setting",0); SWIG_RegisterMapping("_struct__ibsm_port_info","_smPortInfo",0); SWIG_RegisterMapping("_struct__ibsm_port_info","__ibsm_port_info",0); SWIG_RegisterMapping("_struct__ib_node_info","_sacNodeInfo",0); @@ -58905,26 +77022,56 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("__ibsm_switch_info","_struct__ibsm_switch_info",0); SWIG_RegisterMapping("__ibsm_node_info","_smNodeInfo",0); SWIG_RegisterMapping("__ibsm_node_info","_struct__ibsm_node_info",0); + SWIG_RegisterMapping("_ccSWCongestionSetting","_struct__ib_sw_cong_setting",0); + SWIG_RegisterMapping("_ccSWCongestionSetting","__ib_sw_cong_setting",0); SWIG_RegisterMapping("__ib_member_rec","_sacMCMRec",0); SWIG_RegisterMapping("__ib_member_rec","_struct__ib_member_rec",0); + SWIG_RegisterMapping("__ib_sw_port_cong_setting","_ccSWPortCongestionSetting",0); + SWIG_RegisterMapping("__ib_sw_port_cong_setting","_struct__ib_sw_port_cong_setting",0); SWIG_RegisterMapping("__ibis_opt","_ibis_opt_t",0); SWIG_RegisterMapping("__ibis_opt","_struct__ibis_opt",0); + SWIG_RegisterMapping("_struct__ib_time_stamp","_ccTimeStamp",0); + SWIG_RegisterMapping("_struct__ib_time_stamp","__ib_time_stamp",0); + SWIG_RegisterMapping("__ib_sw_cong_setting","_ccSWCongestionSetting",0); + SWIG_RegisterMapping("__ib_sw_cong_setting","_struct__ib_sw_cong_setting",0); SWIG_RegisterMapping("_sacServiceRec","_struct__ib_service_record",0); SWIG_RegisterMapping("_sacServiceRec","__ib_service_record",0); + SWIG_RegisterMapping("_ib_cc_tbl_entry_t","_struct__ibcc_tbl_entry",0); + SWIG_RegisterMapping("_ib_cc_tbl_entry_t","__ibcc_tbl_entry",0); + SWIG_RegisterMapping("_struct__ib_sw_cong_log","_ccSWCongestionLog",0); + SWIG_RegisterMapping("_struct__ib_sw_cong_log","__ib_sw_cong_log",0); + SWIG_RegisterMapping("__ib_sw_cong_log","_ccSWCongestionLog",0); + SWIG_RegisterMapping("__ib_sw_cong_log","_struct__ib_sw_cong_log",0); SWIG_RegisterMapping("_struct__ib_path_rec","_sacPathRec",0); SWIG_RegisterMapping("_struct__ib_path_rec","__ib_path_rec",0); + SWIG_RegisterMapping("_struct__ib_ca_cong_log","_ccCACongestionLog",0); + SWIG_RegisterMapping("_struct__ib_ca_cong_log","__ib_ca_cong_log",0); + SWIG_RegisterMapping("__ib_cong_key_info","_ccCongestionKeyInfo",0); + SWIG_RegisterMapping("__ib_cong_key_info","_struct__ib_cong_key_info",0); + SWIG_RegisterMapping("__ib_ca_cong_log","_ccCACongestionLog",0); + SWIG_RegisterMapping("__ib_ca_cong_log","_struct__ib_ca_cong_log",0); SWIG_RegisterMapping("_struct__ibsac_class_port_info","_sacClassPortInfo",0); SWIG_RegisterMapping("_struct__ibsac_class_port_info","__ibsac_class_port_info",0); SWIG_RegisterMapping("_struct__ibsm_node_desc","_smNodeDesc",0); SWIG_RegisterMapping("_struct__ibsm_node_desc","__ibsm_node_desc",0); + SWIG_RegisterMapping("_struct__ib_sw_port_cong_setting_element","_ib_sw_port_cong_setting_element_t",0); + SWIG_RegisterMapping("_struct__ib_sw_port_cong_setting_element","__ib_sw_port_cong_setting_element",0); SWIG_RegisterMapping("_sacSmInfo","_struct__ib_sm_info",0); SWIG_RegisterMapping("_sacSmInfo","__ib_sm_info",0); SWIG_RegisterMapping("__ibsm_node_desc","_smNodeDesc",0); SWIG_RegisterMapping("__ibsm_node_desc","_struct__ibsm_node_desc",0); SWIG_RegisterMapping("__ibsac_class_port_info","_sacClassPortInfo",0); SWIG_RegisterMapping("__ibsac_class_port_info","_struct__ibsac_class_port_info",0); + SWIG_RegisterMapping("__ib_cong_log_event_sw","_ib_cong_log_event_sw_t",0); + SWIG_RegisterMapping("__ib_cong_log_event_sw","_struct__ib_cong_log_event_sw",0); + SWIG_RegisterMapping("_ccCongestionKeyInfo","_struct__ib_cong_key_info",0); + SWIG_RegisterMapping("_ccCongestionKeyInfo","__ib_cong_key_info",0); SWIG_RegisterMapping("__ib_path_rec","_sacPathRec",0); SWIG_RegisterMapping("__ib_path_rec","_struct__ib_path_rec",0); + SWIG_RegisterMapping("_struct__ib_ca_cong_setting","_ccCACongestionSetting",0); + SWIG_RegisterMapping("_struct__ib_ca_cong_setting","__ib_ca_cong_setting",0); + SWIG_RegisterMapping("__ib_cong_info","_ccCongestionInfo",0); + SWIG_RegisterMapping("__ib_cong_info","_struct__ib_cong_info",0); SWIG_RegisterMapping("__ib_slvl_table","_sacSlVlTbl",0); SWIG_RegisterMapping("__ib_slvl_table","_struct__ib_slvl_table",0); SWIG_RegisterMapping("_unsigned_long","_long",0); @@ -58932,8 +77079,14 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("_smPkeyTable","__ibsm_pkey_table",0); SWIG_RegisterMapping("_struct__ib_lft_record","_sacLFTRec",0); SWIG_RegisterMapping("_struct__ib_lft_record","__ib_lft_record",0); + SWIG_RegisterMapping("_struct__ibcc_tbl_entry","_ib_cc_tbl_entry_t",0); + SWIG_RegisterMapping("_struct__ibcc_tbl_entry","__ibcc_tbl_entry",0); SWIG_RegisterMapping("_sacPathRec","_struct__ib_path_rec",0); SWIG_RegisterMapping("_sacPathRec","__ib_path_rec",0); + SWIG_RegisterMapping("_struct__ib_cong_log_event_sw","_ib_cong_log_event_sw_t",0); + SWIG_RegisterMapping("_struct__ib_cong_log_event_sw","__ib_cong_log_event_sw",0); + SWIG_RegisterMapping("__ibcc_tbl_entry","_ib_cc_tbl_entry_t",0); + SWIG_RegisterMapping("__ibcc_tbl_entry","_struct__ibcc_tbl_entry",0); SWIG_RegisterMapping("_smPortInfo","_struct__ibsm_port_info",0); SWIG_RegisterMapping("_smPortInfo","__ibsm_port_info",0); SWIG_RegisterMapping("_smNodeInfo","_struct__ibsm_node_info",0); @@ -58943,16 +77096,24 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("_signed_int","_int",0); SWIG_RegisterMapping("_struct__ibsm_guid_info","_smGuidInfo",0); SWIG_RegisterMapping("_struct__ibsm_guid_info","__ibsm_guid_info",0); + SWIG_RegisterMapping("_ccSWCongestionLog","_struct__ib_sw_cong_log",0); + SWIG_RegisterMapping("_ccSWCongestionLog","__ib_sw_cong_log",0); SWIG_RegisterMapping("__ibsm_guid_info","_smGuidInfo",0); SWIG_RegisterMapping("__ibsm_guid_info","_struct__ibsm_guid_info",0); SWIG_RegisterMapping("_struct__ib_node_record_t","_sacNodeRec",0); SWIG_RegisterMapping("_struct__ib_node_record_t","__ib_node_record_t",0); SWIG_RegisterMapping("__ib_slvl_table_record","_sacSlVlRec",0); SWIG_RegisterMapping("__ib_slvl_table_record","_struct__ib_slvl_table_record",0); + SWIG_RegisterMapping("_ccClassPortInfo","_struct__ibcc_class_port_info",0); + SWIG_RegisterMapping("_ccClassPortInfo","__ibcc_class_port_info",0); + SWIG_RegisterMapping("_ccCACongestionSetting","_struct__ib_ca_cong_setting",0); + SWIG_RegisterMapping("_ccCACongestionSetting","__ib_ca_cong_setting",0); SWIG_RegisterMapping("_sacInformInfo","_struct__ibsac_inform_info",0); SWIG_RegisterMapping("_sacInformInfo","__ibsac_inform_info",0); SWIG_RegisterMapping("_struct__ibsm_switch_info","_smSwInfo",0); SWIG_RegisterMapping("_struct__ibsm_switch_info","__ibsm_switch_info",0); + SWIG_RegisterMapping("__ib_ca_cong_setting","_ccCACongestionSetting",0); + SWIG_RegisterMapping("__ib_ca_cong_setting","_struct__ib_ca_cong_setting",0); SWIG_RegisterMapping("_unsigned_short","_short",0); SWIG_RegisterMapping("_smSwInfo","_struct__ibsm_switch_info",0); SWIG_RegisterMapping("_smSwInfo","__ibsm_switch_info",0); @@ -58975,12 +77136,16 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("__ib_pkey_table_record","_struct__ib_pkey_table_record",0); SWIG_RegisterMapping("__ib_sminfo_record","_sacSmRec",0); SWIG_RegisterMapping("__ib_sminfo_record","_struct__ib_sminfo_record",0); + SWIG_RegisterMapping("__ib_ca_cong_entry","_ib_ca_cong_entry_t",0); + SWIG_RegisterMapping("__ib_ca_cong_entry","_struct__ib_ca_cong_entry",0); SWIG_RegisterMapping("_sacPortRec","_struct__ib_portinfo_record",0); SWIG_RegisterMapping("_sacPortRec","__ib_portinfo_record",0); SWIG_RegisterMapping("_struct__ibsm_pkey_table","_smPkeyTable",0); SWIG_RegisterMapping("_struct__ibsm_pkey_table","__ibsm_pkey_table",0); SWIG_RegisterMapping("__ibsm_notice","_smNotice",0); SWIG_RegisterMapping("__ibsm_notice","_struct__ibsm_notice",0); + SWIG_RegisterMapping("_struct__ib_cong_key_info","_ccCongestionKeyInfo",0); + SWIG_RegisterMapping("_struct__ib_cong_key_info","__ib_cong_key_info",0); SWIG_RegisterMapping("_unsigned_int","_int",0); SWIG_RegisterMapping("__ibsm_slvl_table","_smSlVlTable",0); SWIG_RegisterMapping("__ibsm_slvl_table","_struct__ibsm_slvl_table",0); @@ -58988,6 +77153,8 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("_short","_signed_short",0); SWIG_RegisterMapping("__ib_port_info","_sacPortInfo",0); SWIG_RegisterMapping("__ib_port_info","_struct__ib_port_info",0); + SWIG_RegisterMapping("_ccCongestionInfo","_struct__ib_cong_info",0); + SWIG_RegisterMapping("_ccCongestionInfo","__ib_cong_info",0); SWIG_RegisterMapping("_sacSmRec","_struct__ib_sminfo_record",0); SWIG_RegisterMapping("_sacSmRec","__ib_sminfo_record",0); SWIG_RegisterMapping("__ib_service_record","_sacServiceRec",0); @@ -58998,6 +77165,10 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("_struct__ibsac_inform_info","__ibsac_inform_info",0); SWIG_RegisterMapping("_struct__ib_vl_arb_table_record","_sacVlArbRec",0); SWIG_RegisterMapping("_struct__ib_vl_arb_table_record","__ib_vl_arb_table_record",0); + SWIG_RegisterMapping("_ccSWPortCongestionSetting","_struct__ib_sw_port_cong_setting",0); + SWIG_RegisterMapping("_ccSWPortCongestionSetting","__ib_sw_port_cong_setting",0); + SWIG_RegisterMapping("__ib_time_stamp","_ccTimeStamp",0); + SWIG_RegisterMapping("__ib_time_stamp","_struct__ib_time_stamp",0); SWIG_RegisterMapping("_sacMCMRec","_struct__ib_member_rec",0); SWIG_RegisterMapping("_sacMCMRec","__ib_member_rec",0); SWIG_RegisterMapping("_struct__ibis_opt","_ibis_opt_t",0); @@ -59008,16 +77179,26 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("__ibsm_pkey_table","_struct__ibsm_pkey_table",0); SWIG_RegisterMapping("_int","_unsigned_int",0); SWIG_RegisterMapping("_int","_signed_int",0); + SWIG_RegisterMapping("_ib_ca_cong_entry_t","_struct__ib_ca_cong_entry",0); + SWIG_RegisterMapping("_ib_ca_cong_entry_t","__ib_ca_cong_entry",0); SWIG_RegisterMapping("_struct__ib_pkey_table_record","_sacPKeyRec",0); SWIG_RegisterMapping("_struct__ib_pkey_table_record","__ib_pkey_table_record",0); SWIG_RegisterMapping("__ib_node_info","_sacNodeInfo",0); SWIG_RegisterMapping("__ib_node_info","_struct__ib_node_info",0); + SWIG_RegisterMapping("_ccCACongestionLog","_struct__ib_ca_cong_log",0); + SWIG_RegisterMapping("_ccCACongestionLog","__ib_ca_cong_log",0); SWIG_RegisterMapping("_sacLFTRec","_struct__ib_lft_record",0); SWIG_RegisterMapping("_sacLFTRec","__ib_lft_record",0); SWIG_RegisterMapping("_struct__ib_member_rec","_sacMCMRec",0); SWIG_RegisterMapping("_struct__ib_member_rec","__ib_member_rec",0); + SWIG_RegisterMapping("__ib_cc_tbl","_ccTable",0); + SWIG_RegisterMapping("__ib_cc_tbl","_struct__ib_cc_tbl",0); + SWIG_RegisterMapping("_struct__ib_cong_info","_ccCongestionInfo",0); + SWIG_RegisterMapping("_struct__ib_cong_info","__ib_cong_info",0); SWIG_RegisterMapping("__ibsac_inform_info","_sacInformInfo",0); SWIG_RegisterMapping("__ibsac_inform_info","_struct__ibsac_inform_info",0); + SWIG_RegisterMapping("_ccTable","_struct__ib_cc_tbl",0); + SWIG_RegisterMapping("_ccTable","__ib_cc_tbl",0); SWIG_RegisterMapping("_struct__ib_switch_info_record","_sacSwRec",0); SWIG_RegisterMapping("_struct__ib_switch_info_record","__ib_switch_info_record",0); SWIG_RegisterMapping("_sacVlArbRec","_struct__ib_vl_arb_table_record",0); @@ -59028,6 +77209,10 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("__ib_switch_info_record","_struct__ib_switch_info_record",0); SWIG_RegisterMapping("__ib_vl_arb_table_record","_sacVlArbRec",0); SWIG_RegisterMapping("__ib_vl_arb_table_record","_struct__ib_vl_arb_table_record",0); + SWIG_RegisterMapping("__ib_sw_port_cong_setting_element","_ib_sw_port_cong_setting_element_t",0); + SWIG_RegisterMapping("__ib_sw_port_cong_setting_element","_struct__ib_sw_port_cong_setting_element",0); + SWIG_RegisterMapping("_struct__ib_sw_port_cong_setting","_ccSWPortCongestionSetting",0); + SWIG_RegisterMapping("_struct__ib_sw_port_cong_setting","__ib_sw_port_cong_setting",0); SWIG_RegisterMapping("__ib_lft_record","_sacLFTRec",0); SWIG_RegisterMapping("__ib_lft_record","_struct__ib_lft_record",0); SWIG_RegisterMapping("_smSMInfo","_struct__ibsm_sm_info",0); @@ -59040,16 +77225,26 @@ SWIGEXPORT(int,Ibis_Init)(Tcl_Interp *interp) { SWIG_RegisterMapping("_smMftBlock","__ibsm_mft_block",0); SWIG_RegisterMapping("_sacSwRec","_struct__ib_switch_info_record",0); SWIG_RegisterMapping("_sacSwRec","__ib_switch_info_record",0); + SWIG_RegisterMapping("_struct__ib_cc_tbl","_ccTable",0); + SWIG_RegisterMapping("_struct__ib_cc_tbl","__ib_cc_tbl",0); SWIG_RegisterMapping("_struct__ib_link_record","_sacLinkRec",0); SWIG_RegisterMapping("_struct__ib_link_record","__ib_link_record",0); SWIG_RegisterMapping("_struct__ibsm_lft_block","_smLftBlock",0); SWIG_RegisterMapping("_struct__ibsm_lft_block","__ibsm_lft_block",0); + SWIG_RegisterMapping("_ccNotice","_struct__ibcc_notice",0); + SWIG_RegisterMapping("_ccNotice","__ibcc_notice",0); + SWIG_RegisterMapping("_ib_sw_port_cong_setting_element_t","_struct__ib_sw_port_cong_setting_element",0); + SWIG_RegisterMapping("_ib_sw_port_cong_setting_element_t","__ib_sw_port_cong_setting_element",0); SWIG_RegisterMapping("_struct__ib_sminfo_record","_sacSmRec",0); SWIG_RegisterMapping("_struct__ib_sminfo_record","__ib_sminfo_record",0); SWIG_RegisterMapping("__ib_link_record","_sacLinkRec",0); SWIG_RegisterMapping("__ib_link_record","_struct__ib_link_record",0); SWIG_RegisterMapping("__ibsm_lft_block","_smLftBlock",0); SWIG_RegisterMapping("__ibsm_lft_block","_struct__ibsm_lft_block",0); + SWIG_RegisterMapping("__ib_cong_log_event_ca","_ib_cong_log_event_ca_t",0); + SWIG_RegisterMapping("__ib_cong_log_event_ca","_struct__ib_cong_log_event_ca",0); + SWIG_RegisterMapping("_struct__ib_ca_cong_entry","_ib_ca_cong_entry_t",0); + SWIG_RegisterMapping("_struct__ib_ca_cong_entry","__ib_ca_cong_entry",0); SWIG_RegisterMapping("_sacPKeyRec","_struct__ib_pkey_table_record",0); SWIG_RegisterMapping("_sacPKeyRec","__ib_pkey_table_record",0); SWIG_RegisterMapping("_struct__ib_slvl_table","_sacSlVlTbl",0); -- 1.5.1.4 From kliteyn at dev.mellanox.co.il Mon Jun 30 01:52:33 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 11:52:33 +0300 Subject: [ofa-general] Re: [PATCH v2] [TRIVIAL] opensm/osm_state_mgr.c: fixing some data types In-Reply-To: <20080630072954.GL12659@sashak.voltaire.com> References: <486881E1.1090603@dev.mellanox.co.il> <20080630072954.GL12659@sashak.voltaire.com> Message-ID: <48689ED1.1000907@dev.mellanox.co.il> Sasha Khapyorsky wrote: > On 09:49 Mon 30 Jun , Yevgeny Kliteynik wrote: >> Fixing some data types and solving compiler warnings >> in the corresponding log messages. >> >> Signed-off-by: Yevgeny Kliteynik >> --- >> opensm/opensm/osm_state_mgr.c | 2 +- >> 1 files changed, 1 insertions(+), 1 deletions(-) >> >> diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c >> index 9a8409b..734df79 100644 >> --- a/opensm/opensm/osm_state_mgr.c >> +++ b/opensm/opensm/osm_state_mgr.c >> @@ -862,7 +862,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) >> osm_port_t *p_port; >> osm_port_t *p_next_port; >> cl_ptr_vector_t *p_port_lid_tbl; >> - size_t max_lid, ref_size, curr_size, lid; >> + uint16_t max_lid, ref_size, curr_size, lid; > > Those values are used for storing result of cl_ptr_vector_get_size() > which has size_t type. cl_ptr_vector_get_size() returns uint32_t, and the value is lid, which means that the vector can't be longer than uint16_t. > I think the proper solution for this is to use > '%zu' format in OSM_LOG(). OK, patch shortly. -- Yevgeny > Sasha > From kliteyn at dev.mellanox.co.il Mon Jun 30 01:52:01 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 11:52:01 +0300 Subject: [ofa-general] [PATCH v3] [TRIVIAL] opensm/osm_state_mgr.c: fixing log message Message-ID: <48689EB1.7080406@dev.mellanox.co.il> Hi Sasha, Fixing some log messages (was "fixing some data types") in osm_state_mgr.c to remove compiler warnings. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_state_mgr.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index 9a8409b..87d6d5b 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -920,7 +920,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * with the new lid we wanted to give it in our * port_lid_tbl. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3322: " - "lid %u is wrongly assigned to port 0x%016" + "lid %zu is wrongly assigned to port 0x%016" PRIx64 " (\'%s\' port %u) in port_lid_tbl\n", lid, cl_ntoh64(osm_port_get_guid(p_port_stored)), @@ -933,7 +933,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * original lid. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3323: " "port 0x%016" PRIx64 " (\'%s\' port %u)" - " exists in new port_lid_tbl under lid %u," + " exists in new port_lid_tbl under lid %zu," " but missing in subnet port_lid_tbl db\n", cl_ntoh64(osm_port_get_guid(p_port_ref)), p_port_ref->p_node->print_desc, @@ -944,7 +944,7 @@ static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) * it didn't reach it, and p_port_ref also didn't get * the lid update. */ OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3324: " - "lid %u has port 0x%016" PRIx64 + "lid %zu has port 0x%016" PRIx64 " (\'%s\' port %u) in new port_lid_tbl db, " "and port 0x%016" PRIx64 " (\'%s\' port %u)" " in subnet port_lid_tbl db\n", lid, -- 1.5.1.4 From sashak at voltaire.com Mon Jun 30 02:00:14 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 12:00:14 +0300 Subject: [ofa-general] Re: [PATCH] opensm/include/iba/ib_types.h: Definition of Congestion Control MADs In-Reply-To: <48688300.5050902@dev.mellanox.co.il> References: <48688300.5050902@dev.mellanox.co.il> Message-ID: <20080630090014.GN12659@sashak.voltaire.com> Hi Yevgeny, On 09:53 Mon 30 Jun , Yevgeny Kliteynik wrote: > @@ -7126,6 +7250,19 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated > uint8_t details[54]; > } PACK_SUFFIX raw_data; > > + struct _ntc_0 { // CC_Key violation [54 bytes] > + ib_net16_t source_lid; // Source LID from offending packet LRH > + uint8_t method; // Method, from common MAD header > + uint8_t resv0; > + ib_net16_t attr_id; // Attribute ID, from common MAD header > + ib_net16_t resv1; > + ib_net32_t attr_mod; // Attribute Modif, from common MAD header > + ib_net32_t qp; // 8b pad, 24b dest QP from BTH Looked at the spec (1.2.1). Should not here be: ib_net16_t attr_id; ib_net32_t attr_mode; ib_net32_t qp; ? > + ib_net64_t cc_key; // CC key of the offending packet > + ib_gid_t source_gid; // GID from GRH of the offending packet > + uint8_t padding[14]; // Padding - ignored on read And uint8_t padding[24]; here? > + } PACK_SUFFIX ntc_0; > + > struct _ntc_64_67 { > uint8_t res[6]; > ib_gid_t gid; // the Node or Multicast Group that came in/out [snip...] > +/****s* IBA Base: Types/ib_cong_info_t > +* NAME > +* ib_cong_info_t > +* > +* DESCRIPTION > +* IBA defined CongestionInfo attribute (A10.4.3.3) > +* > +* SYNOPSIS > +*/ > +#include > +typedef struct _ib_cong_info { > + uint8_t cong_info; > + uint8_t resv; It is probably better to make it ib_net16_t cong_info; , so we will not need to rework later. Bits values we can define in network byte order. > + uint8_t ctrl_table_cap; > +} PACK_SUFFIX ib_cong_info_t; > +#include > +/* > +* FIELDS > +* cong_info > +* Congestion control capabilities of the node. > +* > +* ctrl_table_cap > +* Number of 64 entry blocks in the CongestionControlTable. > +* > +* SEE ALSO > +* ib_cc_mad_t > +*********/ [snip...] > +/****s* IBA Base: Types/ib_cong_log_event_sw_t > +* NAME > +* ib_cong_log_event_sw_t > +* > +* DESCRIPTION > +* IBA defined CongestionLogEvent (SW) entry (A10.4.3.5) > +* > +* SYNOPSIS > +*/ > +#include > +typedef struct _ib_cong_log_event_sw { > + ib_net16_t slid; > + ib_net16_t dlid; > + uint8_t resv0_sl; > + uint8_t resv1; > + ib_net16_t resv2; > + ib_net32_t time_stamp; > +} PACK_SUFFIX ib_cong_log_event_sw_t; > +#include > +/* > +* FIELDS > +* slid > +* Source LID of congestion event. > +* > +* slid ^^^^ typo - 'dlid'. > +* Destination LID of congestion event. > +* > +* sl > +* bits [3:0] SL of congestion event. > +* bits [7:4] reserved. > +* > +* time_stamp > +* Timestamp of congestion event. > +* > +* SEE ALSO > +* ib_cc_mad_t, ib_cong_log_t > +*********/ > + > +/****s* IBA Base: Types/ib_cong_log_event_ca_t > +* NAME > +* ib_cong_log_event_ca_t > +* > +* DESCRIPTION > +* IBA defined CongestionLogEvent (CA) entry (A10.4.3.5) > +* > +* SYNOPSIS > +*/ > +#include > +typedef struct _ib_cong_log_event_ca { > + ib_net32_t resv0_local_qp; > + ib_net32_t remote_qp_sl_service_type; > + ib_net16_t remote_lid; > + ib_net16_t resv1; > + ib_net32_t time_stamp; > +} PACK_SUFFIX ib_cong_log_event_ca_t; > +#include Does it implement table 514? > +/* > +* FIELDS > +* resv0_local_qp > +* bits [23:0] local QP that reached CN threshold. > +* bits [31:24] reserved. > +* > +* remote_qp_sl_service_type > +* bits [23:0] remote QP that is connected to local QP. > +* bits [27:24] SL of the local QP. > +* bits [31:28] Service Type of the local QP. > +* > +* remote_lid > +* LID of the remote port that is connected to local QP. > +* > +* time_stamp > +* Timestamp when threshold reached. > +* > +* SEE ALSO > +* ib_cc_mad_t, ib_cong_log_t > +*********/ > + > +/****s* IBA Base: Types/ib_cong_log_t > +* NAME > +* ib_cong_log_t > +* > +* DESCRIPTION > +* IBA defined CongestionLog attribute (A10.4.3.5) > +* > +* SYNOPSIS > +*/ > +#include > +typedef struct _ib_cong_log { > + uint8_t log_type; > + union _log_details > + { > + struct _log_sw { > + uint8_t cong_flags; > + ib_net16_t event_counter; > + ib_net32_t time_stamp; > + uint8_t port_map[32]; > + ib_cong_log_event_sw_t entry_list[15]; > + } PACK_SUFFIX log_sw; > + > + struct _log_ca { > + uint8_t cong_flags; > + ib_net16_t event_counter; > + ib_net16_t event_map; > + ib_net16_t resv; > + ib_net32_t time_stamp; In the spec I see: ib_net16_t event_map; ib_net16_t time_stamp; > + ib_cong_log_event_ca_t log_event[13]; > + } PACK_SUFFIX log_ca; > + > + } log_details; > +} PACK_SUFFIX ib_cong_log_t; > +#include > +/* > +* FIELDS > +* > +* log_{sw,ca}.log_type > +* Log type: 0x1 is for Switch, 0x2 is for CA > +* > +* log_{sw,ca}.cong_flags > +* Congestion Flags. > +* > +* log_{sw,ca}.event_counter > +* Number of events since log last sent. > +* > +* log_{sw,ca}.time_stamp > +* Timestamp when log sent. > +* > +* log_sw.port_map > +* If a bit set to 1, then the corresponding port > +* has marked packets with a FECN. > +* bits 0 and 255 - reserved > +* bits [254..1] - ports [254..1]. > +* > +* log_sw.entry_list > +* Array of 13 most recent congestion log events. > +* > +* log_ca.event_map > +* array 16 bits, one for each SL. > +* > +* log_ca.log_event > +* Array of 13 most recent congestion log events. > +* > +* SEE ALSO > +* ib_cc_mad_t, ib_cong_log_event_sw_t, ib_cong_log_event_ca_t > +*********/ > + > +/****s* IBA Base: Types/ib_sw_cong_setting_t > +* NAME > +* ib_sw_cong_setting_t > +* > +* DESCRIPTION > +* IBA defined SwitchCongestionSetting attribute (A10.4.3.6) > +* > +* SYNOPSIS > +*/ > +#include > +typedef struct _ib_sw_cong_setting { > + ib_net32_t control_map; > + uint8_t victim_mask[32]; > + uint8_t credit_mask[32]; > + uint8_t threshold; > + uint8_t packet_size; > + uint8_t cs_threshold; > + uint8_t resv0; I don't see this resv0 byte in the spec. > + ib_net16_t cs_return_delay; > + ib_net16_t marking_rate; > +} PACK_SUFFIX ib_sw_cong_setting_t; > +#include > +/* > +* FIELDS > +* > +* control_map > +* Indicates which components of this attribute are valid > +* > +* victim_mask > +* If the bit set to 1, then the port corresponding to > +* that bit shall mark packets that encounter congestion > +* with a FECN, whether they are the source or victim > +* of congestion. (See A10.2.1.1.1) > +* bit 0: port 0 (enhanced port 0 only) > +* bits [254..1]: ports [254..1] > +* bit 255: reserved > +* > +* credit_mask > +* If the bit set to 1, then the port corresponding > +* to that bit shall apply Credit Starvation. > +* bit 0: port 0 (enhanced port 0 only) > +* bits [254..1]: ports [254..1] > +* bit 255: reserved > +* > +* threshold > +* bits [3..0] Indicates how agressive cong. marking should be > +* bits [7..4] Reserved > +* > +* packet_size > +* Any packet less than this size won't be marked with FECN > +* > +* cs_threshold > +* bits [3..0] How agressive Credit Starvation should be > +* bits [7..4] Reserved > +* > +* cs_return_delay > +* Value that controls credit return rate. > +* > +* marking_rate > +* The value that provides the mean number of packets > +* between marking eligible packets with FECN. > +* > +* SEE ALSO > +* ib_cc_mad_t > +*********/ > + > +/****s* IBA Base: Types/ib_sw_port_cong_setting_element_t > +* NAME > +* ib_sw_port_cong_setting_element_t > +* > +* DESCRIPTION > +* IBA defined SwitchPortCongestionSettingElement (A10.4.3.7) > +* > +* SYNOPSIS > +*/ > +#include > +typedef struct _ib_sw_port_cong_setting_element { > + uint8_t valid_ctrl_type_res_threshold; > + uint8_t packet_size; > + ib_net16_t cong_param; packet_size and cong_param is a same 16-bit field in the spec. > +} PACK_SUFFIX ib_sw_port_cong_setting_element_t; > +#include > +/* > +* FIELDS > +* > +* valid_ctrl_type_res_threshold > +* bit 0: "Valid" > +* when set to 1, indicates this switch > +* port congestion setting element is valid. > +* bit 1: "Control Type" > +* Indicates which type of attribute is being set: > +* 0b = Congestion Control parameters are being set. > +* 1b = Credit Starvation parameters are being set. > +* bits [3..2]: reserved > +* bits [7..4]: "Threshold" > +* When Control Type is 0, contains the congestion > +* threshold value (Threshold) for this port. > +* When Control Type is 1, contains the credit > +* starvation threshold (CS_Threshold) value for > +* this port. > +* > +* packet_size > +* When Control Type is 0, this field contains the minimum > +* size of packets that may be marked with a FECN. > +* When Control Type is 1, this field is reserved. > +* > +* cong_parm > +* When Control Type is 0, this field contains the port > +* marking_rate. > +* When Control Type is 1, this field is reserved. > +* > +* SEE ALSO > +* ib_cc_mad_t, ib_sw_port_cong_setting_t > +*********/ > + > +/****d* IBA Base: Types/ib_sw_port_cong_setting_block_t > +* NAME > +* ib_sw_port_cong_setting_block_t > +* > +* DESCRIPTION > +* Defines the SwitchPortCongestionSetting Block (A10.4.3.7). > +* > +* SOURCE > +*/ > +typedef ib_sw_port_cong_setting_element_t ib_sw_port_cong_setting_block_t[32]; In the spec I see that it should be 64 24-bit elements here. Are we using the same spec? Sasha From sashak at voltaire.com Mon Jun 30 02:06:52 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 12:06:52 +0300 Subject: [ofa-general] Re: [PATCH v2] [TRIVIAL] opensm/osm_state_mgr.c: fixing some data types In-Reply-To: <48689ED1.1000907@dev.mellanox.co.il> References: <486881E1.1090603@dev.mellanox.co.il> <20080630072954.GL12659@sashak.voltaire.com> <48689ED1.1000907@dev.mellanox.co.il> Message-ID: <20080630090652.GO12659@sashak.voltaire.com> On 11:52 Mon 30 Jun , Yevgeny Kliteynik wrote: > > cl_ptr_vector_get_size() returns uint32_t, Correct. I meant cl_ptr_vector_at() gets size_t as second parameter. > and the value is lid, > which means that the vector can't be longer than uint16_t. It doesn't matter - we need to use valid types in our APIs. >> I think the proper solution for this is to use >> '%zu' format in OSM_LOG(). > > OK, patch shortly. Thanks. Sasha From vlad at lists.openfabrics.org Mon Jun 30 02:40:01 2008 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 30 Jun 2008 02:40:01 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20080630-0200 daily build status Message-ID: <20080630094001.54025E60260@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26-rc6 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26-rc6 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.16 Failed: Build failed on x86_64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-1.2798.fc6 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-1.2798.fc6_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-1.2798.fc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-53.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-53.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-53.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.19 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -fasynchronous-unwind-tables -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.20 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DIPATH_IDSTR='"QLogic kernel.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_driver)" -D"KBUILD_MODNAME=KBUILD_STR(ib_ipath)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: error: unknown field 'groups' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:155: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.20_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.20' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-42.ELsmp Log: include/asm/apic.h:47: warning: value computed is not used /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c: In function 'ipath_diagpkt_write': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.c:473: error: implicit declaration of function '__iowrite32_copy' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath/ipath_diag.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-42.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-42.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-42.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on x86_64 with linux-2.6.9-55.ELsmp Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c: At top level: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: error: unknown field 'change_queue_depth' specified in initializer /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.c:552: warning: initialization from incompatible pointer type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser/iscsi_iser.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.9-55.ELsmp_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.9-55.ELsmp' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.23 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.23_ia64_check/kernel_addons/backport/2.6.23/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:50: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.23_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.23_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.23_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.23_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.23' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ia64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ia64_check/kernel_addons/backport/2.6.24/include/linux/inetdevice.h:4, from /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.c:32: include/linux/slub_def.h:58: error: field 'kobj' has incomplete type make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ia64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ia64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ia64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ia64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ia64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.18-8.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.18-8.el5_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.18-8.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.17 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2616: error: implicit declaration of function 'pid_nr' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_signal_procs': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:2671: error: implicit declaration of function 'kill_pid' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.17_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.17' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.25 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' Makefile:514: /home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.25/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.25' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.24 Log: from /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.c:32: include/linux/slab.h:270: error: conflicting types for 'kzalloc' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ppc64_check/kernel_addons/backport/2.6.24/include/linux/kobject.h:74: error: previous implicit declaration of 'kzalloc' was here make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ppc64_check/drivers/infiniband/core] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.24_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.24' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.19 Log: /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1260: warning: format '%Lx' expects type 'long long unsigned int', but argument 6 has type 'long unsigned int' /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c: In function 'ipath_reset_availshadow': /home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.c:1459: warning: format '%Lx' expects type 'long long unsigned int', but argument 4 has type 'u64' make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath/ipath_driver.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check/drivers/infiniband/hw/ipath] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20080630-0200_linux-2.6.19_ppc64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.19' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- Build failed on ppc64 with linux-2.6.26-rc6 Log: -Iinclude \ $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \ ' \ modules make[1]: Entering directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' Makefile:518: /home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile: No such file or directory make[1]: *** No rule to make target `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6/arch/ppc64/Makefile'. Stop. make[1]: Leaving directory `/home/vlad/kernel.org/ppc64/linux-2.6.26-rc6' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From Neue at lists.openfabrics.org Mon Jun 30 03:27:13 2008 From: Neue at lists.openfabrics.org (Neue at lists.openfabrics.org) Date: Mon, 30 Jun 2008 11:27:13 +0100 Subject: [ofa-general] Qualitaet Die Nicht Ueberteuert Ist, Supergeil. Message-ID: <01c8daa4$3e03fe80$4ed30d56@psdennis.mitchell> So viel Software fuer so wenig Geld! Sie werden wirklich begeistert sein! Klicken Sie auf dieses Link und besuchen Sie unsere perfekte Softwareseite, es lohnt sich wirklich! Hier finden Sie endlich, was Sie suchen. Top Qualitaet, Super Preise und sehr sehr viel Software. Fuer PC und MAc, in allen moeglichen Sprachen - so viel und so guenstig, kommen Sie doch rein und schauen Sie sich das an From tziporet at mellanox.co.il Mon Jun 30 03:34:19 2008 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 30 Jun 2008 13:34:19 +0300 Subject: [ofa-general] OFED meeting agenda for today (June 30, 2008) Message-ID: <40FA0A8088E8A441973D37502F00933E2B498C@mtlexch01.mtl.com> This is the agenda for OFED meeting today (June 30, 2008) 1. OFED 1.4 status: Reminder: we have two more weeks for feature freeze All must submit code as soon as possible and not wait for last minute. - Kernel: - ipath driver still fails - iSER backport for RHEL4 are still missing - New components: iSER target and NFSoRDMA - we still wait for the maintainers to submit the new code. - Userspace status: - Development ongoing on master git trees - If someone has any update (uDAPL, MGT, verbs, ...) 2. Open discussion Note: All meeting minutes are now places on the web at: http://www.openfabrics.org/txt/documentation/linux/EWG_meeting_minutes/ I started update from June 08 Tziporet From kliteyn at dev.mellanox.co.il Mon Jun 30 03:40:31 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 13:40:31 +0300 Subject: [ofa-general] Re: [PATCH] opensm/include/iba/ib_types.h: Definition of Congestion Control MADs In-Reply-To: <20080630090014.GN12659@sashak.voltaire.com> References: <48688300.5050902@dev.mellanox.co.il> <20080630090014.GN12659@sashak.voltaire.com> Message-ID: <4868B81F.4080500@dev.mellanox.co.il> Hi Sasha, Right, I forgot to include here additional info about IB Spec. Current CC annex version has many problems. Attached file is the right format of CC MADs (tables 508, 509, 511 - 517, 519, 520). Except for the slid <-> dlid typo, all the comments refer to MADs format. I'll repost a patch with this fix. -- Yevgeny Sasha Khapyorsky wrote: > Hi Yevgeny, > > On 09:53 Mon 30 Jun , Yevgeny Kliteynik wrote: >> @@ -7126,6 +7250,19 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated >> uint8_t details[54]; >> } PACK_SUFFIX raw_data; >> >> + struct _ntc_0 { // CC_Key violation [54 bytes] >> + ib_net16_t source_lid; // Source LID from offending packet LRH >> + uint8_t method; // Method, from common MAD header >> + uint8_t resv0; >> + ib_net16_t attr_id; // Attribute ID, from common MAD header >> + ib_net16_t resv1; >> + ib_net32_t attr_mod; // Attribute Modif, from common MAD header >> + ib_net32_t qp; // 8b pad, 24b dest QP from BTH > > Looked at the spec (1.2.1). Should not here be: > > ib_net16_t attr_id; > ib_net32_t attr_mode; > ib_net32_t qp; > > ? > >> + ib_net64_t cc_key; // CC key of the offending packet >> + ib_gid_t source_gid; // GID from GRH of the offending packet >> + uint8_t padding[14]; // Padding - ignored on read > > And > uint8_t padding[24]; > > here? > >> + } PACK_SUFFIX ntc_0; >> + >> struct _ntc_64_67 { >> uint8_t res[6]; >> ib_gid_t gid; // the Node or Multicast Group that came in/out > > [snip...] > >> +/****s* IBA Base: Types/ib_cong_info_t >> +* NAME >> +* ib_cong_info_t >> +* >> +* DESCRIPTION >> +* IBA defined CongestionInfo attribute (A10.4.3.3) >> +* >> +* SYNOPSIS >> +*/ >> +#include >> +typedef struct _ib_cong_info { >> + uint8_t cong_info; >> + uint8_t resv; > > It is probably better to make it > > ib_net16_t cong_info; > > , so we will not need to rework later. Bits values we can define in > network byte order. > >> + uint8_t ctrl_table_cap; >> +} PACK_SUFFIX ib_cong_info_t; >> +#include >> +/* >> +* FIELDS >> +* cong_info >> +* Congestion control capabilities of the node. >> +* >> +* ctrl_table_cap >> +* Number of 64 entry blocks in the CongestionControlTable. >> +* >> +* SEE ALSO >> +* ib_cc_mad_t >> +*********/ > > [snip...] > >> +/****s* IBA Base: Types/ib_cong_log_event_sw_t >> +* NAME >> +* ib_cong_log_event_sw_t >> +* >> +* DESCRIPTION >> +* IBA defined CongestionLogEvent (SW) entry (A10.4.3.5) >> +* >> +* SYNOPSIS >> +*/ >> +#include >> +typedef struct _ib_cong_log_event_sw { >> + ib_net16_t slid; >> + ib_net16_t dlid; >> + uint8_t resv0_sl; >> + uint8_t resv1; >> + ib_net16_t resv2; >> + ib_net32_t time_stamp; >> +} PACK_SUFFIX ib_cong_log_event_sw_t; >> +#include >> +/* >> +* FIELDS >> +* slid >> +* Source LID of congestion event. >> +* >> +* slid > ^^^^ > > typo - 'dlid'. > >> +* Destination LID of congestion event. >> +* >> +* sl >> +* bits [3:0] SL of congestion event. >> +* bits [7:4] reserved. >> +* >> +* time_stamp >> +* Timestamp of congestion event. >> +* >> +* SEE ALSO >> +* ib_cc_mad_t, ib_cong_log_t >> +*********/ >> + >> +/****s* IBA Base: Types/ib_cong_log_event_ca_t >> +* NAME >> +* ib_cong_log_event_ca_t >> +* >> +* DESCRIPTION >> +* IBA defined CongestionLogEvent (CA) entry (A10.4.3.5) >> +* >> +* SYNOPSIS >> +*/ >> +#include >> +typedef struct _ib_cong_log_event_ca { >> + ib_net32_t resv0_local_qp; >> + ib_net32_t remote_qp_sl_service_type; >> + ib_net16_t remote_lid; >> + ib_net16_t resv1; >> + ib_net32_t time_stamp; >> +} PACK_SUFFIX ib_cong_log_event_ca_t; >> +#include > > Does it implement table 514? > >> +/* >> +* FIELDS >> +* resv0_local_qp >> +* bits [23:0] local QP that reached CN threshold. >> +* bits [31:24] reserved. >> +* >> +* remote_qp_sl_service_type >> +* bits [23:0] remote QP that is connected to local QP. >> +* bits [27:24] SL of the local QP. >> +* bits [31:28] Service Type of the local QP. >> +* >> +* remote_lid >> +* LID of the remote port that is connected to local QP. >> +* >> +* time_stamp >> +* Timestamp when threshold reached. >> +* >> +* SEE ALSO >> +* ib_cc_mad_t, ib_cong_log_t >> +*********/ >> + >> +/****s* IBA Base: Types/ib_cong_log_t >> +* NAME >> +* ib_cong_log_t >> +* >> +* DESCRIPTION >> +* IBA defined CongestionLog attribute (A10.4.3.5) >> +* >> +* SYNOPSIS >> +*/ >> +#include >> +typedef struct _ib_cong_log { >> + uint8_t log_type; >> + union _log_details >> + { >> + struct _log_sw { >> + uint8_t cong_flags; >> + ib_net16_t event_counter; >> + ib_net32_t time_stamp; >> + uint8_t port_map[32]; >> + ib_cong_log_event_sw_t entry_list[15]; >> + } PACK_SUFFIX log_sw; >> + >> + struct _log_ca { >> + uint8_t cong_flags; >> + ib_net16_t event_counter; >> + ib_net16_t event_map; >> + ib_net16_t resv; >> + ib_net32_t time_stamp; > > In the spec I see: > > ib_net16_t event_map; > ib_net16_t time_stamp; > >> + ib_cong_log_event_ca_t log_event[13]; >> + } PACK_SUFFIX log_ca; >> + >> + } log_details; >> +} PACK_SUFFIX ib_cong_log_t; >> +#include >> +/* >> +* FIELDS >> +* >> +* log_{sw,ca}.log_type >> +* Log type: 0x1 is for Switch, 0x2 is for CA >> +* >> +* log_{sw,ca}.cong_flags >> +* Congestion Flags. >> +* >> +* log_{sw,ca}.event_counter >> +* Number of events since log last sent. >> +* >> +* log_{sw,ca}.time_stamp >> +* Timestamp when log sent. >> +* >> +* log_sw.port_map >> +* If a bit set to 1, then the corresponding port >> +* has marked packets with a FECN. >> +* bits 0 and 255 - reserved >> +* bits [254..1] - ports [254..1]. >> +* >> +* log_sw.entry_list >> +* Array of 13 most recent congestion log events. >> +* >> +* log_ca.event_map >> +* array 16 bits, one for each SL. >> +* >> +* log_ca.log_event >> +* Array of 13 most recent congestion log events. >> +* >> +* SEE ALSO >> +* ib_cc_mad_t, ib_cong_log_event_sw_t, ib_cong_log_event_ca_t >> +*********/ >> + >> +/****s* IBA Base: Types/ib_sw_cong_setting_t >> +* NAME >> +* ib_sw_cong_setting_t >> +* >> +* DESCRIPTION >> +* IBA defined SwitchCongestionSetting attribute (A10.4.3.6) >> +* >> +* SYNOPSIS >> +*/ >> +#include >> +typedef struct _ib_sw_cong_setting { >> + ib_net32_t control_map; >> + uint8_t victim_mask[32]; >> + uint8_t credit_mask[32]; >> + uint8_t threshold; >> + uint8_t packet_size; >> + uint8_t cs_threshold; >> + uint8_t resv0; > > I don't see this resv0 byte in the spec. > >> + ib_net16_t cs_return_delay; >> + ib_net16_t marking_rate; >> +} PACK_SUFFIX ib_sw_cong_setting_t; >> +#include >> +/* >> +* FIELDS >> +* >> +* control_map >> +* Indicates which components of this attribute are valid >> +* >> +* victim_mask >> +* If the bit set to 1, then the port corresponding to >> +* that bit shall mark packets that encounter congestion >> +* with a FECN, whether they are the source or victim >> +* of congestion. (See A10.2.1.1.1) >> +* bit 0: port 0 (enhanced port 0 only) >> +* bits [254..1]: ports [254..1] >> +* bit 255: reserved >> +* >> +* credit_mask >> +* If the bit set to 1, then the port corresponding >> +* to that bit shall apply Credit Starvation. >> +* bit 0: port 0 (enhanced port 0 only) >> +* bits [254..1]: ports [254..1] >> +* bit 255: reserved >> +* >> +* threshold >> +* bits [3..0] Indicates how agressive cong. marking should be >> +* bits [7..4] Reserved >> +* >> +* packet_size >> +* Any packet less than this size won't be marked with FECN >> +* >> +* cs_threshold >> +* bits [3..0] How agressive Credit Starvation should be >> +* bits [7..4] Reserved >> +* >> +* cs_return_delay >> +* Value that controls credit return rate. >> +* >> +* marking_rate >> +* The value that provides the mean number of packets >> +* between marking eligible packets with FECN. >> +* >> +* SEE ALSO >> +* ib_cc_mad_t >> +*********/ >> + >> +/****s* IBA Base: Types/ib_sw_port_cong_setting_element_t >> +* NAME >> +* ib_sw_port_cong_setting_element_t >> +* >> +* DESCRIPTION >> +* IBA defined SwitchPortCongestionSettingElement (A10.4.3.7) >> +* >> +* SYNOPSIS >> +*/ >> +#include >> +typedef struct _ib_sw_port_cong_setting_element { >> + uint8_t valid_ctrl_type_res_threshold; >> + uint8_t packet_size; >> + ib_net16_t cong_param; > > packet_size and cong_param is a same 16-bit field in the spec. > >> +} PACK_SUFFIX ib_sw_port_cong_setting_element_t; >> +#include >> +/* >> +* FIELDS >> +* >> +* valid_ctrl_type_res_threshold >> +* bit 0: "Valid" >> +* when set to 1, indicates this switch >> +* port congestion setting element is valid. >> +* bit 1: "Control Type" >> +* Indicates which type of attribute is being set: >> +* 0b = Congestion Control parameters are being set. >> +* 1b = Credit Starvation parameters are being set. >> +* bits [3..2]: reserved >> +* bits [7..4]: "Threshold" >> +* When Control Type is 0, contains the congestion >> +* threshold value (Threshold) for this port. >> +* When Control Type is 1, contains the credit >> +* starvation threshold (CS_Threshold) value for >> +* this port. >> +* >> +* packet_size >> +* When Control Type is 0, this field contains the minimum >> +* size of packets that may be marked with a FECN. >> +* When Control Type is 1, this field is reserved. >> +* >> +* cong_parm >> +* When Control Type is 0, this field contains the port >> +* marking_rate. >> +* When Control Type is 1, this field is reserved. >> +* >> +* SEE ALSO >> +* ib_cc_mad_t, ib_sw_port_cong_setting_t >> +*********/ >> + >> +/****d* IBA Base: Types/ib_sw_port_cong_setting_block_t >> +* NAME >> +* ib_sw_port_cong_setting_block_t >> +* >> +* DESCRIPTION >> +* Defines the SwitchPortCongestionSetting Block (A10.4.3.7). >> +* >> +* SOURCE >> +*/ >> +typedef ib_sw_port_cong_setting_element_t ib_sw_port_cong_setting_block_t[32]; > > In the spec I see that it should be 64 24-bit elements here. > > Are we using the same spec? > > Sasha > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cc_tables.txt URL: From kliteyn at dev.mellanox.co.il Mon Jun 30 03:44:56 2008 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 30 Jun 2008 13:44:56 +0300 Subject: [ofa-general] [PATCH v2] opensm/include/iba/ib_types.h: Definition of Congestion Control MADs Message-ID: <4868B928.4070307@dev.mellanox.co.il> Hi Sasha, Adding definition of all the Congestion Control (CC) MADs to ib_types.h. V2 - fixed comment typo Signed-off-by: Yevgeny Kliteynik --- opensm/include/iba/ib_types.h | 786 +++++++++++++++++++++++++++++++++++++++++ 1 files changed, 786 insertions(+), 0 deletions(-) diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index 09ec257..8d0c903 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -443,6 +443,17 @@ BEGIN_C_DECLS */ #define IB_MCLASS_PERF 0x04 /**********/ +/****d* IBA Base: Constants/IB_MCLASS_CC +* NAME +* IB_MCLASS_CC +* +* DESCRIPTION +* Management Class, Congestion Control (A10.4.1) +* +* SOURCE +*/ +#define IB_MCLASS_CC 0x21 +/**********/ /****d* IBA Base: Constants/IB_MCLASS_BM * NAME * IB_MCLASS_BM @@ -1005,6 +1016,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_SMPL_RSLT (CL_HTON16(0x0011)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CONG_INFO +* NAME +* IB_MAD_ATTR_CONG_INFO +* +* DESCRIPTION +* CongestionInfo attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CONG_INFO (CL_HTON16(0x0011)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_SWITCH_INFO * NAME * IB_MAD_ATTR_SWITCH_INFO @@ -1029,6 +1052,30 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_CNTRS (CL_HTON16(0x0012)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CONG_KEY_INFO +* NAME +* IB_MAD_ATTR_CONG_KEY_INFO +* +* DESCRIPTION +* CongestionKeyInfo attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CONG_KEY_INFO (CL_HTON16(0x0012)) +/**********/ + +/****d* IBA Base: Constants/IB_MAD_ATTR_CONG_LOG +* NAME +* IB_MAD_ATTR_CONG_LOG +* +* DESCRIPTION +* CongestionLog attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CONG_LOG (CL_HTON16(0x0013)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_GUID_INFO * NAME * IB_MAD_ATTR_GUID_INFO @@ -1041,6 +1088,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_GUID_INFO (CL_HTON16(0x0014)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_SW_CONG_SETTING +* NAME +* IB_MAD_ATTR_SW_CONG_SETTING +* +* DESCRIPTION +* SwitchCongestionSetting attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_SW_CONG_SETTING (CL_HTON16(0x0014)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_PORT_INFO * NAME * IB_MAD_ATTR_PORT_INFO @@ -1053,6 +1112,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_INFO (CL_HTON16(0x0015)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_SW_PORT_CONG_SETTING +* NAME +* IB_MAD_ATTR_SW_PORT_CONG_SETTING +* +* DESCRIPTION +* SwitchPortCongestionSetting attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_SW_PORT_CONG_SETTING (CL_HTON16(0x0015)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_P_KEY_TABLE * NAME * IB_MAD_ATTR_P_KEY_TABLE @@ -1065,6 +1136,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_P_KEY_TABLE (CL_HTON16(0x0016)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CA_CONG_SETTING +* NAME +* IB_MAD_ATTR_CA_CONG_SETTING +* +* DESCRIPTION +* CACongestionSetting attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CA_CONG_SETTING (CL_HTON16(0x0016)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_SLVL_TABLE * NAME * IB_MAD_ATTR_SLVL_TABLE @@ -1077,6 +1160,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_SLVL_TABLE (CL_HTON16(0x0017)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_CC_TBL +* NAME +* IB_MAD_ATTR_CC_TBL +* +* DESCRIPTION +* CongestionControlTable attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_CC_TBL (CL_HTON16(0x0017)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_VL_ARBITRATION * NAME * IB_MAD_ATTR_VL_ARBITRATION @@ -1089,6 +1184,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_VL_ARBITRATION (CL_HTON16(0x0018)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_TIME_STAMP +* NAME +* IB_MAD_ATTR_TIME_STAMP +* +* DESCRIPTION +* TimeStamp attribute (A10.4.3) +* +* SOURCE +*/ +#define IB_MAD_ATTR_TIME_STAMP (CL_HTON16(0x0018)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_LIN_FWD_TBL * NAME * IB_MAD_ATTR_LIN_FWD_TBL @@ -3246,6 +3353,23 @@ ib_path_rec_hop_limit(IN const ib_path_rec_t * const p_rec) #define IB_CLASS_RESP_TIME_MASK 0x1F /*********/ +/****s* IBA Base: Constants/IB_CLASS_ENH_PORT0_CC_MASK +* NAME +* IB_CLASS_ENH_PORT0_CC_MASK +* +* DESCRIPTION +* ClassPortInfo CapabilityMask bits. +* Switch only: This bit will be set if the EnhacedPort0 +* supports CA Congestion Control (A10.4.3.1). +* +* SEE ALSO +* ib_class_port_info_t +* +* SOURCE +*/ +#define IB_CLASS_ENH_PORT0_CC_MASK 0x0100 +/*********/ + /****s* IBA Base: Types/ib_class_port_info_t * NAME * ib_class_port_info_t @@ -7126,6 +7250,19 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated uint8_t details[54]; } PACK_SUFFIX raw_data; + struct _ntc_0 { // CC_Key violation [54 bytes] + ib_net16_t source_lid; // Source LID from offending packet LRH + uint8_t method; // Method, from common MAD header + uint8_t resv0; + ib_net16_t attr_id; // Attribute ID, from common MAD header + ib_net16_t resv1; + ib_net32_t attr_mod; // Attribute Modif, from common MAD header + ib_net32_t qp; // 8b pad, 24b dest QP from BTH + ib_net64_t cc_key; // CC key of the offending packet + ib_gid_t source_gid; // GID from GRH of the offending packet + uint8_t padding[14]; // Padding - ignored on read + } PACK_SUFFIX ntc_0; + struct _ntc_64_67 { uint8_t res[6]; ib_gid_t gid; // the Node or Multicast Group that came in/out @@ -7219,6 +7356,11 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated #define TRAP_144_MASK_LINK_WIDTH_ENABLE_CHANGE (CL_HTON16(0x0002)) #define TRAP_144_MASK_NODE_DESCRIPTION_CHANGE (CL_HTON16(0x0001)) +/** + * Trap 0 masks + */ +#define TRAP_0_MASK_QP (CL_HTON32(0x00FFFFFF)) + /****f* IBA Base: Types/ib_notice_is_generic * NAME * ib_notice_is_generic @@ -7843,6 +7985,650 @@ typedef struct _ib_port_samples_result { } PACK_SUFFIX ib_port_samples_result_t; #include +/****s* IBA Base: Types/ib_cc_mad_t +* NAME +* ib_cc_mad_t +* +* DESCRIPTION +* IBA defined Congestion Control MAD format. (A10.4.1) +* +* SYNOPSIS +*/ +#define IB_CC_LOG_DATA_SIZE 32 +#define IB_CC_MGT_DATA_SIZE 192 +#define IB_CC_MAD_HDR_SIZE (sizeof(ib_sa_mad_t) - IB_CC_LOG_DATA_SIZE \ + - IB_CC_MGT_DATA_SIZE) + +#include +typedef struct _ib_cc_mad { + ib_mad_t header; + ib_net64_t cc_key; + uint8_t log_data[IB_CC_LOG_DATA_SIZE]; + uint8_t mgt_data[IB_CC_MGT_DATA_SIZE]; +} PACK_SUFFIX ib_cc_mad_t; +#include +/* +* FIELDS +* header +* Common MAD header. +* +* cc_key +* CC_Key of the Congestion Control MAD. +* +* log_data +* Congestion Control log data of the CC MAD. +* +* mgt_data +* Congestion Control management data of the CC MAD. +* +* SEE ALSO +* ib_mad_t +*********/ + +/****f* IBA Base: Types/ib_cc_mad_get_cc_key +* NAME +* ib_cc_mad_get_cc_key +* +* DESCRIPTION +* Gets a CC_Key of the CC MAD. +* +* SYNOPSIS +*/ +static inline ib_net64_t OSM_API +ib_cc_mad_get_cc_key(IN const ib_cc_mad_t * const p_cc_mad) +{ + return p_cc_mad->cc_key; +} +/* +* PARAMETERS +* p_cc_mad +* [in] Pointer to the CC MAD packet. +* +* RETURN VALUES +* CC_Key of the provided CC MAD packet. +* +* NOTES +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****f* IBA Base: Types/ib_cc_mad_get_log_data_ptr +* NAME +* ib_cc_mad_get_mgt_data_ptr +* +* DESCRIPTION +* Gets a pointer to the CC MAD's log data area. +* +* SYNOPSIS +*/ +static inline void * OSM_API +ib_cc_mad_get_log_data_ptr(IN const ib_cc_mad_t * const p_cc_mad) +{ + return ((void *)p_cc_mad->log_data); +} +/* +* PARAMETERS +* p_cc_mad +* [in] Pointer to the CC MAD packet. +* +* RETURN VALUES +* Pointer to CC MAD log data area. +* +* NOTES +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****f* IBA Base: Types/ib_cc_mad_get_mgt_data_ptr +* NAME +* ib_cc_mad_get_mgt_data_ptr +* +* DESCRIPTION +* Gets a pointer to the CC MAD's management data area. +* +* SYNOPSIS +*/ +static inline void * OSM_API +ib_cc_mad_get_mgt_data_ptr(IN const ib_cc_mad_t * const p_cc_mad) +{ + return ((void *)p_cc_mad->mgt_data); +} +/* +* PARAMETERS +* p_cc_mad +* [in] Pointer to the CC MAD packet. +* +* RETURN VALUES +* Pointer to CC MAD management data area. +* +* NOTES +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cong_info_t +* NAME +* ib_cong_info_t +* +* DESCRIPTION +* IBA defined CongestionInfo attribute (A10.4.3.3) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_info { + uint8_t cong_info; + uint8_t resv; + uint8_t ctrl_table_cap; +} PACK_SUFFIX ib_cong_info_t; +#include +/* +* FIELDS +* cong_info +* Congestion control capabilities of the node. +* +* ctrl_table_cap +* Number of 64 entry blocks in the CongestionControlTable. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cong_key_info_t +* NAME +* ib_cong_key_info_t +* +* DESCRIPTION +* IBA defined CongestionKeyInfo attribute (A10.4.3.4) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_key_info { + ib_net64_t cc_key; + uint8_t protect_bit; + uint8_t resv; + ib_net16_t lease_period; + ib_net16_t violations; +} PACK_SUFFIX ib_cong_key_info_t; +#include +/* +* FIELDS +* cc_key +* 8-byte CC Key. +* +* protect_bit +* LSB is a CC Key Protect Bit, other 7 bits are reserved. +* +* lease_period +* How long the CC Key protect bit is to remain non-zero. +* +* violations +* Number of received MADs that violated CC Key. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cong_log_event_sw_t +* NAME +* ib_cong_log_event_sw_t +* +* DESCRIPTION +* IBA defined CongestionLogEvent (SW) entry (A10.4.3.5) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_log_event_sw { + ib_net16_t slid; + ib_net16_t dlid; + uint8_t resv0_sl; + uint8_t resv1; + ib_net16_t resv2; + ib_net32_t time_stamp; +} PACK_SUFFIX ib_cong_log_event_sw_t; +#include +/* +* FIELDS +* slid +* Source LID of congestion event. +* +* dlid +* Destination LID of congestion event. +* +* sl +* bits [3:0] SL of congestion event. +* bits [7:4] reserved. +* +* time_stamp +* Timestamp of congestion event. +* +* SEE ALSO +* ib_cc_mad_t, ib_cong_log_t +*********/ + +/****s* IBA Base: Types/ib_cong_log_event_ca_t +* NAME +* ib_cong_log_event_ca_t +* +* DESCRIPTION +* IBA defined CongestionLogEvent (CA) entry (A10.4.3.5) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_log_event_ca { + ib_net32_t resv0_local_qp; + ib_net32_t remote_qp_sl_service_type; + ib_net16_t remote_lid; + ib_net16_t resv1; + ib_net32_t time_stamp; +} PACK_SUFFIX ib_cong_log_event_ca_t; +#include +/* +* FIELDS +* resv0_local_qp +* bits [23:0] local QP that reached CN threshold. +* bits [31:24] reserved. +* +* remote_qp_sl_service_type +* bits [23:0] remote QP that is connected to local QP. +* bits [27:24] SL of the local QP. +* bits [31:28] Service Type of the local QP. +* +* remote_lid +* LID of the remote port that is connected to local QP. +* +* time_stamp +* Timestamp when threshold reached. +* +* SEE ALSO +* ib_cc_mad_t, ib_cong_log_t +*********/ + +/****s* IBA Base: Types/ib_cong_log_t +* NAME +* ib_cong_log_t +* +* DESCRIPTION +* IBA defined CongestionLog attribute (A10.4.3.5) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cong_log { + uint8_t log_type; + union _log_details + { + struct _log_sw { + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net32_t time_stamp; + uint8_t port_map[32]; + ib_cong_log_event_sw_t entry_list[15]; + } PACK_SUFFIX log_sw; + + struct _log_ca { + uint8_t cong_flags; + ib_net16_t event_counter; + ib_net16_t event_map; + ib_net16_t resv; + ib_net32_t time_stamp; + ib_cong_log_event_ca_t log_event[13]; + } PACK_SUFFIX log_ca; + + } log_details; +} PACK_SUFFIX ib_cong_log_t; +#include +/* +* FIELDS +* +* log_{sw,ca}.log_type +* Log type: 0x1 is for Switch, 0x2 is for CA +* +* log_{sw,ca}.cong_flags +* Congestion Flags. +* +* log_{sw,ca}.event_counter +* Number of events since log last sent. +* +* log_{sw,ca}.time_stamp +* Timestamp when log sent. +* +* log_sw.port_map +* If a bit set to 1, then the corresponding port +* has marked packets with a FECN. +* bits 0 and 255 - reserved +* bits [254..1] - ports [254..1]. +* +* log_sw.entry_list +* Array of 13 most recent congestion log events. +* +* log_ca.event_map +* array 16 bits, one for each SL. +* +* log_ca.log_event +* Array of 13 most recent congestion log events. +* +* SEE ALSO +* ib_cc_mad_t, ib_cong_log_event_sw_t, ib_cong_log_event_ca_t +*********/ + +/****s* IBA Base: Types/ib_sw_cong_setting_t +* NAME +* ib_sw_cong_setting_t +* +* DESCRIPTION +* IBA defined SwitchCongestionSetting attribute (A10.4.3.6) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_sw_cong_setting { + ib_net32_t control_map; + uint8_t victim_mask[32]; + uint8_t credit_mask[32]; + uint8_t threshold; + uint8_t packet_size; + uint8_t cs_threshold; + uint8_t resv0; + ib_net16_t cs_return_delay; + ib_net16_t marking_rate; +} PACK_SUFFIX ib_sw_cong_setting_t; +#include +/* +* FIELDS +* +* control_map +* Indicates which components of this attribute are valid +* +* victim_mask +* If the bit set to 1, then the port corresponding to +* that bit shall mark packets that encounter congestion +* with a FECN, whether they are the source or victim +* of congestion. (See A10.2.1.1.1) +* bit 0: port 0 (enhanced port 0 only) +* bits [254..1]: ports [254..1] +* bit 255: reserved +* +* credit_mask +* If the bit set to 1, then the port corresponding +* to that bit shall apply Credit Starvation. +* bit 0: port 0 (enhanced port 0 only) +* bits [254..1]: ports [254..1] +* bit 255: reserved +* +* threshold +* bits [3..0] Indicates how agressive cong. marking should be +* bits [7..4] Reserved +* +* packet_size +* Any packet less than this size won't be marked with FECN +* +* cs_threshold +* bits [3..0] How agressive Credit Starvation should be +* bits [7..4] Reserved +* +* cs_return_delay +* Value that controls credit return rate. +* +* marking_rate +* The value that provides the mean number of packets +* between marking eligible packets with FECN. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_sw_port_cong_setting_element_t +* NAME +* ib_sw_port_cong_setting_element_t +* +* DESCRIPTION +* IBA defined SwitchPortCongestionSettingElement (A10.4.3.7) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_sw_port_cong_setting_element { + uint8_t valid_ctrl_type_res_threshold; + uint8_t packet_size; + ib_net16_t cong_param; +} PACK_SUFFIX ib_sw_port_cong_setting_element_t; +#include +/* +* FIELDS +* +* valid_ctrl_type_res_threshold +* bit 0: "Valid" +* when set to 1, indicates this switch +* port congestion setting element is valid. +* bit 1: "Control Type" +* Indicates which type of attribute is being set: +* 0b = Congestion Control parameters are being set. +* 1b = Credit Starvation parameters are being set. +* bits [3..2]: reserved +* bits [7..4]: "Threshold" +* When Control Type is 0, contains the congestion +* threshold value (Threshold) for this port. +* When Control Type is 1, contains the credit +* starvation threshold (CS_Threshold) value for +* this port. +* +* packet_size +* When Control Type is 0, this field contains the minimum +* size of packets that may be marked with a FECN. +* When Control Type is 1, this field is reserved. +* +* cong_parm +* When Control Type is 0, this field contains the port +* marking_rate. +* When Control Type is 1, this field is reserved. +* +* SEE ALSO +* ib_cc_mad_t, ib_sw_port_cong_setting_t +*********/ + +/****d* IBA Base: Types/ib_sw_port_cong_setting_block_t +* NAME +* ib_sw_port_cong_setting_block_t +* +* DESCRIPTION +* Defines the SwitchPortCongestionSetting Block (A10.4.3.7). +* +* SOURCE +*/ +typedef ib_sw_port_cong_setting_element_t ib_sw_port_cong_setting_block_t[32]; +/**********/ + +/****s* IBA Base: Types/ib_sw_port_cong_setting_t +* NAME +* ib_sw_port_cong_setting_t +* +* DESCRIPTION +* IBA defined SwitchPortCongestionSetting attribute (A10.4.3.7) +* +* SYNOPSIS +*/ + +#include +typedef struct _ib_sw_port_cong_setting { + ib_sw_port_cong_setting_block_t block; +} PACK_SUFFIX ib_sw_port_cong_setting_t; +#include +/* +* FIELDS +* +* block +* SwitchPortCongestionSetting block. +* +* SEE ALSO +* ib_cc_mad_t, ib_sw_port_cong_setting_element_t +*********/ + +/****s* IBA Base: Types/ib_ca_cong_entry_t +* NAME +* ib_ca_cong_entry_t +* +* DESCRIPTION +* IBA defined CACongestionEntry (A10.4.3.8) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_ca_cong_entry { + ib_net16_t ccti_timer; + uint8_t ccti_increase; + uint8_t trigger_threshold; + uint8_t ccti_min; + uint8_t resv0; + ib_net16_t resv1; +} PACK_SUFFIX ib_ca_cong_entry_t; +#include +/* +* FIELDS +* +* ccti_timer +* When the timer expires it will be reset to its specified +* value, and 1 will be decremented from the CCTI. +* +* ccti_increase +* The number to be added to the table Index (CCTI) +* on the receipt of a BECN. +* +* trigger_threshold +* When the CCTI is equal to this value, an event +* is logged in the CAs cyclic event log. +* +* ccti_min +* The minimum value permitted for the CCTI. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_ca_cong_setting_t +* NAME +* ib_ca_cong_setting_t +* +* DESCRIPTION +* IBA defined CACongestionSetting attribute (A10.4.3.8) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_ca_cong_setting { + uint8_t port_control; + uint8_t resv0; + ib_net16_t control_map; + ib_ca_cong_entry_t entry_list[16]; +} PACK_SUFFIX ib_ca_cong_setting_t; +#include +/* +* FIELDS +* +* port_control +* Congestion attributes for this port: +* LSB = 0: QP based CC +* LSB = 1: SL/Port based CC +* All other bits are reserved +* +* control_map +* An array of sixteen bits, one for each SL. Each bit indicates +* whether or not the corresponding entry is to be modified. +* +* entry_list +* List of 16 CACongestionEntries, one per SL. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cc_tbl_entry_t +* NAME +* ib_cc_tbl_entry_t +* +* DESCRIPTION +* IBA defined CongestionControlTableEntry (A10.4.3.9) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cc_tbl_entry { + ib_net16_t shift_multiplier; +} PACK_SUFFIX ib_cc_tbl_entry_t; +#include +/* +* FIELDS +* +* shift_multiplier +* bits [1..0] - CCT Shift +* used when calculating the injection rate delay +* bits [15..2] - CCT Multiplier +* used when calculating the injection rate delay +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_cc_tbl_t +* NAME +* ib_cc_tbl_t +* +* DESCRIPTION +* IBA defined CongestionControlTable attribute (A10.4.3.9) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_cc_tbl { + ib_net16_t ccti_limit; + ib_net16_t resv; + ib_cc_tbl_entry_t entry_list[64]; +} PACK_SUFFIX ib_cc_tbl_t; +#include +/* +* FIELDS +* +* ccti_limit +* Maximum valid CCTI for this table. +* +* entry_list +* List of up to 64 CongestionControlTableEntries. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + +/****s* IBA Base: Types/ib_time_stamp_t +* NAME +* ib_time_stamp_t +* +* DESCRIPTION +* IBA defined TimeStamp attribute (A10.4.3.10) +* +* SOURCE +*/ +#include +typedef struct _ib_time_stamp { + ib_net32_t value; +} PACK_SUFFIX ib_time_stamp_t; +#include +/* +* FIELDS +* +* value +* Free running clock that provides relative time info +* for a device. Time is kept in 1.024 usec units. +* +* SEE ALSO +* ib_cc_mad_t +*********/ + /****d* IBA Base: Types/DM_SVC_NAME * NAME * DM_SVC_NAME -- 1.5.1.4 From ronli.voltaire at gmail.com Mon Jun 30 04:19:22 2008 From: ronli.voltaire at gmail.com (Ron Livne) Date: Mon, 30 Jun 2008 14:19:22 +0300 Subject: [ofa-general] Re: [PATCH 3/3 v4] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: References: Message-ID: <3b5e77ad0806300419w5c291b01ibb9b5a2ecd08efea@mail.gmail.com> Thank you, I'll be sure to fill both your requests for my next patches. We would like to start working on user space verbs for blocking multicast packets as was discussed in a previous thread. I would like to implement it simply by adding a new verb (kind of extended create_qp). I think this implementation will have similar characteristics with the XRC. Do you have any suggestions or preferences? Ron On Mon, Jun 30, 2008 at 8:55 AM, Roland Dreier wrote: > thanks, applied all 3. > > two requests for future patches that make it easier for me to apply > things without editing by hand: > > > There is also an improvment of 12% in cpu usage. > > > > > > Signed-off-by: Ron Livne > > > > Changes in v2: > > IB_QP_BLOCK_LOOPBACK QP creation flag in version 1, is now called > > IB_QP_CREATE_MULTICAST_BLOCK_LOOPBACK. > > > > Changes in v3: > > Whitespaces deleted. > > > > Changes in v4: > > The redundant initialization for create_flags (create_flags = 0) was deleted. > > > > --- > > Please put the patch's changelog and any other information that should > not be included in the actual commit message after the "---" line; then > it will be automatically stripped by standard patch import scripts. > > > --- kernels.orig/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-26 14:12:37.000000000 +0300 > > +++ kernels/linux-2.6.26-rc2/drivers/infiniband/ulp/ipoib/ipoib_verbs.c 2008-06-29 09:42:41.000000000 +0300 > > Please generate your patch so it applied with "git apply" or "patch -p1" -- > in other words the paths should just be like "old/drivers/infiniband/...". > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hrosenstock at xsigo.com Mon Jun 30 04:36:46 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 30 Jun 2008 04:36:46 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080627230339.GV29456@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> <20080627203745.GA29198@obsidianresearch.com> <20080627230339.GV29456@sashak.voltaire.com> Message-ID: <1214825806.3486.173.camel@hrosenstock-ws.xsigo.com> Hi Sasha, On Sat, 2008-06-28 at 02:03 +0300, Sasha Khapyorsky wrote: > Hi Jason, > > On 14:37 Fri 27 Jun , Jason Gunthorpe wrote: > > > > How is this better than just patching opensm directly for people who > > want these kinds of things? > > It is not better, it is different. Some things can be unrelated to > "pure" SM/SA functionality and not be part of OpenSM project, but > sometimes we may want to have it as part of OpenSM process, examples > are existing plugin 'opensmskumme' (it loads performance counters to > MySQL db), we wanted to have IB diagnostics plugin and management related > plugin. If this idea catches on, aside from exposing the internals in an unstructured manner, I see a mix and match issue with plugins developing which may lead to certain OpenSM versions being selected for compatibility or worse no choice due to plugin version compatibility as they get to support whatever versions they choose. Might this be better handled as packaging with separate packages based on licenses ? > > You can do alot of good by making the internal APIs 'plug in like' so > > adding new things doesn't require touching lots of places without > > going down the whole messy road of actual dynamically loadable plug ins.. > > > > But if you can't identify a fixed, clean API for a dynamically > > loadable plugin then you almost certainly should not have them in an > > open source project... > > > > This is probably better long term as far as encouraging more > > contribution to opensm rather than encouraging contributions to be > > kept as plugins.. I think this is safer from an engineering and support standpoint. > GPL is some sort of an issue - OFA requires dual licensing, so GPL only > code cannot enter OpenSM. Yes, OFA licensing is dual BSD and GPL. Perhaps some modification is needed to handle this scenario for OpenSM better. Was that entertained/explored ? -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From psdrcarpenter at ctc.net Mon Jun 30 05:06:38 2008 From: psdrcarpenter at ctc.net (Top Softwarehaus) Date: Mon, 30 Jun 2008 20:06:38 +0800 Subject: [ofa-general] Softwarepreise, Die Sie Noch Nie Gesehen Haben Message-ID: <01c8daec$cdcd9b00$300d037a@psdrcarpenter> Sehr geehrter Besucher, haben Sie vielleicht schon nachgedacht, warum die Software in Laeden so teuer ist? Ja, die Geschaefte, die viel Personal unterhalen und Miete zahlen, konnen nicht die billigsten sein. Die Lieferung erfolgt sehr schnell und Sie werden auch von unserem kompetenten Team unterstuetzt. So viel Service fuer so wenig Geld... probieren Sie es aus. http://vapwosoft.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From throwawaysmva7 at ewins.org Mon Jun 30 06:43:45 2008 From: throwawaysmva7 at ewins.org (Jaime Wesley) Date: Mon, 30 Jun 2008 10:43:45 -0300 Subject: [ofa-general] No test, No class, buy yourself Bacheelor/MasteerMBA/Doctoraate dip1omas, VALID in all countries Message-ID: <01c8da9e$2b86c680$ee853ebd@throwawaysmva7> Bacheelor, MasteerMBA, and Doctoraate diplomas available in the field of your choice that's right, you can even become a Doctor and receive all the benefits that comes with it! Our Diplomas/Certificates are recognised in most countries No required examination, tests, classes, books, or interviews. ** No one is turned down ** Confidentiality assured CALL US 24 HOURS A DAY, 7 DAYS A WEEK For US: 1-501-647-0781 Outside US: +1-501-647-0781 "Just leave your NAME & PHONE NO. (with CountryCode)" in the voicemail our staff will get back to you in next few days -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Mon Jun 30 07:27:05 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 17:27:05 +0300 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <1214825806.3486.173.camel@hrosenstock-ws.xsigo.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> <20080627203745.GA29198@obsidianresearch.com> <20080627230339.GV29456@sashak.voltaire.com> <1214825806.3486.173.camel@hrosenstock-ws.xsigo.com> Message-ID: <20080630142705.GF7229@sashak.voltaire.com> On 04:36 Mon 30 Jun , Hal Rosenstock wrote: > > If this idea catches on, aside from exposing the internals in an > unstructured manner, I see a mix and match issue with plugins developing > which may lead to certain OpenSM versions being selected for > compatibility or worse no choice due to plugin version compatibility as > they get to support whatever versions they choose. As we learned recently having structured API leads to such problems just well, in this case one can expect "stable API" even if it was never declared. I think I agree with Ira - OpenSM version exact match enforcement will make it clearer that plugin writer should be ready to rebuild and possibly update its code. > Might this be better handled as packaging with separate packages based > on licenses ? What do you mean? > > > You can do alot of good by making the internal APIs 'plug in like' so > > > adding new things doesn't require touching lots of places without > > > going down the whole messy road of actual dynamically loadable plug ins.. > > > > > > But if you can't identify a fixed, clean API for a dynamically > > > loadable plugin then you almost certainly should not have them in an > > > open source project... > > > > > > This is probably better long term as far as encouraging more > > > contribution to opensm rather than encouraging contributions to be > > > kept as plugins.. > > I think this is safer from an engineering and support standpoint. > > > GPL is some sort of an issue - OFA requires dual licensing, so GPL only > > code cannot enter OpenSM. > > Yes, OFA licensing is dual BSD and GPL. Perhaps some modification is > needed to handle this scenario for OpenSM better. Was that > entertained/explored ? Not by me. Sasha From 25kamil.krivanek at czechinvest.org Mon Jun 30 05:44:32 2008 From: 25kamil.krivanek at czechinvest.org (clive kaiser) Date: Mon, 30 Jun 2008 12:44:32 +0000 Subject: [ofa-general] Best of Adidas, Paul Smith, Versace Message-ID: <000401c8dabe$02f8f0b6$1615b6b9@edxajr> The worldwide largest luxury store for shoes and bags is just one click away. Recommended by tens thousands of grateful customers all around the world, we carry dozens of populars brands including: Hermes Bally Burberry Gucci Shoes UGG Here you willc find hundred thousands of stunning designs for shoes, and leather products, at at temp't1ng priceE. Sale close this weekend, so visit web site now and start pampering yourself and your loved second half. - Visit our site: www.eaopteail[DOT]com (copy this link then replace "[DOT]" to ".") From swise at opengridcomputing.com Mon Jun 30 07:45:55 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 09:45:55 -0500 Subject: [ofa-general] OFED meeting agenda for today (June 30, 2008) In-Reply-To: <40FA0A8088E8A441973D37502F00933E2B498C@mtlexch01.mtl.com> References: <40FA0A8088E8A441973D37502F00933E2B498C@mtlexch01.mtl.com> Message-ID: <4868F1A3.1030708@opengridcomputing.com> The MEM_MGT_EXTENSIONS patches are in Roland's infiniband.git for_2_6_27 tree. Since 2.6.27-rc1 will be merged into ofed-1.4, I shouldn't worry about generating patches for ofed-1.4, right? Tziporet Koren wrote: > > This is the agenda for OFED meeting today (June 30, 2008) > > 1. OFED 1.4 status: > Reminder: we have two more weeks for feature freeze > All must submit code as soon as possible and not wait for last > minute. > - Kernel: > - ipath driver still fails > - iSER backport for RHEL4 are still missing > - New components: iSER target and NFSoRDMA - we still wait for the > maintainers to submit the new code. > > - Userspace status: > - Development ongoing on master git trees > - If someone has any update (uDAPL, MGT, verbs, ...) > > 2. Open discussion > > Note: All meeting minutes are now places on the web at: > http://www.openfabrics.org/txt/documentation/linux/EWG_meeting_minutes/ > > I started update from June 08 > > Tziporet > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From xma at us.ibm.com Mon Jun 30 07:45:49 2008 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 30 Jun 2008 07:45:49 -0700 Subject: [ofa-general] Re: [PATCH] - [resend] Corrects a race in ipoib_cm_post_receive_nonsrq() In-Reply-To: Message-ID: Hello Roland, The patch looks ok to me. We will test it. Thanks Shirley -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Mon Jun 30 08:07:38 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 10:07:38 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <48686660.9020001@voltaire.com> References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> <48686660.9020001@voltaire.com> Message-ID: <4868F6BA.4070601@opengridcomputing.com> Or Gerlitz wrote: > Steve Wise wrote: >> Sort of. It is not exactly a dma_mr. dma_mrs allow remote IO. The >> zero-stag is an iWARP verbs concept. zero-stag can be used by kernel >> users as the lkey in any SGE that requires only local access rights. >> If the lkey is zero, then the rnic interprets the addr field in the >> SGE as a bus address. >> >> > I see. > > Assuming we want to narrow the differences between RDMA transports > from the ULP point of view and that IB does not support this verb - > I'd like to see if its possible to have zero stags implemented as dma > mrs. > > How about enhancing the iWARP HW drivers implementation of get_dma_mr > to switch on the access flags provided by the caller, and if no remote > access it needed, just return zero stag for mr.lkey without doing any > registration. > I think that sounds reasonable. I can add that for chelsio. Steve. From sashak at voltaire.com Mon Jun 30 09:00:30 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 19:00:30 +0300 Subject: [ofa-general] Re: [PATCH] opensm/include/iba/ib_types.h: Definition of Congestion Control MADs In-Reply-To: <4868B81F.4080500@dev.mellanox.co.il> References: <48688300.5050902@dev.mellanox.co.il> <20080630090014.GN12659@sashak.voltaire.com> <4868B81F.4080500@dev.mellanox.co.il> Message-ID: <20080630160030.GI7229@sashak.voltaire.com> On 13:40 Mon 30 Jun , Yevgeny Kliteynik wrote: > > Right, I forgot to include here additional info about IB Spec. > Current CC annex version has many problems. Attached file > is the right format of CC MADs (tables 508, 509, 511 - 517, > 519, 520). I see. Probably it is better to wait for at least "approved" state? Sasha From eli at dev.mellanox.co.il Mon Jun 30 09:06:51 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 30 Jun 2008 19:06:51 +0300 Subject: [ofa-general] [PATCH] IB/IPoIB: Fix change mtu when switching to UD mode Message-ID: <20080630160651.GA6024@mtls03> >From 83e0e38b113980112c4ed34d0de367afd2b2b198 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Mon, 30 Jun 2008 18:59:54 +0300 Subject: [PATCH] IB/IPoIB: Fix change mtu when switching to UD mode When changing from CM mode to UD mode, use the kernel function dev_set_mtu() instead of directly setting dev->mtu. This is necessary so any other subsystem that need to know about this will (this is done through the call to call_netdevice_notifiers(). Signed-off-by: Eli Cohen --- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 6223fc3..82f3f2f 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -1410,7 +1410,7 @@ static ssize_t set_mode(struct device *d, struct device_attribute *attr, if (!strcmp(buf, "datagram\n")) { clear_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags); - dev->mtu = min(priv->mcast_mtu, dev->mtu); + dev_set_mtu(dev, min(priv->mcast_mtu, dev->mtu)) ipoib_flush_paths(dev); if (test_bit(IPOIB_FLAG_CSUM, &priv->flags)) { -- 1.5.6 From cysmechanism at co-operations.com Mon Jun 30 09:15:33 2008 From: cysmechanism at co-operations.com (Charles Amesbury) Date: Mon, 30 Jun 2008 16:15:33 -0000 Subject: [ofa-general] Try, it is pleasant for you Message-ID: <001301c89fc3$862f6030$0032fa7c@liteextreme> It must be borne in perfect price! Being wise, buy their medicine from the best-known suppliers since 1911! Special prices for a guarantee! This is incredible! -------------- next part -------------- An HTML attachment was scrubbed... URL: From eli at dev.mellanox.co.il Mon Jun 30 09:18:28 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Mon, 30 Jun 2008 19:18:28 +0300 Subject: [ofa-general] performance drop for datagram mode with the new connectx FW In-Reply-To: <48679D21.20200@voltaire.com> References: <48679D21.20200@voltaire.com> Message-ID: <20080630161828.GB6024@mtls03> Or, I think this problem was introduced with the patch ipoib_0390_restore_cm_mtu.patch in ofed. This patch attempts to set the MTU when changing to CM mode to the max defined for CM mode (e.g. 65520). However the change was done by setting dev->mtu which is not how it should be done since it will not call any function registered to be notified on MTU change. The following patch solved the problem for me and you're welcome to try if it works for you too. Index: ofa_1_3_dev_kernel/drivers/infiniband/ulp/ipoib/ipoib_cm.c =================================================================== --- ofa_1_3_dev_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib_cm.c 2008-06-30 11:37:59.000000000 +0300 +++ ofa_1_3_dev_kernel/drivers/infiniband/ulp/ipoib/ipoib_cm.c 2008-06-30 18:53:31.000000000 +0300 @@ -1433,15 +1433,15 @@ static ssize_t set_mode(struct class_dev if (ipoib_cm_max_mtu(dev) > priv->mcast_mtu) ipoib_warn(priv, "mtu > %d will cause multicast packet drops.\n", priv->mcast_mtu); - dev->mtu = ipoib_cm_max_mtu(dev); + dev_set_mtu(dev, ipoib_cm_max_mtu(dev)); ipoib_flush_paths(dev); return count; } if (!strcmp(buf, "datagram\n")) { clear_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags); - dev->mtu = min(priv->mcast_mtu, dev->mtu); + dev_set_mtu(dev, min(priv->mcast_mtu, dev->mtu)); ipoib_flush_paths(dev); if (priv->ca->flags & IB_DEVICE_IP_CSUM) You can "fix" this problem by setting the MTU of CM mode manually from the shell after switching mode in which case you will not need to use this patch. From weiny2 at llnl.gov Mon Jun 30 09:27:16 2008 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 30 Jun 2008 09:27:16 -0700 Subject: [ofa-general] Re: [PATCH RFC] Create a new library "libibnetdisc" In-Reply-To: <20080627233409.GW29456@sashak.voltaire.com> References: <20080627160952.600be0e2.weiny2@llnl.gov> <20080627233409.GW29456@sashak.voltaire.com> Message-ID: <20080630092716.476ddf1a.weiny2@llnl.gov> On Sat, 28 Jun 2008 02:34:09 +0300 Sasha Khapyorsky wrote: > Hi Ira, > > On 16:09 Fri 27 Jun , Ira Weiny wrote: > > I have been wanted to do this for some time and ever since mentioning it to > > Sasha at Sonoma have worked on it in my "spare" time... ;-) > > > > This creates an ibnetdiscover library. It seems as time goes on that more and > > more people have tried to "print" something from ibnetdiscover to get that data > > out with some parser. I figure it should be available from a library. I have > > only written test programs "ibnetdisctest" and "iblinkinfotest" to prototype > > performance. Also the interface is a bit rought around the edges but all the > > data is there. > > > > At this point I have some pretty good results from iblinkinfotest (~80% > > faster!) > > > > So I wanted to get opinions on where this should go. > > Do you think it should be a separate package? I thought rather about > having it in infiniband-diags so the tools there will use it. > Yes I think it should be part of the diags tree, however at this point I did not want to break any of the existing tools until I get some feedback. I have done quite a bit of testing and I am pretty sure that all functionality of ibnetdiscover is preserved. Next step is to integrate and find a home for it... ;-) Ira From rdreier at cisco.com Mon Jun 30 09:53:48 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 09:53:48 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <4868F6BA.4070601@opengridcomputing.com> (Steve Wise's message of "Mon, 30 Jun 2008 10:07:38 -0500") References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> <48686660.9020001@voltaire.com> <4868F6BA.4070601@opengridcomputing.com> Message-ID: > > How about enhancing the iWARP HW drivers implementation of > > get_dma_mr to switch on the access flags provided by the caller, and > > if no remote access it needed, just return zero stag for mr.lkey > > without doing any registration. > I think that sounds reasonable. I can add that for chelsio. that makes some sense although I don't know of any code that uses the zero STag or ib_get_dma_mr() with only local permissions to be honest, with the exception of IPoIB, which of course has no relevance to iwarp. Also using the zero STag breaks the semantics slightly in that there is no longer any PD attached to the key returned by ib_get_dma_mr(). Another point to note is that IB has the analogous concept of "reserved L_Key" which we could expose if anyone actually cared. - R. From swise at opengridcomputing.com Mon Jun 30 09:57:37 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 11:57:37 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> <48686660.9020001@voltaire.com> <4868F6BA.4070601@opengridcomputing.com> Message-ID: <48691081.5080907@opengridcomputing.com> Roland Dreier wrote: > > > How about enhancing the iWARP HW drivers implementation of > > > get_dma_mr to switch on the access flags provided by the caller, and > > > if no remote access it needed, just return zero stag for mr.lkey > > > without doing any registration. > > > I think that sounds reasonable. I can add that for chelsio. > > that makes some sense although I don't know of any code that uses the > zero STag or ib_get_dma_mr() with only local permissions to be honest, > with the exception of IPoIB, which of course has no relevance to iwarp. > > NFSRDMA was planning on using the zero stag for recvs on iwarp transports. This boils down to only needing the zero stag + fastreg mrs for NFSRDMA server over T3, avoiding dma_mrs which have limits in the T3 HW. > Also using the zero STag breaks the semantics slightly in that there is > no longer any PD attached to the key returned by ib_get_dma_mr(). > > True. I didn't think about this. > Another point to note is that IB has the analogous concept of "reserved > L_Key" which we could expose if anyone actually cared. > Hmm. A new API then? ib_get_local_lkey() or something? If IB mandates the lkey be value of 0, then we need don't need to do anything... Steve. From rdreier at cisco.com Mon Jun 30 09:59:22 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 09:59:22 -0700 Subject: [ofa-general] [PATCH V2] IB/ipoib: refresh paths instead of fushing them on SM change event In-Reply-To: <48624168.3050305@Voltaire.COM> (Moni Shoua's message of "Wed, 25 Jun 2008 16:00:24 +0300") References: <484F8A20.9070101@Voltaire.COM> <48624168.3050305@Voltaire.COM> Message-ID: I wanted to apply this but I see some locking issues; > @@ -409,6 +427,9 @@ static void path_rec_completion(int status, > > spin_lock_irqsave(&priv->lock, flags); > > + if (path->ah) > + ipoib_put_ah(path->ah); Look at what ipoib_free_ah() does if this drops the last reference to the ah (hint -- it takes priv->lock). Is there any guarantee that there are more references to this ah still held? If so there needs to be a comment explaining this (as other places in ipoib_main.c have). > list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) { > + if (neigh->ah) > + old_ah = neigh->ah; is there any guarantee that this list has only one entry? If so why? - R. From sashak at voltaire.com Mon Jun 30 09:58:47 2008 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 30 Jun 2008 19:58:47 +0300 Subject: [ofa-general] Re: [PATCH v3] [TRIVIAL] opensm/osm_state_mgr.c: fixing log message In-Reply-To: <48689EB1.7080406@dev.mellanox.co.il> References: <48689EB1.7080406@dev.mellanox.co.il> Message-ID: <20080630165847.GK7229@sashak.voltaire.com> On 11:52 Mon 30 Jun , Yevgeny Kliteynik wrote: > Hi Sasha, > > Fixing some log messages (was "fixing some data types") > in osm_state_mgr.c to remove compiler warnings. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. Sasha From rdreier at cisco.com Mon Jun 30 10:01:27 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 10:01:27 -0700 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <48691081.5080907@opengridcomputing.com> (Steve Wise's message of "Mon, 30 Jun 2008 11:57:37 -0500") References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> <48686660.9020001@voltaire.com> <4868F6BA.4070601@opengridcomputing.com> <48691081.5080907@opengridcomputing.com> Message-ID: > NFSRDMA was planning on using the zero stag for recvs on iwarp > transports. This boils down to only needing the zero stag + fastreg > mrs for NFSRDMA server over T3, avoiding dma_mrs which have limits in > the T3 HW. OK, that makes sense. > Hmm. A new API then? ib_get_local_lkey() or something? If IB > mandates the lkey be value of 0, then we need don't need to do > anything... No, IB doesn't mandate that. I don't know if we want a new API or just a new field in struct ib_device. We already have the ZERO_STAG capability flag to show when this makes sense (and we could rename it to be more transport neutral), and we could just add a local_lkey field, since the value is fixed at device initialization time. - R. From swise at opengridcomputing.com Mon Jun 30 10:39:26 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 12:39:26 -0500 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> <48686660.9020001@voltaire.com> <4868F6BA.4070601@opengridcomputing.com> <48691081.5080907@opengridcomputing.com> Message-ID: <48691A4E.2080703@opengridcomputing.com> Roland Dreier wrote: > I don't know if we want a new API or just a new field in struct > ib_device. We already have the ZERO_STAG capability flag to show when > this makes sense (and we could rename it to be more transport neutral), > and we could just add a local_lkey field, since the value is fixed at > device initialization time. > > - R. > like this? > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h > index 74c24b9..2155a33 100644 > --- a/include/rdma/ib_verbs.h > +++ b/include/rdma/ib_verbs.h > @@ -91,7 +91,7 @@ enum ib_device_cap_flags { > IB_DEVICE_RC_RNR_NAK_GEN = (1<<12), > IB_DEVICE_SRQ_RESIZE = (1<<13), > IB_DEVICE_N_NOTIFY_CQ = (1<<14), > - IB_DEVICE_ZERO_STAG = (1<<15), > + IB_DEVICE_LOCAL_DMA_LKEY = (1<<15), > IB_DEVICE_RESERVED = (1<<16), /* old SEND_W_INV */ > IB_DEVICE_MEM_WINDOW = (1<<17), > /* > @@ -1096,6 +1096,7 @@ struct ib_device { > __be64 node_guid; > u8 node_type; > u8 phys_port_cnt; > + u32 local_dma_lkey; > }; > > struct ib_client { From swise at opengridcomputing.com Mon Jun 30 10:45:02 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 12:45:02 -0500 Subject: [ofa-general] [PATCH RFCv2 1/2] RDMA/Core: Add local dma lkey support. Message-ID: <20080630174502.15446.11459.stgit@dell3.ogc.int> From: Steve Wise - Change the IB_DEVICE_ZERO_STAG flag to the transport-neutral name IB_DEVICE_LOCAL_DMA_LKEY. - Add local_dma_lkey u32 to the ib_device struct. Drivers will fill this in with the appropriate local dma lkey if they support it. - Fix up the drivers using this flag. --- drivers/infiniband/hw/cxgb3/iwch_provider.c | 3 ++- drivers/infiniband/hw/nes/nes_hw.c | 2 +- include/rdma/ib_verbs.h | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 3101e3c..64d62a4 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1254,8 +1254,9 @@ int iwch_register_device(struct iwch_dev *dev) memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid)); memcpy(&dev->ibdev.node_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); dev->ibdev.owner = THIS_MODULE; - dev->device_cap_flags = IB_DEVICE_ZERO_STAG | + dev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY | IB_DEVICE_MEM_WINDOW; + dev->local_dma_lkey = 0; if (fw_supports_fastreg(dev)) dev->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; diff --git a/drivers/infiniband/hw/nes/nes_hw.c b/drivers/infiniband/hw/nes/nes_hw.c index d3278f1..070b0c7 100644 --- a/drivers/infiniband/hw/nes/nes_hw.c +++ b/drivers/infiniband/hw/nes/nes_hw.c @@ -398,7 +398,7 @@ struct nes_adapter *nes_init_adapter(struct nes_device *nesdev, u8 hw_rev) { nesadapter->base_pd = 1; nesadapter->device_cap_flags = - IB_DEVICE_ZERO_STAG | IB_DEVICE_MEM_WINDOW; + IB_DEVICE_LOCAL_DMA_LKEY | IB_DEVICE_MEM_WINDOW; nesadapter->allocated_qps = (unsigned long *)&(((unsigned char *)nesadapter) [(sizeof(struct nes_adapter)+(sizeof(unsigned long)-1))&(~(sizeof(unsigned long)-1))]); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 74c24b9..2155a33 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -91,7 +91,7 @@ enum ib_device_cap_flags { IB_DEVICE_RC_RNR_NAK_GEN = (1<<12), IB_DEVICE_SRQ_RESIZE = (1<<13), IB_DEVICE_N_NOTIFY_CQ = (1<<14), - IB_DEVICE_ZERO_STAG = (1<<15), + IB_DEVICE_LOCAL_DMA_LKEY = (1<<15), IB_DEVICE_RESERVED = (1<<16), /* old SEND_W_INV */ IB_DEVICE_MEM_WINDOW = (1<<17), /* @@ -1096,6 +1096,7 @@ struct ib_device { __be64 node_guid; u8 node_type; u8 phys_port_cnt; + u32 local_dma_lkey; }; struct ib_client { From swise at opengridcomputing.com Mon Jun 30 10:45:05 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 12:45:05 -0500 Subject: [ofa-general] [PATCH RFCv2 2/2] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <20080630174502.15446.11459.stgit@dell3.ogc.int> References: <20080630174502.15446.11459.stgit@dell3.ogc.int> Message-ID: <20080630174504.15446.16864.stgit@dell3.ogc.int> From: Steve Wise Handling the zero-stag in recv WRs requires some extra logic: - Only set the QP_PRIV bit if it is a kernel mode qp. - Add a zero-stag build function for recv wrs. The uP needs a PBL allocated and passed down in the recv WR so it can construct a hw pbl for the zero-stag sges. Note: we need to place a few restrictions on zero-stag usage: 1) all SGEs in a RECV WR must either be zero-stag or not. No mixing. 2) an individual SGE length cannot exceed 128MB for a zero-stag SGE. This is ok since its not possible to allocate such a large chunk of pinned contiguous dma-coherent memory. - Add an optimized non zero-stag recv wr format for kernel users. This is needed to optimize both zero and non-zero stag cracking in the recv path for kernel users. - Remove the iwch_ prefix from the static build functions. TODO: Bump the required fw major number since the driver/fw interface has changed. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 12 ++- drivers/infiniband/hw/cxgb3/cxio_wr.h | 13 ++- drivers/infiniband/hw/cxgb3/iwch_provider.c | 4 - drivers/infiniband/hw/cxgb3/iwch_qp.c | 127 +++++++++++++++++++++++---- 4 files changed, 130 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 340e418..9c049ba 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -278,7 +278,7 @@ int cxio_create_qp(struct cxio_rdev *rdev_p, u32 kernel_domain, if (!wq->qpid) return -ENOMEM; - wq->rq = kzalloc(depth * sizeof(u64), GFP_KERNEL); + wq->rq = kzalloc(depth * sizeof(struct t3_swrq), GFP_KERNEL); if (!wq->rq) goto err1; @@ -302,6 +302,7 @@ int cxio_create_qp(struct cxio_rdev *rdev_p, u32 kernel_domain, if (!kernel_domain) wq->udb = (u64)rdev_p->rnic_info.udbell_physbase + (wq->qpid << rdev_p->qpshift); + wq->rdev = rdev_p; PDBG("%s qpid 0x%x doorbell 0x%p udb 0x%llx\n", __func__, wq->qpid, wq->doorbell, (unsigned long long) wq->udb); return 0; @@ -1266,13 +1267,16 @@ proc_cqe: wq->sq_rptr = CQE_WRID_SQ_WPTR(*hw_cqe); PDBG("%s completing sq idx %ld\n", __func__, Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2)); - *cookie = (wq->sq + - Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2))->wr_id; + *cookie = wq->sq[Q_PTR2IDX(wq->sq_rptr,wq->sq_size_log2)].wr_id; wq->sq_rptr++; } else { PDBG("%s completing rq idx %ld\n", __func__, Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)); - *cookie = *(wq->rq + Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)); + *cookie = wq->rq[Q_PTR2IDX(wq->rq_rptr,wq->rq_size_log2)].wr_id; + if (wq->rq[Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)].pbl_addr) + cxio_hal_pblpool_free(wq->rdev, + wq->rq[Q_PTR2IDX(wq->rq_rptr, + wq->rq_size_log2)].pbl_addr, T3_STAG0_PBL_SIZE); wq->rq_rptr++; } diff --git a/drivers/infiniband/hw/cxgb3/cxio_wr.h b/drivers/infiniband/hw/cxgb3/cxio_wr.h index 952074f..4a8b7fb 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_wr.h +++ b/drivers/infiniband/hw/cxgb3/cxio_wr.h @@ -39,6 +39,9 @@ #define T3_MAX_SGE 4 #define T3_MAX_INLINE 64 +#define T3_STAG0_PBL_SIZE (2 * T3_MAX_SGE << 3) +#define T3_STAG0_MAX_PBE_LEN (128 * 1024 * 1024) +#define T3_STAG0_PAGE_SHIFT 15 #define Q_EMPTY(rptr,wptr) ((rptr)==(wptr)) #define Q_FULL(rptr,wptr,size_log2) ( (((wptr)-(rptr))>>(size_log2)) && \ @@ -664,6 +667,11 @@ struct t3_swsq { int signaled; }; +struct t3_swrq { + __u64 wr_id; + __u32 pbl_addr; +}; + /* * A T3 WQ implements both the SQ and RQ. */ @@ -680,14 +688,15 @@ struct t3_wq { u32 sq_wptr; /* sq_wptr - sq_rptr == count of */ u32 sq_rptr; /* pending wrs */ u32 sq_size_log2; /* sq size */ - u64 *rq; /* SW RQ (holds consumer wr_ids */ + struct t3_swrq *rq; /* SW RQ (holds consumer wr_ids */ u32 rq_wptr; /* rq_wptr - rq_rptr == count of */ u32 rq_rptr; /* pending wrs */ - u64 *rq_oldest_wr; /* oldest wr on the SW RQ */ + struct t3_swrq *rq_oldest_wr; /* oldest wr on the SW RQ */ u32 rq_size_log2; /* rq size */ u32 rq_addr; /* rq adapter address */ void __iomem *doorbell; /* kernel db */ u64 udb; /* user db if any */ + struct cxio_rdev *rdev; }; struct t3_cq { diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 64d62a4..8438c3c 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1005,10 +1005,10 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, qhp->ibqp.qp_num = qhp->wq.qpid; init_timer(&(qhp->timer)); PDBG("%s sq_num_entries %d, rq_num_entries %d " - "qpid 0x%0x qhp %p dma_addr 0x%llx size %d\n", + "qpid 0x%0x qhp %p dma_addr 0x%llx size %d rq_addr 0x%x\n", __func__, qhp->attr.sq_num_entries, qhp->attr.rq_num_entries, qhp->wq.qpid, qhp, (unsigned long long) qhp->wq.dma_addr, - 1 << qhp->wq.size_log2); + 1 << qhp->wq.size_log2, qhp->wq.rq_addr); return &qhp->ibqp; } diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 169056e..30085c7 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -33,10 +33,11 @@ #include "iwch.h" #include "iwch_cm.h" #include "cxio_hal.h" +#include "cxio_resource.h" #define NO_SUPPORT -1 -static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, u8 * flit_cnt) { int i; @@ -81,7 +82,7 @@ static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { int i; @@ -122,7 +123,7 @@ static int iwch_build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { if (wr->num_sge > 1) @@ -140,7 +141,7 @@ static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt, int *wr_cnt, struct t3_wq *wq) { int i; @@ -182,7 +183,7 @@ static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { wqe->local_inv.stag = cpu_to_be32(wr->ex.invalidate_rkey); @@ -241,23 +242,106 @@ static int iwch_sgl2pbl_map(struct iwch_dev *rhp, struct ib_sge *sg_list, return 0; } -static int iwch_build_rdma_recv(struct iwch_dev *rhp, union t3_wr *wqe, +static int build_rdma_recv(struct iwch_qp *qhp, union t3_wr *wqe, struct ib_recv_wr *wr) { - int i; - if (wr->num_sge > T3_MAX_SGE) - return -EINVAL; + int i, err = 0; + u32 pbl_addr[T3_MAX_SGE]; + u8 page_size[T3_MAX_SGE]; + + err = iwch_sgl2pbl_map(qhp->rhp, wr->sg_list, wr->num_sge, pbl_addr, + page_size); + if (err) + return err; + wqe->recv.pagesz[0] = page_size[0]; + wqe->recv.pagesz[1] = page_size[1]; + wqe->recv.pagesz[2] = page_size[2]; + wqe->recv.pagesz[3] = page_size[3]; wqe->recv.num_sgle = cpu_to_be32(wr->num_sge); for (i = 0; i < wr->num_sge; i++) { wqe->recv.sgl[i].stag = cpu_to_be32(wr->sg_list[i].lkey); wqe->recv.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); + + /* to in the WQE == the offset into the page */ + wqe->recv.sgl[i].to = cpu_to_be64(((u32) wr->sg_list[i].addr) % + (1UL << (12 + page_size[i]))); + + /* pbl_addr is the adapters address in the PBL */ + wqe->recv.pbl_addr[i] = cpu_to_be32(pbl_addr[i]); + } + for (; i < T3_MAX_SGE; i++) { + wqe->recv.sgl[i].stag = 0; + wqe->recv.sgl[i].len = 0; + wqe->recv.sgl[i].to = 0; + wqe->recv.pbl_addr[i] = 0; + } + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].wr_id = wr->wr_id; + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].pbl_addr = 0; + return 0; +} + +static int build_zero_stag_recv(struct iwch_qp *qhp, union t3_wr *wqe, + struct ib_recv_wr *wr) +{ + int i; + u32 pbl_addr; + u32 pbl_offset; + + + /* + * The T3 HW requires the PBL in the hw recv descriptor to reference + * a PBL entry. So we allocate the max needed PBL memory here and pass + * it to the uP in the recv WR. The uP will build the PBL and setup + * the hw recv descriptor. + */ + pbl_addr = cxio_hal_pblpool_alloc(&qhp->rhp->rdev, T3_STAG0_PBL_SIZE); + if (!pbl_addr) + return -ENOMEM; + + /* + * Compute the 8B aligned offset. + */ + pbl_offset = (pbl_addr - qhp->rhp->rdev.rnic_info.pbl_base) >> 3; + + wqe->recv.num_sgle = cpu_to_be32(wr->num_sge); + + for (i = 0; i < wr->num_sge; i++) { + + /* + * Use a 128MB page size. This and an imposed 128MB + * sge length limit allows us to require only a 2-entry HW + * PBL for each SGE. This restriction is acceptable since + * since it is not possible to allocate 128MB of contiguous + * DMA coherent memory! + */ + if (wr->sg_list[i].length > T3_STAG0_MAX_PBE_LEN) + return -EINVAL; + wqe->recv.pagesz[i] = T3_STAG0_PAGE_SHIFT; + + /* + * T3 restricts a recv to all zero-stag or all non-zero-stag. + */ + if (wr->sg_list[i].lkey != 0) + return -EINVAL; + wqe->recv.sgl[i].stag = 0; + wqe->recv.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); wqe->recv.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); + wqe->recv.pbl_addr[i] = cpu_to_be32(pbl_offset); + pbl_offset += 2; } for (; i < T3_MAX_SGE; i++) { + wqe->recv.pagesz[i] = 0; wqe->recv.sgl[i].stag = 0; wqe->recv.sgl[i].len = 0; wqe->recv.sgl[i].to = 0; + wqe->recv.pbl_addr[i] = 0; } + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].wr_id = wr->wr_id; + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].pbl_addr = pbl_addr; return 0; } @@ -309,17 +393,17 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, if (wr->send_flags & IB_SEND_FENCE) t3_wr_flags |= T3_READ_FENCE_FLAG; t3_wr_opcode = T3_WR_SEND; - err = iwch_build_rdma_send(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_send(wqe, wr, &t3_wr_flit_cnt); break; case IB_WR_RDMA_WRITE: case IB_WR_RDMA_WRITE_WITH_IMM: t3_wr_opcode = T3_WR_WRITE; - err = iwch_build_rdma_write(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_write(wqe, wr, &t3_wr_flit_cnt); break; case IB_WR_RDMA_READ: t3_wr_opcode = T3_WR_READ; t3_wr_flags = 0; /* T3 reads are always signaled */ - err = iwch_build_rdma_read(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_read(wqe, wr, &t3_wr_flit_cnt); if (err) break; sqp->read_len = wqe->read.local_len; @@ -328,14 +412,14 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case IB_WR_FAST_REG_MR: t3_wr_opcode = T3_WR_FASTREG; - err = iwch_build_fastreg(wqe, wr, &t3_wr_flit_cnt, + err = build_fastreg(wqe, wr, &t3_wr_flit_cnt, &wr_cnt, &qhp->wq); break; case IB_WR_LOCAL_INV: if (wr->send_flags & IB_SEND_FENCE) t3_wr_flags |= T3_LOCAL_FENCE_FLAG; t3_wr_opcode = T3_WR_INV_STAG; - err = iwch_build_inv_stag(wqe, wr, &t3_wr_flit_cnt); + err = build_inv_stag(wqe, wr, &t3_wr_flit_cnt); break; default: PDBG("%s post of type=%d TBD!\n", __func__, @@ -394,18 +478,24 @@ int iwch_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, return -EINVAL; } while (wr) { + if (wr->num_sge > T3_MAX_SGE) { + err = -EINVAL; + *bad_wr = wr; + break; + } idx = Q_PTR2IDX(qhp->wq.wptr, qhp->wq.size_log2); wqe = (union t3_wr *) (qhp->wq.queue + idx); if (num_wrs) - err = iwch_build_rdma_recv(qhp->rhp, wqe, wr); + if (wr->sg_list[0].lkey) + err = build_rdma_recv(qhp, wqe, wr); + else + err = build_zero_stag_recv(qhp, wqe, wr); else err = -ENOMEM; if (err) { *bad_wr = wr; break; } - qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, qhp->wq.rq_size_log2)] = - wr->wr_id; build_fw_riwrh((void *) wqe, T3_WR_RCV, T3_COMPLETION_FLAG, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), 0, sizeof(struct t3_receive_wr) >> 3, T3_SOPEOP); @@ -806,7 +896,8 @@ static int rdma_init(struct iwch_dev *rhp, struct iwch_qp *qhp, init_attr.qp_dma_size = (1UL << qhp->wq.size_log2); init_attr.rqe_count = iwch_rqes_posted(qhp); init_attr.flags = qhp->attr.mpa_attr.initiator ? MPA_INITIATOR : 0; - init_attr.flags |= capable(CAP_NET_BIND_SERVICE) ? PRIV_QP : 0; + if (!qhp->ibqp.uobject) + init_attr.flags |= PRIV_QP; if (peer2peer) { init_attr.rtr_type = RTR_READ; if (init_attr.ord == 0 && qhp->attr.mpa_attr.initiator) From 63shalisak at saulttribe.net Mon Jun 30 09:02:34 2008 From: 63shalisak at saulttribe.net (conrado jianping) Date: Mon, 30 Jun 2008 16:02:34 +0000 Subject: [ofa-general] Best of Chloe, Dsquared, UGG Message-ID: <000701c8dad9$01500694$eaa298ab@fwaxbaa> The biggest and broad de luxe store for shoes and bags is just one click away. Estimated by hundred thousands of grateful customers worldwide, we carry dozens of populars brands including: Prada Armani Chanel Shoes Dior Louis Vuitton Here you willc find thousands of stunning designs for shoes, and leather products, at rock bottom pri'c'ing. Sale ends this weekend, so visit our store now and start pampering yourself and your loved ones. - Visit our site: www.eaopteail[DOT]com (copy this link and replace "[DOT]" to ".") From swise at opengridcomputing.com Mon Jun 30 10:51:12 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 12:51:12 -0500 Subject: [ofa-general] [PATCH REPOST RFCv2 0/2] RDMA: Local dma lkey support. Message-ID: <20080630175112.15731.20476.stgit@dell3.ogc.int> Reposting because I got ahead of myself on the last email. This series adds the local_dma_lkey concept to the core and fixes amso, nes, and cxgb3 to use the renamed capability flag. RFC Question: should query_device return the local_dma_lkey? Or is hanging it of the ib device struct sufficient. (I like the latter). -- Steve. From swise at opengridcomputing.com Mon Jun 30 10:53:01 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 12:53:01 -0500 Subject: [ofa-general] [PATCH REPOST RFCv2 1/2] RDMA/Core: Add local dma lkey support. In-Reply-To: <20080630175112.15731.20476.stgit@dell3.ogc.int> References: <20080630175112.15731.20476.stgit@dell3.ogc.int> Message-ID: <20080630175301.15731.47648.stgit@dell3.ogc.int> From: Steve Wise - Change the IB_DEVICE_ZERO_STAG flag to the transport-neutral name IB_DEVICE_LOCAL_DMA_LKEY. - Add local_dma_lkey u32 to the ib_device struct. Drivers will fill this in with the appropriate local dma lkey if they support it. - Fix up the drivers using this flag. --- drivers/infiniband/hw/amso1100/c2_rnic.c | 2 +- drivers/infiniband/hw/cxgb3/iwch_provider.c | 3 ++- drivers/infiniband/hw/nes/nes_hw.c | 2 +- include/rdma/ib_verbs.h | 3 ++- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/amso1100/c2_rnic.c b/drivers/infiniband/hw/amso1100/c2_rnic.c index b1441ae..dd05c48 100644 --- a/drivers/infiniband/hw/amso1100/c2_rnic.c +++ b/drivers/infiniband/hw/amso1100/c2_rnic.c @@ -454,7 +454,7 @@ int __devinit c2_rnic_init(struct c2_dev *c2dev) (IB_DEVICE_RESIZE_MAX_WR | IB_DEVICE_CURR_QP_STATE_MOD | IB_DEVICE_SYS_IMAGE_GUID | - IB_DEVICE_ZERO_STAG | + IB_DEVICE_LOCAL_DMA_LKEY | IB_DEVICE_MEM_WINDOW); /* Allocate the qptr_array */ diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index 3101e3c..f27dedb 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1254,8 +1254,9 @@ int iwch_register_device(struct iwch_dev *dev) memset(&dev->ibdev.node_guid, 0, sizeof(dev->ibdev.node_guid)); memcpy(&dev->ibdev.node_guid, dev->rdev.t3cdev_p->lldev->dev_addr, 6); dev->ibdev.owner = THIS_MODULE; - dev->device_cap_flags = IB_DEVICE_ZERO_STAG | + dev->device_cap_flags = IB_DEVICE_LOCAL_DMA_LKEY | IB_DEVICE_MEM_WINDOW; + dev->ibdev.local_dma_lkey = 0; if (fw_supports_fastreg(dev)) dev->device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; diff --git a/drivers/infiniband/hw/nes/nes_hw.c b/drivers/infiniband/hw/nes/nes_hw.c index d3278f1..070b0c7 100644 --- a/drivers/infiniband/hw/nes/nes_hw.c +++ b/drivers/infiniband/hw/nes/nes_hw.c @@ -398,7 +398,7 @@ struct nes_adapter *nes_init_adapter(struct nes_device *nesdev, u8 hw_rev) { nesadapter->base_pd = 1; nesadapter->device_cap_flags = - IB_DEVICE_ZERO_STAG | IB_DEVICE_MEM_WINDOW; + IB_DEVICE_LOCAL_DMA_LKEY | IB_DEVICE_MEM_WINDOW; nesadapter->allocated_qps = (unsigned long *)&(((unsigned char *)nesadapter) [(sizeof(struct nes_adapter)+(sizeof(unsigned long)-1))&(~(sizeof(unsigned long)-1))]); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 74c24b9..2155a33 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -91,7 +91,7 @@ enum ib_device_cap_flags { IB_DEVICE_RC_RNR_NAK_GEN = (1<<12), IB_DEVICE_SRQ_RESIZE = (1<<13), IB_DEVICE_N_NOTIFY_CQ = (1<<14), - IB_DEVICE_ZERO_STAG = (1<<15), + IB_DEVICE_LOCAL_DMA_LKEY = (1<<15), IB_DEVICE_RESERVED = (1<<16), /* old SEND_W_INV */ IB_DEVICE_MEM_WINDOW = (1<<17), /* @@ -1096,6 +1096,7 @@ struct ib_device { __be64 node_guid; u8 node_type; u8 phys_port_cnt; + u32 local_dma_lkey; }; struct ib_client { From swise at opengridcomputing.com Mon Jun 30 10:53:03 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 12:53:03 -0500 Subject: [ofa-general] [PATCH REPOST RFCv2 2/2] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <20080630175112.15731.20476.stgit@dell3.ogc.int> References: <20080630175112.15731.20476.stgit@dell3.ogc.int> Message-ID: <20080630175303.15731.88673.stgit@dell3.ogc.int> From: Steve Wise Handling the zero-stag in recv WRs requires some extra logic: - Only set the QP_PRIV bit if it is a kernel mode qp. - Add a zero-stag build function for recv wrs. The uP needs a PBL allocated and passed down in the recv WR so it can construct a hw pbl for the zero-stag sges. Note: we need to place a few restrictions on zero-stag usage: 1) all SGEs in a RECV WR must either be zero-stag or not. No mixing. 2) an individual SGE length cannot exceed 128MB for a zero-stag SGE. This is ok since its not possible to allocate such a large chunk of pinned contiguous dma-coherent memory. - Add an optimized non zero-stag recv wr format for kernel users. This is needed to optimize both zero and non-zero stag cracking in the recv path for kernel users. - Remove the iwch_ prefix from the static build functions. TODO: Bump the required fw major number since the driver/fw interface has changed. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 12 ++- drivers/infiniband/hw/cxgb3/cxio_wr.h | 13 ++- drivers/infiniband/hw/cxgb3/iwch_provider.c | 4 - drivers/infiniband/hw/cxgb3/iwch_qp.c | 127 +++++++++++++++++++++++---- 4 files changed, 130 insertions(+), 26 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 340e418..9c049ba 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -278,7 +278,7 @@ int cxio_create_qp(struct cxio_rdev *rdev_p, u32 kernel_domain, if (!wq->qpid) return -ENOMEM; - wq->rq = kzalloc(depth * sizeof(u64), GFP_KERNEL); + wq->rq = kzalloc(depth * sizeof(struct t3_swrq), GFP_KERNEL); if (!wq->rq) goto err1; @@ -302,6 +302,7 @@ int cxio_create_qp(struct cxio_rdev *rdev_p, u32 kernel_domain, if (!kernel_domain) wq->udb = (u64)rdev_p->rnic_info.udbell_physbase + (wq->qpid << rdev_p->qpshift); + wq->rdev = rdev_p; PDBG("%s qpid 0x%x doorbell 0x%p udb 0x%llx\n", __func__, wq->qpid, wq->doorbell, (unsigned long long) wq->udb); return 0; @@ -1266,13 +1267,16 @@ proc_cqe: wq->sq_rptr = CQE_WRID_SQ_WPTR(*hw_cqe); PDBG("%s completing sq idx %ld\n", __func__, Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2)); - *cookie = (wq->sq + - Q_PTR2IDX(wq->sq_rptr, wq->sq_size_log2))->wr_id; + *cookie = wq->sq[Q_PTR2IDX(wq->sq_rptr,wq->sq_size_log2)].wr_id; wq->sq_rptr++; } else { PDBG("%s completing rq idx %ld\n", __func__, Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)); - *cookie = *(wq->rq + Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)); + *cookie = wq->rq[Q_PTR2IDX(wq->rq_rptr,wq->rq_size_log2)].wr_id; + if (wq->rq[Q_PTR2IDX(wq->rq_rptr, wq->rq_size_log2)].pbl_addr) + cxio_hal_pblpool_free(wq->rdev, + wq->rq[Q_PTR2IDX(wq->rq_rptr, + wq->rq_size_log2)].pbl_addr, T3_STAG0_PBL_SIZE); wq->rq_rptr++; } diff --git a/drivers/infiniband/hw/cxgb3/cxio_wr.h b/drivers/infiniband/hw/cxgb3/cxio_wr.h index 952074f..4a8b7fb 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_wr.h +++ b/drivers/infiniband/hw/cxgb3/cxio_wr.h @@ -39,6 +39,9 @@ #define T3_MAX_SGE 4 #define T3_MAX_INLINE 64 +#define T3_STAG0_PBL_SIZE (2 * T3_MAX_SGE << 3) +#define T3_STAG0_MAX_PBE_LEN (128 * 1024 * 1024) +#define T3_STAG0_PAGE_SHIFT 15 #define Q_EMPTY(rptr,wptr) ((rptr)==(wptr)) #define Q_FULL(rptr,wptr,size_log2) ( (((wptr)-(rptr))>>(size_log2)) && \ @@ -664,6 +667,11 @@ struct t3_swsq { int signaled; }; +struct t3_swrq { + __u64 wr_id; + __u32 pbl_addr; +}; + /* * A T3 WQ implements both the SQ and RQ. */ @@ -680,14 +688,15 @@ struct t3_wq { u32 sq_wptr; /* sq_wptr - sq_rptr == count of */ u32 sq_rptr; /* pending wrs */ u32 sq_size_log2; /* sq size */ - u64 *rq; /* SW RQ (holds consumer wr_ids */ + struct t3_swrq *rq; /* SW RQ (holds consumer wr_ids */ u32 rq_wptr; /* rq_wptr - rq_rptr == count of */ u32 rq_rptr; /* pending wrs */ - u64 *rq_oldest_wr; /* oldest wr on the SW RQ */ + struct t3_swrq *rq_oldest_wr; /* oldest wr on the SW RQ */ u32 rq_size_log2; /* rq size */ u32 rq_addr; /* rq adapter address */ void __iomem *doorbell; /* kernel db */ u64 udb; /* user db if any */ + struct cxio_rdev *rdev; }; struct t3_cq { diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index f27dedb..b4b05a6 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1005,10 +1005,10 @@ static struct ib_qp *iwch_create_qp(struct ib_pd *pd, qhp->ibqp.qp_num = qhp->wq.qpid; init_timer(&(qhp->timer)); PDBG("%s sq_num_entries %d, rq_num_entries %d " - "qpid 0x%0x qhp %p dma_addr 0x%llx size %d\n", + "qpid 0x%0x qhp %p dma_addr 0x%llx size %d rq_addr 0x%x\n", __func__, qhp->attr.sq_num_entries, qhp->attr.rq_num_entries, qhp->wq.qpid, qhp, (unsigned long long) qhp->wq.dma_addr, - 1 << qhp->wq.size_log2); + 1 << qhp->wq.size_log2, qhp->wq.rq_addr); return &qhp->ibqp; } diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 169056e..30085c7 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -33,10 +33,11 @@ #include "iwch.h" #include "iwch_cm.h" #include "cxio_hal.h" +#include "cxio_resource.h" #define NO_SUPPORT -1 -static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, u8 * flit_cnt) { int i; @@ -81,7 +82,7 @@ static int iwch_build_rdma_send(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { int i; @@ -122,7 +123,7 @@ static int iwch_build_rdma_write(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { if (wr->num_sge > 1) @@ -140,7 +141,7 @@ static int iwch_build_rdma_read(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt, int *wr_cnt, struct t3_wq *wq) { int i; @@ -182,7 +183,7 @@ static int iwch_build_fastreg(union t3_wr *wqe, struct ib_send_wr *wr, return 0; } -static int iwch_build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, +static int build_inv_stag(union t3_wr *wqe, struct ib_send_wr *wr, u8 *flit_cnt) { wqe->local_inv.stag = cpu_to_be32(wr->ex.invalidate_rkey); @@ -241,23 +242,106 @@ static int iwch_sgl2pbl_map(struct iwch_dev *rhp, struct ib_sge *sg_list, return 0; } -static int iwch_build_rdma_recv(struct iwch_dev *rhp, union t3_wr *wqe, +static int build_rdma_recv(struct iwch_qp *qhp, union t3_wr *wqe, struct ib_recv_wr *wr) { - int i; - if (wr->num_sge > T3_MAX_SGE) - return -EINVAL; + int i, err = 0; + u32 pbl_addr[T3_MAX_SGE]; + u8 page_size[T3_MAX_SGE]; + + err = iwch_sgl2pbl_map(qhp->rhp, wr->sg_list, wr->num_sge, pbl_addr, + page_size); + if (err) + return err; + wqe->recv.pagesz[0] = page_size[0]; + wqe->recv.pagesz[1] = page_size[1]; + wqe->recv.pagesz[2] = page_size[2]; + wqe->recv.pagesz[3] = page_size[3]; wqe->recv.num_sgle = cpu_to_be32(wr->num_sge); for (i = 0; i < wr->num_sge; i++) { wqe->recv.sgl[i].stag = cpu_to_be32(wr->sg_list[i].lkey); wqe->recv.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); + + /* to in the WQE == the offset into the page */ + wqe->recv.sgl[i].to = cpu_to_be64(((u32) wr->sg_list[i].addr) % + (1UL << (12 + page_size[i]))); + + /* pbl_addr is the adapters address in the PBL */ + wqe->recv.pbl_addr[i] = cpu_to_be32(pbl_addr[i]); + } + for (; i < T3_MAX_SGE; i++) { + wqe->recv.sgl[i].stag = 0; + wqe->recv.sgl[i].len = 0; + wqe->recv.sgl[i].to = 0; + wqe->recv.pbl_addr[i] = 0; + } + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].wr_id = wr->wr_id; + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].pbl_addr = 0; + return 0; +} + +static int build_zero_stag_recv(struct iwch_qp *qhp, union t3_wr *wqe, + struct ib_recv_wr *wr) +{ + int i; + u32 pbl_addr; + u32 pbl_offset; + + + /* + * The T3 HW requires the PBL in the hw recv descriptor to reference + * a PBL entry. So we allocate the max needed PBL memory here and pass + * it to the uP in the recv WR. The uP will build the PBL and setup + * the hw recv descriptor. + */ + pbl_addr = cxio_hal_pblpool_alloc(&qhp->rhp->rdev, T3_STAG0_PBL_SIZE); + if (!pbl_addr) + return -ENOMEM; + + /* + * Compute the 8B aligned offset. + */ + pbl_offset = (pbl_addr - qhp->rhp->rdev.rnic_info.pbl_base) >> 3; + + wqe->recv.num_sgle = cpu_to_be32(wr->num_sge); + + for (i = 0; i < wr->num_sge; i++) { + + /* + * Use a 128MB page size. This and an imposed 128MB + * sge length limit allows us to require only a 2-entry HW + * PBL for each SGE. This restriction is acceptable since + * since it is not possible to allocate 128MB of contiguous + * DMA coherent memory! + */ + if (wr->sg_list[i].length > T3_STAG0_MAX_PBE_LEN) + return -EINVAL; + wqe->recv.pagesz[i] = T3_STAG0_PAGE_SHIFT; + + /* + * T3 restricts a recv to all zero-stag or all non-zero-stag. + */ + if (wr->sg_list[i].lkey != 0) + return -EINVAL; + wqe->recv.sgl[i].stag = 0; + wqe->recv.sgl[i].len = cpu_to_be32(wr->sg_list[i].length); wqe->recv.sgl[i].to = cpu_to_be64(wr->sg_list[i].addr); + wqe->recv.pbl_addr[i] = cpu_to_be32(pbl_offset); + pbl_offset += 2; } for (; i < T3_MAX_SGE; i++) { + wqe->recv.pagesz[i] = 0; wqe->recv.sgl[i].stag = 0; wqe->recv.sgl[i].len = 0; wqe->recv.sgl[i].to = 0; + wqe->recv.pbl_addr[i] = 0; } + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].wr_id = wr->wr_id; + qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, + qhp->wq.rq_size_log2)].pbl_addr = pbl_addr; return 0; } @@ -309,17 +393,17 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, if (wr->send_flags & IB_SEND_FENCE) t3_wr_flags |= T3_READ_FENCE_FLAG; t3_wr_opcode = T3_WR_SEND; - err = iwch_build_rdma_send(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_send(wqe, wr, &t3_wr_flit_cnt); break; case IB_WR_RDMA_WRITE: case IB_WR_RDMA_WRITE_WITH_IMM: t3_wr_opcode = T3_WR_WRITE; - err = iwch_build_rdma_write(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_write(wqe, wr, &t3_wr_flit_cnt); break; case IB_WR_RDMA_READ: t3_wr_opcode = T3_WR_READ; t3_wr_flags = 0; /* T3 reads are always signaled */ - err = iwch_build_rdma_read(wqe, wr, &t3_wr_flit_cnt); + err = build_rdma_read(wqe, wr, &t3_wr_flit_cnt); if (err) break; sqp->read_len = wqe->read.local_len; @@ -328,14 +412,14 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case IB_WR_FAST_REG_MR: t3_wr_opcode = T3_WR_FASTREG; - err = iwch_build_fastreg(wqe, wr, &t3_wr_flit_cnt, + err = build_fastreg(wqe, wr, &t3_wr_flit_cnt, &wr_cnt, &qhp->wq); break; case IB_WR_LOCAL_INV: if (wr->send_flags & IB_SEND_FENCE) t3_wr_flags |= T3_LOCAL_FENCE_FLAG; t3_wr_opcode = T3_WR_INV_STAG; - err = iwch_build_inv_stag(wqe, wr, &t3_wr_flit_cnt); + err = build_inv_stag(wqe, wr, &t3_wr_flit_cnt); break; default: PDBG("%s post of type=%d TBD!\n", __func__, @@ -394,18 +478,24 @@ int iwch_post_receive(struct ib_qp *ibqp, struct ib_recv_wr *wr, return -EINVAL; } while (wr) { + if (wr->num_sge > T3_MAX_SGE) { + err = -EINVAL; + *bad_wr = wr; + break; + } idx = Q_PTR2IDX(qhp->wq.wptr, qhp->wq.size_log2); wqe = (union t3_wr *) (qhp->wq.queue + idx); if (num_wrs) - err = iwch_build_rdma_recv(qhp->rhp, wqe, wr); + if (wr->sg_list[0].lkey) + err = build_rdma_recv(qhp, wqe, wr); + else + err = build_zero_stag_recv(qhp, wqe, wr); else err = -ENOMEM; if (err) { *bad_wr = wr; break; } - qhp->wq.rq[Q_PTR2IDX(qhp->wq.rq_wptr, qhp->wq.rq_size_log2)] = - wr->wr_id; build_fw_riwrh((void *) wqe, T3_WR_RCV, T3_COMPLETION_FLAG, Q_GENBIT(qhp->wq.wptr, qhp->wq.size_log2), 0, sizeof(struct t3_receive_wr) >> 3, T3_SOPEOP); @@ -806,7 +896,8 @@ static int rdma_init(struct iwch_dev *rhp, struct iwch_qp *qhp, init_attr.qp_dma_size = (1UL << qhp->wq.size_log2); init_attr.rqe_count = iwch_rqes_posted(qhp); init_attr.flags = qhp->attr.mpa_attr.initiator ? MPA_INITIATOR : 0; - init_attr.flags |= capable(CAP_NET_BIND_SERVICE) ? PRIV_QP : 0; + if (!qhp->ibqp.uobject) + init_attr.flags |= PRIV_QP; if (peer2peer) { init_attr.rtr_type = RTR_READ; if (init_attr.ord == 0 && qhp->attr.mpa_attr.initiator) From hrosenstock at xsigo.com Mon Jun 30 13:17:04 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 30 Jun 2008 13:17:04 -0700 Subject: [ofa-general] Re: [PATCH] opensm/include/iba/ib_types.h: Definition of Congestion Control MADs In-Reply-To: <20080630160030.GI7229@sashak.voltaire.com> References: <48688300.5050902@dev.mellanox.co.il> <20080630090014.GN12659@sashak.voltaire.com> <4868B81F.4080500@dev.mellanox.co.il> <20080630160030.GI7229@sashak.voltaire.com> Message-ID: <1214857024.3486.208.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-30 at 19:00 +0300, Sasha Khapyorsky wrote: > On 13:40 Mon 30 Jun , Yevgeny Kliteynik wrote: > > > > Right, I forgot to include here additional info about IB Spec. > > Current CC annex version has many problems. Attached file > > is the right format of CC MADs (tables 508, 509, 511 - 517, > > 519, 520). > > I see. Probably it is better to wait for at least "approved" state? Current spec is approved but wrong in certain formats AFAIK :-( -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Mon Jun 30 13:15:21 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 13:15:21 -0700 Subject: [ofa-general] [PATCH REPOST RFCv2 0/2] RDMA: Local dma lkey support. In-Reply-To: <20080630175112.15731.20476.stgit@dell3.ogc.int> (Steve Wise's message of "Mon, 30 Jun 2008 12:51:12 -0500") References: <20080630175112.15731.20476.stgit@dell3.ogc.int> Message-ID: > Question: should query_device return the local_dma_lkey? Or is hanging > it of the ib device struct sufficient. (I like the latter). If the value is in struct ib_device then it seems silly to me to add code to copy that value somewhere else... anyone who calls ib_query_device() already has the ib_device pointer and can just use the value directly. - R. From rdreier at cisco.com Mon Jun 30 13:18:51 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 13:18:51 -0700 Subject: [ofa-general] Re: [PATCH REPOST RFCv2 1/2] RDMA/Core: Add local dma lkey support. In-Reply-To: <20080630175301.15731.47648.stgit@dell3.ogc.int> (Steve Wise's message of "Mon, 30 Jun 2008 12:53:01 -0500") References: <20080630175112.15731.20476.stgit@dell3.ogc.int> <20080630175301.15731.47648.stgit@dell3.ogc.int> Message-ID: > --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c > + dev->ibdev.local_dma_lkey = 0; why this specific initialization but not in nes or amso1100? I think the device structure is zeroed out on allocation anyway. > __be64 node_guid; > u8 node_type; > u8 phys_port_cnt; > + u32 local_dma_lkey; It's a little silly of me to mention this, given how few of these structures are used, but I think it would be nicer to put local_dma_lkey after node_guid to avoid leaving padding in the structure. - R. From swise at opengridcomputing.com Mon Jun 30 13:31:51 2008 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Jun 2008 15:31:51 -0500 Subject: [ofa-general] Re: [PATCH REPOST RFCv2 1/2] RDMA/Core: Add local dma lkey support. In-Reply-To: References: <20080630175112.15731.20476.stgit@dell3.ogc.int> <20080630175301.15731.47648.stgit@dell3.ogc.int> Message-ID: <486942B7.4030402@opengridcomputing.com> Roland Dreier wrote: > > --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c > > + dev->ibdev.local_dma_lkey = 0; > > why this specific initialization but not in nes or amso1100? I think > the device structure is zeroed out on allocation anyway. > > I wasn't sure if I should assume that. Plus setting it explicitly shows that we're supporting the feature. WRT amso and nes, they don't even set the capability bit in the device structure at registration time. They only return when querying the device. So I punted. :) > > __be64 node_guid; > > u8 node_type; > > u8 phys_port_cnt; > > + u32 local_dma_lkey; > > It's a little silly of me to mention this, given how few of these > structures are used, but I think it would be nicer to put local_dma_lkey > after node_guid to avoid leaving padding in the structure. > > - R. > Can do. From rdreier at cisco.com Mon Jun 30 13:47:07 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 13:47:07 -0700 Subject: [ofa-general] Re: [PATCH v4 for-2.6.27] IPOIB: add LRO support. In-Reply-To: <4867D58C.5030404@gmail.com> (Vladimir Sokolovsky's message of "Sun, 29 Jun 2008 21:33:48 +0300") References: <4867D58C.5030404@gmail.com> Message-ID: thanks, applied... a couple comments though (I fixed everything up myself, no need to resend): > From c859eed63331e43993b0c3f0c903e48d1e27def2 Mon Sep 17 00:00:00 2001 > From: Vladimir Sokolovsky > Date: Mon, 23 Jun 2008 15:38:27 +0300 > Subject: [PATCH] IPOIB: add LRO support. Can you guys stop including this header in the patches you send? I just have to edit it out by hand to avoid it getting included in the changelog. If you want to control the author field (which makes sense in this case, since Vlad sent it from his gmail account), then just put the From: line in. > +extern int ipoib_use_lro; > +extern int ipoib_lro_max_aggr; these variables can just be static to ipoib_main.c, since no other file uses them. Which is fortunate because you put the extern declaration inside #ifdef CONFIG_INFINIBAND_IPOIB_DEBUG. > + /* check if ip header and tcp header are complete */ > + if (iph->tot_len < ip_len + tcp_hdrlen(skb)) iph->tot_len is a __be16, so I assume this line should be fixed to read: if (be16_to_cpu(iph->tot_len) < ip_len + tcp_hdrlen(skb)) (running sparse found this bug -- another reason to run sparse on your patches before submitting them). > + lro_receive_skb(&priv->lro.lro_mgr, skb, 0); sparse prefers NULL for a pointer to 0 here. > +#define IPOIB_STATS_LEN ARRAY_SIZE(ipoib_gstrings_stats) seems cleaner just to use the expression itself in the one place that it is referenced. > +ipoib_get_strings(struct net_device *netdev, u32 stringset, u8 *data) > +{ > + switch (stringset) { > + case ETH_SS_STATS: > + memcpy(data, *ipoib_gstrings_stats, > + sizeof(ipoib_gstrings_stats)); > + data += sizeof(ipoib_gstrings_stats); indentation broke, and the "data +=" line is pointless anyway, since data is a local variable in a function that doesn't do anything more after that line. - R. From hrosenstock at xsigo.com Mon Jun 30 14:28:14 2008 From: hrosenstock at xsigo.com (Hal Rosenstock) Date: Mon, 30 Jun 2008 14:28:14 -0700 Subject: [ofa-general] Re: [PATCH RFC] opensm/event_plugin: plugin API version 2 In-Reply-To: <20080630142705.GF7229@sashak.voltaire.com> References: <20080625222331.GB28318@sashak.voltaire.com> <1214501218.13056.620.camel@hrosenstock-ws.xsigo.com> <20080627181931.GO29456@sashak.voltaire.com> <20080627130333.08d783c6.weiny2@llnl.gov> <20080627203745.GA29198@obsidianresearch.com> <20080627230339.GV29456@sashak.voltaire.com> <1214825806.3486.173.camel@hrosenstock-ws.xsigo.com> <20080630142705.GF7229@sashak.voltaire.com> Message-ID: <1214861294.3486.216.camel@hrosenstock-ws.xsigo.com> On Mon, 2008-06-30 at 17:27 +0300, Sasha Khapyorsky wrote: > On 04:36 Mon 30 Jun , Hal Rosenstock wrote: > > > > If this idea catches on, aside from exposing the internals in an > > unstructured manner, I see a mix and match issue with plugins developing > > which may lead to certain OpenSM versions being selected for > > compatibility or worse no choice due to plugin version compatibility as > > they get to support whatever versions they choose. > > As we learned recently having structured API leads to such problems just > well, in this case one can expect "stable API" even if it was never > declared. I hardly think that situation is comparable if you are referring to the library APIs exposed inside of OpenSM for the vendors who wanted these without OpenSM itself (for diags and ibutils). > I think I agree with Ira - OpenSM version exact match enforcement will > make it clearer that plugin writer should be ready to rebuild and > possibly update its code. should being the operative word. > > Might this be better handled as packaging with separate packages based > > on licenses ? > > What do you mean? This is related to what I wrote below about the OpenFabrics licensing requirements. The idea is if GPL licensing were to be allowed (perhaps only in some limited context), then there could be two different packages: dual and GPL. In that way plugins would be more assured of being compatible with each other and OpenSM. -- Hal > > > > You can do alot of good by making the internal APIs 'plug in like' so > > > > adding new things doesn't require touching lots of places without > > > > going down the whole messy road of actual dynamically loadable plug ins.. > > > > > > > > But if you can't identify a fixed, clean API for a dynamically > > > > loadable plugin then you almost certainly should not have them in an > > > > open source project... > > > > > > > > This is probably better long term as far as encouraging more > > > > contribution to opensm rather than encouraging contributions to be > > > > kept as plugins.. > > > > I think this is safer from an engineering and support standpoint. > > > > > GPL is some sort of an issue - OFA requires dual licensing, so GPL only > > > code cannot enter OpenSM. > > > > Yes, OFA licensing is dual BSD and GPL. Perhaps some modification is > > needed to handle this scenario for OpenSM better. Was that > > entertained/explored ? > > Not by me. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From chu11 at llnl.gov Mon Jun 30 15:11:29 2008 From: chu11 at llnl.gov (Al Chu) Date: Mon, 30 Jun 2008 15:11:29 -0700 Subject: [ofa-general] [Infiniband-diags] Fix regenerate cache corner case. Message-ID: <1214863890.14097.73.camel@cardanus.llnl.gov> Hey Sasha, Fixes a corner case I noticed. Nothing too fancy. Al -- Albert Chu chu11 at llnl.gov 925-422-5311 Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-fix-regenerate-cache-corner-case.patch Type: text/x-patch Size: 1297 bytes Desc: not available URL: From gel at bnet.worms.geis.com Mon Jun 30 15:19:25 2008 From: gel at bnet.worms.geis.com (Edward Bruno) Date: Mon, 30 Jun 2008 14:19:25 -0800 Subject: [ofa-general] Vagrea, Celias (qcxqx) Message-ID: <01c8dabc$4c5e1480$9ca2c8d4@gel> Vagria and Calias pills may help you. We also have many more things in our store! Now. Visit us now. http://answerfair.com/?said=c4 From corbet at lwn.net Mon Jun 30 15:55:43 2008 From: corbet at lwn.net (Jonathan Corbet) Date: Mon, 30 Jun 2008 16:55:43 -0600 Subject: [ofa-general] Re: [PATCH] IB/uverbs: BKL is not needed for ib_uverbs_open() In-Reply-To: References: <20080627095418.2353267d@bike.lwn.net> Message-ID: <20080630165543.1ccd8858@bike.lwn.net> On Fri, 27 Jun 2008 14:43:20 -0700 Roland Dreier wrote: > Is this sort of patch the next step in the BKL-removal plan? Is now > an appropriate time to send such things? > > If this looks good to you I will do a careful audit of the rest of > drivers/infiniband/core and send you a few more similar patches. That's the general idea: audit the code where the BKL appears and remove it when it's safe to do so. It's always an appropriate time; the only question is whether you think this change is good to go into the tree for 2.6.27, or whether it should hang out in linux-next for one more cycle. I would assume that the extra time would not be necessary. Thanks, jon From rdreier at cisco.com Mon Jun 30 17:15:51 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 17:15:51 -0700 Subject: [ofa-general] Re: [PATCH] IB/uverbs: BKL is not needed for ib_uverbs_open() In-Reply-To: <20080630165543.1ccd8858@bike.lwn.net> (Jonathan Corbet's message of "Mon, 30 Jun 2008 16:55:43 -0600") References: <20080627095418.2353267d@bike.lwn.net> <20080630165543.1ccd8858@bike.lwn.net> Message-ID: > That's the general idea: audit the code where the BKL appears and > remove it when it's safe to do so. It's always an appropriate time; > the only question is whether you think this change is good to go into > the tree for 2.6.27, or whether it should hang out in linux-next for > one more cycle. I would assume that the extra time would not be > necessary. Yes, this change should be good to go for 2.6.27. Thanks, Roland From sean.hefty at intel.com Mon Jun 30 18:31:17 2008 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 30 Jun 2008 18:31:17 -0700 Subject: [ofa-general] libibumad umad_recv and umad_poll Message-ID: <000501c8db1a$288470a0$bb37170a@amr.corp.intel.com> I have a couple questions regarding the libibumad interface, and I'm hoping someone might know the answers to. Do any apps call umad_recv()/umad_poll() from multiple threads? I'm specifically interested in opensm and the diagnostic utilities. If so, can umad_recv() support a mix of blocking and non-blocking calls concurrently? Or are all calls from a single app always blocking or always non-blocking? Is the user expected to modify the fd to support non-blocking calls? I'm more concerned with the current, actual usage of the interface, rather than the theoretical operation. Thanks, - Sean From rdreier at cisco.com Mon Jun 30 20:12:46 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 20:12:46 -0700 Subject: [ofa-general] Re: [PATCH] IB/IPoIB: Fix change mtu when switching to UD mode In-Reply-To: <20080630160651.GA6024@mtls03> (Eli Cohen's message of "Mon, 30 Jun 2008 19:06:51 +0300") References: <20080630160651.GA6024@mtls03> Message-ID: > When changing from CM mode to UD mode, use the kernel function > dev_set_mtu() instead of directly setting dev->mtu. This is necessary > so any other subsystem that need to know about this will (this is done > through the call to call_netdevice_notifiers(). What type of breakage do we get without this patch? Does the assignment of dev->mtu in ipoib_mcast_join_task() need the same conversion? - R. From eli at dev.mellanox.co.il Mon Jun 30 21:11:27 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 1 Jul 2008 07:11:27 +0300 Subject: [ofa-general] Re: [PATCH] IB/IPoIB: Fix change mtu when switching to UD mode In-Reply-To: References: <20080630160651.GA6024@mtls03> Message-ID: <20080701041127.GA21531@mtls03> On Mon, Jun 30, 2008 at 08:12:46PM -0700, Roland Dreier wrote: > > When changing from CM mode to UD mode, use the kernel function > > dev_set_mtu() instead of directly setting dev->mtu. This is necessary > > so any other subsystem that need to know about this will (this is done > > through the call to call_netdevice_notifiers(). > > What type of breakage do we get without this patch? Nothing that I could see but it seems the right way to do it anyway. Here's a short history I got to it. I sent you a few weeks ago a patch that sets the MTU to the max supported by CM mode when moving to CM mode (through the sysfs mode file, sorry I can't send url now since the openfabrics web server is down at the moment), for which you did not response yet. I took that patch to ofed however. Then, a week ago, I started getting complains from the mailing list on drop of performance in CM mode. I noticed it happens when changing from UD to CM mode but if I changed the MTU from the command line, I did not experience this problem. Investigating a litte farther, I came to this solution which fixed the CM problem too. So I intend to send, later, the original patch again but with this modification as well. > Does the assignment > of dev->mtu in ipoib_mcast_join_task() need the same conversion? I think it does, though I haven't checked the effect of this yet. From ogerlitz at voltaire.com Mon Jun 30 22:07:10 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 01 Jul 2008 08:07:10 +0300 Subject: [ofa-general] [PATCH RFC] RDMA/cxgb3: Fixes for zero-stag. In-Reply-To: <48691081.5080907@opengridcomputing.com> References: <20080626225945.27989.73122.stgit@dell3.ogc.int> <48672023.9050108@voltaire.com> <4867B28D.5090603@opengridcomputing.com> <48686660.9020001@voltaire.com> <4868F6BA.4070601@opengridcomputing.com> <48691081.5080907@opengridcomputing.com> Message-ID: <4869BB7E.8050600@voltaire.com> Steve Wise wrote: > NFSRDMA was planning on using the zero stag for recvs on iwarp > transports. This boils down to only needing the zero stag + fastreg > mrs for NFSRDMA server over T3, avoiding dma_mrs which have limits in > the T3 HW. >> the zero STag breaks the semantics slightly in that there is no >> longer any PD attached to the key returned by ib_get_dma_mr Steve, I am fine with the approach of new verb, however, I still want to make sure that this can be used under a unified IB/iWARP design of a ULP without much pain... the kernel ULPs: rNFS, iSER initiator (Lustre, RDS, etc) use the dma mr for recv nd send work-requests. So how you envision the ULP code in the presence of the new verb - is it the ULP to check if the device supports the capability and if yes, use dev->local_dma_lkey else call get_dma_mr and use the returned lkey? in you comment above you say "using the zero stag for recvs on iwarp transports" does zero stag applies also to sends? if not, will sends use dma-mr? Or. From ogerlitz at voltaire.com Mon Jun 30 22:10:31 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 01 Jul 2008 08:10:31 +0300 Subject: [ofa-general] Re: [PATCH REPOST RFCv2 1/2] RDMA/Core: Add local dma lkey support. In-Reply-To: References: <20080630175112.15731.20476.stgit@dell3.ogc.int> <20080630175301.15731.47648.stgit@dell3.ogc.int> Message-ID: <4869BC47.7080405@voltaire.com> Roland Dreier wrote: > > --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c > > + dev->ibdev.local_dma_lkey = 0; > > why this specific initialization but not in nes or amso1100? I think the device structure is zeroed out on allocation anyway. To prevent confusion, lets have it documented somewhere that in the iwarp case IB_DEVICE_LOCAL_DMA_LKEY refers to zero-stag and as the name suggest its an lkey which equals zero, how does this sound? Or. From ogerlitz at voltaire.com Mon Jun 30 22:17:28 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 01 Jul 2008 08:17:28 +0300 Subject: [ofa-general] Re: [PATCH 3/3 v4] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: References: Message-ID: <4869BDE8.2000102@voltaire.com> Roland Dreier wrote: > Please put the patch's changelog and any other information that should not be included in the actual commit message after the "---" line; then it will be automatically stripped by standard patch import scripts. Did you mean the change log before the --- line and the other information (v2,3,4 changes, etc) after the --- line? Or. From eli at dev.mellanox.co.il Mon Jun 30 23:04:37 2008 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Tue, 1 Jul 2008 09:04:37 +0300 Subject: [ofa-general] [PATCH] IB/ipoib: set max CM MTU when moving to CM mode Message-ID: <20080701060437.GC6024@mtls03> >From fce9f2d3a91abe2df8676cd709b72c7a2c7670ed Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Tue, 1 Jul 2008 08:54:39 +0300 Subject: [PATCH] IB/ipoib: set max CM MTU when moving to CM mode This will relieve the user from the need to restore CM mode MTU every time he switchs from UD to CM mode. With the current code, if the user fails to reset the MTU to a decent higher value, he will get poor performance. Signed-off-by: Eli Cohen --- drivers/infiniband/ulp/ipoib/ipoib_cm.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c index 82f3f2f..1ad8ef6 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c @@ -1404,6 +1404,7 @@ static ssize_t set_mode(struct device *d, struct device_attribute *attr, dev->features &= ~(NETIF_F_IP_CSUM | NETIF_F_SG | NETIF_F_TSO); priv->tx_wr.send_flags &= ~IB_SEND_IP_CSUM; + dev_set_mtu(dev, ipoib_cm_max_mtu(dev)); ipoib_flush_paths(dev); return count; } -- 1.5.6 From rdreier at cisco.com Mon Jun 30 23:14:59 2008 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Jun 2008 23:14:59 -0700 Subject: [ofa-general] Re: [PATCH 3/3 v4] ib/ipoib: blocking multicast loopback ipoib packets In-Reply-To: <4869BDE8.2000102@voltaire.com> (Or Gerlitz's message of "Tue, 01 Jul 2008 08:17:28 +0300") References: <4869BDE8.2000102@voltaire.com> Message-ID: > > Please put the patch's changelog and any other information that > > should not be included in the actual commit message after the "---" > > line; then it will be automatically stripped by standard patch > > import scripts. > Did you mean the change log before the --- line and the other > information (v2,3,4 changes, etc) after the --- line? Sorry, by "changelog" I meant (as I said) the patch's changelog. Basically anything that should be stripped before committing should go after the "---" line, while the changelog for kernel history should go before. From ogerlitz at voltaire.com Mon Jun 30 23:28:44 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 01 Jul 2008 09:28:44 +0300 Subject: [ofa-general] [PATCH] IB/IPoIB: Fix change mtu when switching to UD mode In-Reply-To: <20080630160651.GA6024@mtls03> References: <20080630160651.GA6024@mtls03> Message-ID: <4869CE9C.2050408@voltaire.com> Eli Cohen wrote: > > use the kernel function dev_set_mtu() instead of directly setting dev->mtu. This is necessary so any other subsystem that need to know about this will (this is done through the call to call_netdevice_notifiers(). Please note that a flow which invokes call_netdevice_notifiers() should be wrapped by rtnl_lock()/rtnl_unlock() and the context should be made sleepable /non-atomic, etc the the consumers of this event might go to sleep (eg on memory allocation for the skb of the netlink event to user space etc). Eli, yes, using dev_set_mtu() seems the correct way to go, even though I don't think it explains any of the ipoib problems I have came across over the last days, which I reported to you on the list, so in that respect I don't really understand your other email where you have pointed to me the mtu/setting issue. Or. From ogerlitz at voltaire.com Mon Jun 30 23:41:30 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 1 Jul 2008 09:41:30 +0300 (IDT) Subject: [ofa-general] Re: performance drop for datagram mode with the new connectx FW In-Reply-To: References: Message-ID: > Using the new connectx FW (2.5), I see performance drop to almost > zero with ipoib datagram mode. The code that runs on these systems > is ofed 1.3 and not mainline kernel, details below. > > Running netperf With connected mode (64k MTU) I get about 950MB/s > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to > see about 650MB/s and above with FW 2.3 and datagram mode. Not that > it could explain the drop, but the NIC reports to the OS stateless > offload support - /sys/class/net/ib1/features is 0x11423 I think to manage narrowing this a little further, the issue seems most notable under long messages sent over datagram mode, could be an issue in the LSO engine ?! see in this table, ==================================================== mode mtu size bw note ==================================================== datagram 2044 64000 30 <---------- problem datagram 2044 2000 430 datagram 2044 2000 300 TCP_NODELAY ----------------------------------------------------- connected 2044 64000 450 connected 2044 2000 450 connected 2044 2000 300 TCP_NODELAY --------------------------------------------------- connected 64000 64000 930 connected 64000 2000 930 connected 64000 2000 470 TCP_NODELAY ==================================================== notes: - in all cases, I have set the mtu manually - verbs tests of bidirectional bandwidth shows that the HCA UD and RC engines work very well, and that the cables/switch etc are operating fine as the SDR limit is easily reached. # qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 rc_bi_bw rc_bi_bw: bw = 1.91 GB/sec # qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 ud_bi_bw ud_bi_bw: send_bw = 1.95 GB/sec recv_bw = 1.95 GB/sec datagram mode mtu 2044 # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 33.00 MBytes/s over 10.31 seconds Interim result: 22.27 MBytes/s over 1.48 seconds Interim result: 34.22 MBytes/s over 2.51 seconds Interim result: 34.05 MBytes/s over 1.01 seconds Interim result: 22.88 MBytes/s over 1.49 seconds Interim result: 30.03 MBytes/s over 1.00 seconds Interim result: 28.26 MBytes/s over 1.01 seconds Interim result: 28.21 MBytes/s over 1.00 seconds Interim result: 14.26 MBytes/s over 1.98 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 429.54 MBytes/s over 1.00 seconds Interim result: 429.22 MBytes/s over 1.00 seconds Interim result: 429.50 MBytes/s over 1.00 seconds Interim result: 429.47 MBytes/s over 1.00 seconds Interim result: 429.35 MBytes/s over 1.00 seconds Interim result: 429.10 MBytes/s over 1.00 seconds Interim result: 429.16 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 294.15 MBytes/s over 1.32 seconds Interim result: 294.27 MBytes/s over 1.00 seconds Interim result: 294.98 MBytes/s over 1.00 seconds Interim result: 294.74 MBytes/s over 1.00 seconds Interim result: 294.45 MBytes/s over 1.00 seconds Interim result: 294.53 MBytes/s over 1.00 seconds Interim result: 294.35 MBytes/s over 1.00 seconds Interim result: 294.43 MBytes/s over 1.00 seconds connected mode mtu 2044 # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 161.05 MBytes/s over 1.71 seconds Interim result: 231.38 MBytes/s over 1.00 seconds Interim result: 231.16 MBytes/s over 1.00 seconds Interim result: 232.40 MBytes/s over 1.00 seconds Interim result: 322.28 MBytes/s over 1.00 seconds Interim result: 448.25 MBytes/s over 1.00 seconds Interim result: 448.54 MBytes/s over 1.00 seconds Interim result: 448.36 MBytes/s over 1.00 seconds Interim result: 448.52 MBytes/s over 1.00 seconds Interim result: 448.18 MBytes/s over 1.00 seconds Interim result: 448.06 MBytes/s over 1.00 seconds Interim result: 448.26 MBytes/s over 1.00 seconds Interim result: 448.41 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 454.45 MBytes/s over 1.00 seconds Interim result: 455.10 MBytes/s over 1.00 seconds Interim result: 454.87 MBytes/s over 1.00 seconds Interim result: 454.64 MBytes/s over 1.00 seconds Interim result: 454.30 MBytes/s over 1.00 seconds Interim result: 454.11 MBytes/s over 1.00 seconds Interim result: 454.08 MBytes/s over 1.00 seconds Interim result: 454.07 MBytes/s over 1.00 seconds Interim result: 454.06 MBytes/s over 1.00 seconds Interim result: 454.19 MBytes/s over 1.00 seconds Interim result: 454.42 MBytes/s over 1.00 seconds Interim result: 454.31 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 305.93 MBytes/s over 1.00 seconds Interim result: 306.84 MBytes/s over 1.00 seconds Interim result: 306.88 MBytes/s over 1.00 seconds Interim result: 306.80 MBytes/s over 1.00 seconds Interim result: 306.69 MBytes/s over 1.00 seconds Interim result: 306.41 MBytes/s over 1.00 seconds Interim result: 307.25 MBytes/s over 1.00 seconds Interim result: 307.56 MBytes/s over 1.00 seconds Interim result: 307.51 MBytes/s over 1.00 seconds Interim result: 307.58 MBytes/s over 1.00 seconds connected mode mtu 64000 # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 932.87 MBytes/s over 1.21 seconds Interim result: 931.33 MBytes/s over 1.00 seconds Interim result: 930.20 MBytes/s over 1.00 seconds Interim result: 929.64 MBytes/s over 1.00 seconds Interim result: 931.16 MBytes/s over 1.00 seconds Interim result: 929.96 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo Interim result: 932.82 MBytes/s over 1.46 seconds Interim result: 930.69 MBytes/s over 1.00 seconds Interim result: 931.36 MBytes/s over 1.00 seconds Interim result: 930.05 MBytes/s over 1.00 seconds Interim result: 931.40 MBytes/s over 1.00 seconds Interim result: 930.07 MBytes/s over 1.00 seconds # netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 475.01 MBytes/s over 1.58 seconds Interim result: 474.22 MBytes/s over 1.00 seconds Interim result: 474.25 MBytes/s over 1.00 seconds Interim result: 473.14 MBytes/s over 1.00 seconds Interim result: 473.62 MBytes/s over 1.00 seconds Interim result: 474.37 MBytes/s over 1.00 seconds Interim result: 473.81 MBytes/s over 1.00 seconds From ogerlitz at voltaire.com Mon Jun 30 23:52:00 2008 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 1 Jul 2008 09:52:00 +0300 (IDT) Subject: [ofa-general] Re: performance drop for datagram mode with the new connectx FW In-Reply-To: References: Message-ID: some details on the setup / nodes I used the fw-25408-2_5_000-MHGH28-XTC_A1.bin (size 487388) firmware file which was downloaded from the mellanox website and this command $ mstflint -d 07:00.0 -i /tmp/fw-25408-2_5_000-MHGH28-XTC_A1.bin b to burn it (eg on a node with the hca at position 07:00.0) The connectx HCAs we use here are eagle-DDR, I can double check it if this matters node A (internal IP 172.25.5.157) ================================= - RH 5.1 kernel 2.6.18-53.el5 SMP x86_84 - 8GB RAM - 2 quad core 2.6 GHZ Intel CPUs $ lspci | grep IB 07:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR] (rev a0) $ mstflint -d 07:00.0 q Image type: ConnectX FW Version: 2.5.0 Device ID: 25418 Chip Revision: A0 Description: Node Port1 Port2 Sys image GUIDs: 0002c90200258630 0002c90200258631 0002c90200258632 0002c90200258633 Board ID: (MT_04A0110002) VSD: PSID: MT_04A0110002 node B (internal IP 172.25.5.77) ================================= - RH 5 kernel 2.6.18-8.el5 SMP x86_84 - 8GB RAM - 2 dual core 2.6 GHZ Intel CPUs $ lspci | grep Inf 03:00.0 InfiniBand: Mellanox Technologies Unknown device 634a (rev a0) $ mstflint -d 03:00.0 q Image type: ConnectX FW Version: 2.5.0 Device ID: 25418 Chip Revision: A0 Description: Node Port1 Port2 Sys image GUIDs: 0002c9020025821c 0002c9020025821d 0002c9020025821e 0002c9020025821f Board ID: (MT_04A0110002) VSD: PSID: MT_04A0110002 From makc at sgi.com Mon Jun 30 23:56:09 2008 From: makc at sgi.com (Max Matveev) Date: Tue, 1 Jul 2008 16:56:09 +1000 Subject: [ofa-general] [PATCH 2/2] libibmad: print warning when _do_madrpc fails with destination port information In-Reply-To: <20080625160934.5025c073.weiny2@llnl.gov> References: <20080624144839.35b6d38f.weiny2@llnl.gov> <18529.37449.500476.509711@kuku.melbourne.sgi.com> <20080624192251.38214cf1.weiny2@llnl.gov> <20080625160934.5025c073.weiny2@llnl.gov> Message-ID: <18537.54537.494764.490582@kuku.melbourne.sgi.com> On Wed, 25 Jun 2008 16:09:34 -0700, Ira Weiny wrote: weiny2> Included are 2 patches which accomplish the same thing "print weiny2> warning when _do_madrpc fails with destination port weiny2> information". However, they now use the existing portid2str weiny2> function and modify that to A) print gids in nice ipv6 format weiny2> and B) print slid,dlid and drpath. Looks good now. max